Adam Clarke on Job 20:19

He hath oppressed and hath forsaken the poor - Literally, He hath broken in pieces the forsaken of the poor; כי רצץ עזב דלים ki ritstsats azab dallim. The poor have fled from famine, and left their children behind them; and this hard-hearted wretch, meaning Job all the while, has suffered them to perish, when he might have saved them alive.

He hath violently taken away a house which he builded not - Or rather, He hath thrown down a house, and hath not rebuilt it. By neglecting or destroying the forsaken orphans of the poor, mentioned above, he has destroyed a house, (a family), while he might, by helping the wretched, have preserved the family from becoming extinct.