Someone who is “dirt-poor” is extremely poor, owning little except, perhaps, dirt. The expression began in the United States in the second half of the 19th century, although it’s not known exactly where or when. “That Navasa Guano is nearly ‘dirt poor’ as a fertilizer"was cited in newspapers in 1860. ” He was poor when he went into (political—ed.) office and claimed that he was dirt poor when he went out” was cited in 1878. “They ain’t dirt poor” was cited in 1890.
 
The “dirt-poor” expression was popularized in the 1930s during the Great Depression and received hyphenation about this time.
 
   
(Oxford English Dictionary)
dirt-poor adj. (orig. U.S.) extremely poor.
1937   Time 26 Apr. 41/1   Nearly blind and dirt-poor, Inventor Dave Mallory (Karloff) devises a burglar alarm worked by electric eyes.
1971   D. O’Connor Eye of Eagle xix. 128   They were dirt-poor; they could not bribe people.
 
Chronicling America
30 October 1860, The Daily Exchange (Baltimore, MD), pg. 2, col. 5:
AS TO THE NAVASA GUANO:
1st. That Navasa Guano is nearly “dirt poor” as a fertilizer.
       
21 June 1878, The Daily Independent (Helena, MT), pg. 2, col. 1:
A couple of weeks ago the Collector ot Baltimore retired from office. He was poor when he went into office and claimed that he was dirt poor when he went out, but in leas than three days there after some thief entered his house and robbed him of $75,000. It will be seen that poor men have afflictions thrust upon them.
 
Chronicling America
28 May 1890, The True Northerner (Paw Paw, MI), “Mark’s Substitute” by Velma Caldwell Melville, pg. 6, col. 2:
“They ain’t dirt poor.”
 
18 November 1902, Indianapolis (IN) Sun, “The Busted City: Will Receive a Handout Shortly,” pg. 5, col. 6:
The city of Indianapolis is dirt poor, and It will be with a great sigh of relief that Comptroller Breunlg will receive the December settlement from the treasurer.
 
14 September 1911, Cleveland (OH) Plain Dealer, “Shadows of the Footlights” by Archie Bell, pg. 6, col. 7:
“That makes me smile, but ‘poor’ doesn’t describe it; it was dirt poor and then some.”
 
25 October 1921, Greensboro (NC) Daily News, “Grissom Speaks Kind Words to All as He Takes Bailey’s Place,” pg. 10, col. 3:
Thus far nobody has taken up the challenge, and the county is dirt poor since the Southern and the Norfolk SOuthern are saying “gimme back.”
 
4 February 1926, Morning World-Herald (Omaha, NE), “Progressives of Yesteryear” (New York Times), pg. 12, col. 4:
To be sure, the editor of The Emporia Gazette explains that the senator is “not dirt poor.”
 
13 November 1929, The Evening Repository (Canton, OH), “Fireside Philosophy” by Robert Quillen, pg. 4, col. 3:
The people are dirt poor—ignorant, backward, oppressed.
 
Google Books
October 1936, Boys’ Life, “For the Honor” by Leonard K. Smith, pg. 6, col. 2:
“I’m poor — dirt poor.”
 
Twitter
How to move from dirt poor to the middle class
Author: Lawrence Williams
Publisher: Ruston, La. : M & M Print. Co., 1984?
Edition/Format:   Print book : English : 1st ed
 
OCLC WorldCat record
“Dirt poor, family rich”
Author: Jean Marlene Slover Chellos
Publisher: East Moline, Ill. : J. Chellos, ©2010.
Edition/Format:   Print book : English