Ivey, the former Prince George's County state's attorney, said he was quitting the race because he was unable to raise enough money to get his message out, particularly since the primary during a presidential election year is pushed up from September to April.
"Since this was going to be a short timeline," Ivey said, "you couldn't do your typical canvassing." Plus, he said, "A lot of my previous donors in Prince George's County have been hammered by the real estate collapse."
Donna Edwards could not be reached for comment.
Ivey, of Cheverly, announced his candidacy on Oct. 26, and the matchup in the heavily Democratic 4th District was expected to be tough. Ivey said he was able to raise about $150,000, although his Federal Election Commission reports show no money was raised. Year-end campaign finance reports are not due to the FEC until the end of this month.
Edwards' FEC reports show she has collected $231,967, before year-end reports are released.
The withdrawal came just a few hours before the deadline to file for candidacy with the Maryland State Board of Elections. Also running on the Democratic side, according to the board's Web site, are Ian Garner and George McDermott. On the Republican side, Randy Gearhart, Greg Holmes and Charles Shepherd have filed as candidates.
"That's just a sigh of relief," for Edwards, said Todd Eberly, professor of political science at St. Mary's College of Maryland. "Ivey would've been a serious challenger."
Eberly said money likely wasn't the only reason that Ivey dropped out. Edwards, of Fort Washington, had the support of the other Democrats in the state's congressional delegation.
"Parties work hard to make sure incumbents don't face challenges," he said. "There's every incentive for party incumbents to have an advantage."
Edwards won a tough primary fight to gain her seat in Congress, defeating eight-term Rep. Al Wynn in 2008, and causing Wynn to resign his seat early. She went on to win a special election and claim her seat in Congress. In that race she had significant financial support from the Service Employees International Union and MoveOn.org.
"Ivey wouldn't have been as well connected or had a coalition of support," said Eberly, explaining that Ivey would have been a new congressman in a reconfigured district, making the seat more vulnerable to a Republican challenge on the next go-round.
In the redrawing of congressional districts this year, Edwards lost some of Montgomery County's Democrats to a redrawn Rep. Roscoe Bartlett's 6th Congressional District, and picked up a less-reliably Democratic part of Anne Arundel County. Most of Laurel was also moved into the 4th Congressional District.
Ivey said he will continue to practice law at the Washington firm of Venable LLP and work to re-elect President Obama.