Rutherford, who is African-American, came to the conclusion that Democrats saw blacks as victims.
"I'm not a political and social victim, and I don't live my life that way," he said. "They maintain poverty. They keep people in place."
Rutherford, 57, has served in Republican administrations in both Annapolis and the nation's capital. He says he never seriously considered running for office until Hogan asked him to run for lieutenant governor.
"With some hesitation, I accepted," Rutherford said. "I always saw myself as the operations person when it comes to government — making the trains run on time."
Often described as reserved, Rutherford moved easily along High Street in Cambridge as he chatted with voters on a recent campaign swing through the Eastern Shore city.
Courtney Glass, a friend since grade school, predicted Marylanders will take to his quiet personality.
"People will know that he's serious, that he doesn't come across as the slap-on-the-back, ha-ha-ha kind of cat," said Glass, who still lives in Washington.
After attending a Catholic high school, Rutherford stayed in Washington to attend historically black Howard University, then crossed the country to earn master's and law degrees at the University of Southern California. He is of counsel to the law firm of Benton Potter & Murdock in Columbia, which specializes in government contracts and small-business law.
His first stint in the George W. Bush administration was in a high-ranking position at the General Services Administration, which manages federal buildings. He joined the agency in 2001, a week before 9/11, and took part in the GSA's efforts to provide security in the aftermath of the attacks.
Two years later, a top aide to Robert L. Ehrlich Jr., who had just been elected Maryland's first Republican governor in more than 30 years, came recruiting.
Rutherford said he was inclined to say no but accepted when he was offered a post in the Cabinet as secretary of the Department of General Services, overseeing such things as maintenance and construction of state buildings, real estate dealings and many procurements.
"It's not a policy position," Rutherford said. "People tried to make it political."
While Rutherford says he tried to keep politics out of the department, he found himself enmeshed in controversy over his role in an aborted land deal in Southern Maryland between the state and a politically connected developer.
Under a plan negotiated in secrecy, the late Willard Hackerman would have bought 836 acres of forest in St. Mary's River State Park for the same price the state paid for it. The agreement would have let him build homes on some parcels while gaining millions of dollars in tax breaks. Critics decried the proposed land swap as a sweetheart deal that would sell off environmentally sensitive property. The plan was the subject of General Assembly hearings, where Rutherford was grilled by lawmakers over Ehrlich's role before Hackerman backed out.
Maryland House Speaker Michael E. Busch, a Democrat, says he thought Rutherford was doing the bidding of superiors but contends his actions should concern voters nonetheless.
"I see that as a lack of independent integrity," Busch said.
Rutherford says the proposed deal was entirely aboveboard.
"The plan was transparent every step of the way. No official decision was ever made on whether to accept the offer or not, and the purchase did not take place," he said.
In 2006, Rutherford returned to Washington as assistant secretary of agriculture, where he remained until Bush left office in 2009. He went on to serve as chief administrative officer of the Republican National Committee, where he wrestled with such problems as paying off Sarah Palin's legal debts.
Rutherford has less name recognition than Ken Ulman, the Howard County executive who is the running mate of Democrat Anthony G. Brown. But if Hogan is elected, Rutherford could wind up playing a more important role in a gubernatorial administration than any of his predecessors.
As outlined by Rutherford and Hogan, the lieutenant governor would function as chief operating officer to Hogan's chief executive role. According to their plan, Cabinet secretaries would report to the governor through the lieutenant governor rather than through deputy chiefs of staff.
Unlike most recent lieutenant governors, Rutherford said he would not take on a specific policy assignment from the governor — as Kathleen Kennedy Townsend did with crime under Parris N. Glendening, and as Brown did with military base realignment and health care policy under Martin O'Malley.
He said he would spend two or three days a week working at the state government complex on West Preston Street in Baltimore because that's where many departments are located.
"I think it's important to be there," Rutherford said.
He said he would concentrate on making government more efficient.
"Do we need these giant SUVs [for state employees]?" he said. "I'm not a fan of SUVs."
Rutherford counts himself as a firm economic conservative but does not share the deep concern many Republicans feel about social issues such as abortion.
"I'm not necessarily a religious conservative. I'd say I'm more moderate on some social issues," he said.
On the topic of gay marriage, which many Maryland Republicans fought bitterly two years ago, Rutherford is accepting.
"If two people want to commit to each other, more power to them," he said, noting that he's been married to his wife, Monica, for 27 years. They have three children.
Todd Eberly, a political scientist at St. Mary's College, said the job of lieutenant governor might be a better fit for Rutherford than for Ulman, who has shown he has gubernatorial ambitions of his own. Eberly said that makes it easier for Hogan to define a role for his lieutenant governor.
"I have not picked up anything that Rutherford thinks of himself as a future governor," Eberly said.
According to Rutherford, Eberly is correct.
"After eight years as lieutenant governor, I want to ride off into the sunset," Rutherford said. "The way I'm thinking now, I'm not interested."
But his disclaimer came with a caveat: "I said I would never run for office at one point, and now I'm proving myself wrong."