Haskins said he's worked long hours in recent years, updating regulators monthly, talking to prospective buyers of foreclosed properties, dealing with lawyers and other banks. He needed to give himself an occasional pep talk.

"So I could get through the next day, I had to say, 'Well, Joe, look, you're getting roughed up a little bit, but you are still operational, you are still in the game,' " he said.

Banking consultant Bert Ely said Harbor's capital level is strong but still has some loan quality issues.

In 2013's first quarter, the bank set aside nearly $1.8 million for potential loan losses, wrote off $1.46 million in troubled loans and lost $901,000. "They aren't out of the woods, not that they were that deep in the woods," Ely said.

Haskins said the bank is dealing with two troubled loans, and its performance next quarter will be "much improved."

The bank is looking to the future. The consent order required Harbor to set up a succession plan, and in April it named chief operating officer Darius Davis as bank president.

Other banks and individuals have approached Haskins about buying Harbor. These suitors, he said, were looking for "deals/steals," and the low-ball offers were rejected.

Harbor may merge with another bank someday to reach a higher level of success, Haskins said, but there's no pressure to rush into a deal.

"It's more important to find the right partner."