"I didn't want to talk to anybody," he recalled. "I just wanted to crawl into a dark hole and be left alone."
Four years later, as he makes plans to teach water safety to kids, to invest in his own businesses, to watch his son, Boomer, learn to walk, Phelps feels none of the old dread. The retired Olympian is confident he'll continue to feel contented with a life no longer defined by the dark lines at the bottom of a swimming pool.
"I think just where I am in life, that's why it's so different," he said. "Having a family, but also just being engaged in everything in my life. Before, I was just so distant from everything. I didn't want to talk to anybody. I didn't want to be around anybody. I didn't want anything. That's the biggest difference."
The 23-degree temperature was a bit of a shock compared to the 75 degrees Phelps and his wife, Nicole, felt before they boarded a flight from their home in Arizona. But the cold did not diminish Phelps' holiday cheer.
"Michael, Michael, Michael," the elementary school kids chanted as Phelps ducked in to share lunch with them. They shrieked with glee after he handed out gift bags containing swim goggles and Beats by Dre headphones (another of his sponsors).
He said the $20,000 will allow the club to expand its use of the "im" program to its summer camp in White Hall.
Phelps urged the kids to ask for help from the adults in their lives. "It's hard to go through life by yourself," he said. "I learned that the hard way." But he also advised them to write down their greatest ambitions, post them in a visible place and make plans to pursue them step by step. It was the path he followed from a seemingly mundane childhood in Towson to 28 Olympic medals.
Phelps broke into laughter after one boy raised his hand and wished him well should he decide to unretire for the 2020 Olympics in Tokyo.
"I was like, 'Wow man, I'm getting shots thrown at me from every side.' Even little kids are telling me to come back," he said afterward, grinning.
He knows he'll face questions about a comeback for at least the next three years. Why wouldn't people wonder after he won five gold medals and a silver at age 31?
Just as he said in Rio de Janeiro, Phelps told the kids he would "probably" never swim competitively again. He can't help leaving the slightest wiggle room. But at the same time, he recalled how his mother, Debbie, greeted him at his hotel in Rio after his last Olympic race in August.
And yet, even Phelps' agent and close friend, Peter Carlisle, recently said that if Los Angeles wins its bid for the 2024 Olympics, he wouldn't be surprised to see Phelps swimming at age 39.
"I was like, 'Wow, thanks Peter,'" Phelps said. "I really appreciate you firing that one out there."
Phelps has gone through the past few months in the usual post-Olympic whirlwind, striking up new sponsorship agreements, besting fellow Rio gold medalists Simone Biles and Aly Raisman on the television program Lip Sync Battle and traveling to China for appearances on behalf of Under Armour and Beats by Dre.
Earlier this week, he was in New York for Sports Illustrated's Sportsperson of the Year gala, where LeBron James told him, "You're literally a fish. Literally. I have no idea how you do what you do."
He felt a minor epiphany while he was in New York, running on the treadmill at his hotel. He realized his goals no longer had anything to do with a clock above a pool.
"I think that was my biggest transition, just setting goals where, my whole life, it's been times," he said. "That's what I've done my whole life. Now it's trying to make the transition to the real world. … We get to start a new chapter and we get to do some of the things we're extremely passionate about. And that's the fun part."
After his appearance in Edgewood, Phelps was scheduled to fly to London with Nicole and then back to the couple's home in Arizona for Boomer's first Christmas.
Asked what he looks forward to most over the next year, Phelps quickly said his son's first steps and first words.
"I'm pumped to see it, even though he's probably going to crush me when he starts to walk," he said. "I'm going to be running after him nonstop when he's running around the house, but it's going to be fun."
He and Nicole, who quietly married in June, will also finally take a honeymoon in 2017. And a sibling for Boomer? "I'm not pregnant!" Nicole quickly interjected.
"It's safe to say we want him to be over the age of 1 before," Phelps said.
"We would like this year with Boomer to kind of enjoy him a little bit more," Nicole said.
The old fire still rises in Phelps on occasion. He was cleaning out his closet recently and came across the sheet of goals he posted in the run-up to Rio. Hard as it might be to believe, he achieved just one of five. He especially wanted to break one more world record.
Phelps texted his longtime coach, Bob Bowman, a picture of the sheet. Bowman texted back a beaming, sunglasses-wearing emoji. "Let it go," he told the greatest swimmer of all time.
Phelps said he has done just that. When he's home in Arizona, which hasn't been often lately, he visits Bowman at the Arizona State pool two or three times a week. For the first time since he was in grade school, he's able to enjoy a casual swim. He and his longtime friend and training partner, Allison Schmitt, recently drove over for a workout, and Phelps joked that his heart filled with dread when Bowman ambled up to watch. Sure enough, Bowman began calling out their times.
"I was like, 'No!'" Phelps said laughing. "Just leave us alone. Let us be. But it's so peaceful being in the water now. I love it. It's the only time when I'm alone."