A person on a nearby boat named Lori M reported the capsized catamaran 32 miles northeast of Ocean City at about 9 a.m., the Coast Guard said.
People on another nearby boat, named the Just Right Five, rescued two 55-year-old men who had been onboard the catamaran, the Coast Guard said.
A MH-65 Dolphin helicopter then arrived, and a Coast Guard rescue swimmer was deployed into the water, where he was able to recover two other men, aged 53 and 65, with the help of another vessel, the Lori M, the Coast Guard said.
Both men, who have died, were unresponsive as they were brought onto the Lori M, the Coast Guard said.
"Our thoughts and prayers are with the family members and loved ones of those recovered," Capt. John Little, commander of the Coast Guard sector that responded out of Hampton Roads, said in a statement. "We sincerely appreciate the efforts of the good Samaritans who acted quickly to report this and assist the Coast Guard in this rescue."
Sun reporter Kevin Rector contributed to this report.