The U.S. Secret Service is set to announce agent resignations amid a prostitution scandal that erupted in Colombia ahead of a visit by President Barack Obama, according to a source with knowledge of the investigation.

The resignations could occur as early as Wednesday evening, said the source, who added that not all 11 agents allegedly involved in the scandal are expected to resign.

In addition to the Secret Service members implicated in the probe, as many as 10 U.S. military personnel from all branches of the armed forces are being questioned about potential misconduct.

Five members of America's elite Army Special Forces are among those under investigation, two U.S. officials said.

The alleged prostitutes, the youngest of whom were in their early 20s, had all signed in at Cartagena's Hotel El Caribe, flashing their local IDs. But one of the women, the source said, was involved in a dispute with Secret Service personnel about how much she was to be paid. That dispute brought the incident to light.

Days after Secret Service members were placed on administrative leave for their alleged role in the scandal, investigators are now looking into whether drugs were involved, a separate source with knowledge of the probe said Wednesday.

Secret Service Director Mark Sullivan has told subordinates to use "all tools available" to conduct the investigation, and has made it known that he believes drug testing is within his rights, the source said.

It is not clear if any of those accused have been tested, and authorities say drug use is not consistent with their findings so far, though they are continuing to investigate the allegations alongside local police.

Meanwhile, Senate lawmakers have scheduled a hearing on the controversy for 10 a.m. ET next Wednesday, inviting Homeland Security Secretary Janet Napolitano to testify.

At least one congressman, Randy Forbes, a Virginia Republican, has called for Sullivan to be replaced.

"There's only so many strikes you get, in baseball it's three," said Forbes, a senior member of the House Armed Serves Committee, referencing a prior security breach when a Virginia couple crashed President Barack Obama's first White House state dinner in 2009 as well as apparent agency overspending in that same year.

"I think he's had three," Forbes added. "I think it's time to put somebody else in there to make sure we're getting a different culture in the Secret Service."

Sullivan has directed the Secret Service since May 2006. He has been with the agency since 1983.

At the time of the incident in Colombia, the Secret Service members apparently had not yet received classified documents detailing the planned movements of the president, who was coming to Cartagena for last weekend's Summit of the Americas, the source noted. In addition, the source said, all weapons apparently were in a secure location.

Obama has said he expects a "rigorous" investigation of the incident.

In Colombia, fallout from the incident led an official to say it has overshadowed his country's showcase of the city of Cartagena during the summit.

Nausicrate Perez, a municipal official in Cartagena, said authorities were waiting to get more information on the allegations regarding the agents.

Lauding the Secret Service director for taking "immediate and decisive action," Secretary Napolitano said the probe reflects the "professionalism, honor, and integrity" of the service.

"The U.S. Secret Service has a history of executing its mission with professionalism, honor and integrity and Director Sullivan's six-year stewardship of the agency has been marked by these traits," she said in a statement issued Tuesday.

Sen. Patrick Leahy, Senate Judiciary Committee chairman, issued a statement saying, "I believe that Director Sullivan is taking serious action to investigate the incident, and will be staying in close contact with him in the days and weeks ahead."