Baltimore’s information technology office issued a detailed warning that the city was using computer systems that were out of date, highly vulnerable to attack and not backed up, calling them “a natural target for hackers and a path for more attacks in the system.”
The warning, in an undated risk assessment obtained by The Baltimore Sun, foreshadowed the attack this month that brought down the city’s network. It specifically highlights the danger posed by ransomware, saying “extortionists are an increasing threat to any internet-connected systems.”
Senior city IT officials had said publicly in recent months that the city’s security systems were out of date — they also were struck by ransomware in 2018. But the risk assessment report lays out a specific vulnerability in greater detail.
“If and when the systems are materially compromised, it is no doubt that addressing the fallout from the compromise would be a drain on an already tight budget,” the IT office wrote in the risk assessment.
“There is no way of estimating the financial loss that could occur in trying to counteract and clean up the resulting mess,” it said.
While the report is not dated, it refers to a federal review in August 2016 of the city’s computer systems and uses a name for the IT office that was changed after the city hired the current director in September 2017.
Now, the city is calculating the costs of being attacked. An initial estimate released this week puts the cost at $18.2 million — a combination of expenditures by the IT office and lost and delayed revenue.
Sheryl Goldstein, a recently hired deputy chief of staff to Democratic Mayor Bernard C. “Jack” Young, declined to say whether the two servers mentioned in the report were in use when the current ransomware struck May 7, citing a federal investigation and the need to protect the city’s systems.
“It is obvious that our system has vulnerabilities,” Goldstein said. “Like many government entities, as well as corporations, we have systems that span the range of being more current and being outdated.”
The ransomware attack shut down the city’s email systems and its ability to take card payments for bills and fines. The real estate market was halted for several days because the finance department couldn’t verify if sellers owed the city any outstanding bills.
The city is slowly recovering. It restored some employees’ email access Wednesday and Goldstein said IT teams are working on getting people logged back onto their computers.
The risk assessment report recommended abandoning the outdated servers, rewriting the applications on the servers or buying commercially available versions. Servers are computer devices that host programs and databases used by other computers on a network.
“It’s imperative that all applications are migrated to a secure environment, especially those containing sensitive information and may fall under the state and federal privacy protection laws,” the report says.
The assessment focuses on two city servers running versions of Microsoft’s Windows Server that the company no longer supported, meaning that they would not receive routine security updates.
The U.S. Department of Homeland Security assessment of the city’s systems “identified 18 different vulnerabilities” in one of the servers, the report said.
The report said the servers hosted a combined 104 city applications; however, the document doesn’t describe their functions.
The two servers presented a particular problem if they were hit by ransomware, which involves hackers locking files and demanding payment for the keys to open them. Stricken systems can be restored from backed-up versions of files, but according to the report, “both servers have no active backups and represent single points of failure.”
While Goldstein couldn’t say the city would recover all of its data, she said it “had a pretty extensive backup system.”
Former Democratic Mayor Catherine Pugh overhauled the city’s IT office, increasing its budget and hiring director Frank Johnson for higher pay than the previous director.
Jim Smith, who was a senior adviser to Pugh with responsibility for IT, said he was not familiar with the risk assessment, but officials recognized there were problems with the city’s defenses.
“We didn’t have everything we should have to protect yourself in place,” Smith said. “It just wasn’t getting attention in the past.”
Smith, who resigned in April while Pugh was on leave amid investigations into her financial dealings, credited the former mayor with giving computer security the attention the issue deserved.
“I thought that we were turning the corner and we were setting a path,” Smith said. “As we committed more resources to it, it was going to just change the technology of the city dramatically.”
But Chris Tonjes, a former city IT director, said the ransomware attack shows that the city didn’t take basic steps to protect itself. Tonjes said he moved to upgrade outdated servers before he left the office in 2014 and said the city was taking a chance if it had left any in place.
“They rolled the dice and they lost,” he said. “I really have no sympathy.”
The New York Times reported Saturday that a tool developed by the National Security Agency and leaked online was involved in the Baltimore attack. That prompted Young and Democratic City Council President Brandon Scott to seek financial help from the federal government for the recovery.
Microsoft issued a security patch in 2017 for the vulnerability the NSA tool targeted.
Tonjes said that means the responsibility for protecting Baltimore’s systems lies with his successors at the city’s IT department.
“The bottom line is, they didn’t patch and this is why this happened," he said. “They left themselves vulnerable.”