WASHINGTON -- A year after the National Security Agency nearly maxed out its electrical capacity, some offices are experiencing significant power disruptions as the agency confronts the increasingly urgent problem of an infrastructure stretched to its limits, intelligence officials said.
The spy agency has delayed the deployment of some new data-processing equipment because it is short on power and space. Outages have shut down some offices in NSA headquarters for up to half a day. And some officials fear that major problems could occur this summer as temperatures climb.
The NSA has been working to develop and implement short- and long-term plans to ensure a steady supply of electricity to the nation's largest intelligence agency; they range from creating rapid-response teams to revamping power substations, internal documents show.
The current shortage has been projected for nearly a decade. Some of the rooms that house the NSA's enormous computer systems were not designed to handle newer computers that generate considerably more heat and draw far more electricity than their predecessors.
It is the result of "mismanagement at very high levels," said Ira Winkler, a former NSA analyst. "They let it get out of hand."
NSA spokeswoman Andrea Martino declined to comment on the reason for the electrical problems. "We cannot discuss the specifics that may or may not affect the agency's operations for national security purposes," she said, adding that Congress has been kept informed.
The agency has already been forced to delay installing some high-tech equipment to avoid overloading the system, according to a senior intelligence official who spoke on condition of anonymity because he is not authorized to speak to the news media.
New equipment for data processing, as well as some purchased for one of the agency's signature initiatives, the mammoth modernization effort dubbed Turbulence, are among those that have been held up, the senior official said. The lengths of the delays are classified.
The issue has become a top priority for the NSA's director, Lt. Gen. Keith B. Alexander. In recent classified testimony to Congress, he warned that the agency would have to shut down significant amounts of equipment and resort to rolling blackouts if drastic action were not taken, the official said. Alexander also told Congress that the NSA was delaying the deployment and installation of equipment, the official said.
His testimony was part of an effort to persuade lawmakers to add more than $800 million - the exact sum is classified - to the NSA's 2007 budget, the senior official added.
Congress recently approved the NSA request in a classified spending bill, said Rep. C.A. Dutch Ruppersberger, a Maryland Democrat who chairs the House Intelligence subcommittee that oversees the agency.
However, lawmakers also reprimanded the NSA, intelligence officials said, for using money for spy operations to pay for electrical expenses without congressional approval.
"It got to a point where it became a serious problem," Ruppersberger said, referring to the NSA's power shortage. "We're attempting to deal with it now."
For brief periods last summer, the NSA hit the ceiling of the power capacity at its Fort Meade campus, forcing the agency to turn off or idle technical equipment, the senior intelligence official said. The agency also had to shift the timing of power use for some computers responsible for processing data, to even out electrical loads, and continues to do so.
Some intelligence officials said they are worried that as this summer heats up, anticipated spikes in power demand could have dire consequences.
"I don't think it's going well," said one government source with direct knowledge of the power problem, who was speaking on condition of anonymity. "I am concerned with the possibility of a large-scale blackout and the damage it would cause across the board. ... We're experiencing problems in all of our buildings."
As the NSA has attempted to reduce electricity consumption, it has turned down air-conditioning and heating systems in parts of some buildings.
"In the morning, it's like a sweatshop," the government source said.
Last winter, according to one intelligence analyst, some employees wore gloves in the office to try to keep warm, adding that it made for challenging typing.
As part of its short-term effort to redirect power use and upgrade systems, the NSA has had to resort to partial, rolling brownouts at its computer "farms" and scheduled power outages, the senior intelligence official said, adding that these have become more frequent in recent months.
Among the most significant electrical issues was a series of outages in several buildings at NSA headquarters April 30 and May 1, which caused computers to unexpectedly restart and triggered blackouts that lasted between 45 minutes and four hours in some offices, according to the government source.
Ruppersberger said he was told the outages were scheduled. Martino would not comment on whether they had been planned but said in a statement that "routine power outages are carefully coordinated to ensure that backups are in place for mission assurance during repairs and upgrades to infrastructure."
The NSA's impending electrical crisis has been predicted since at least 1998. It had become an urgent priority by last summer.
As Alexander explained in a classified March 2007 memo, the three power substations that serve the main Fort Meade campus cannot support the agency's demands, according to the senior intelligence official. Each substation serves certain buildings, and power cannot easily be transferred to other substations if demand spikes, the senior official said.
The NSA's buildings are also exceeding the limits of their 1970s- and 1980s-era wiring, the senior official said.
Routine power demands have been disrupting work in some offices for up to half a day, the government source said. He noted, for example, that new office equipment has been overloading existing circuits in one office up to three times a week, taking anywhere from 30 minutes to four hours to get the power back on. This problem, however, is not yet widespread, the senior official said.
The problems cannot be fixed quickly because substantial re-engineering is required, the senior intelligence official explained.
The NSA's top brass is also concerned about a major space crunch. The agency's post-9/11 data deluge has pushed its computerized "data centers" to near capacity, the senior official said.
In an unclassified April 2007 memo, Alexander told NSA employees that he had created a "triage team" to address power problems as they arose. He said the NSA was taking steps to re-engineer some facilities to "better take advantage of existing power."
The agency is also employing conservation measures, according to the senior intelligence official, referring to the March classified memo. One example: incorporating "thrift savings" provisions into contracts, to create incentives for the use of power-saving equipment on new projects.
Other, longer-term measures include upgrading electrical infrastructure in buildings on the Fort Meade campus and the three substations, moving additional operations elsewhere to reduce demand and building new facilities. To handle the data overload, there are additional plans to move some of the agency's data-storage equipment to new government facilities in Tennessee and Texas. But those sites are not expected to be ready until about 2010.
"These measures are effectively addressing many of the short-term data-center, distribution, and campus-wide challenges," Alexander wrote in his April memo.
Ruppersberger said he believes Alexander's plan puts the NSA "on track" to fix the electrical problems. But he added that the agency still needs to show it can implement the plan.
Senate Intelligence Committee Chairman John D. Rockefeller IV said these issues will not be resolved quickly.
"The NSA is committing significant resources to fixing the problem," the West Virginia Democrat said in a statement. "We believe they have a plan in place to address the situation, but it will take time and a considerable amount of money."
Some current and former officials question whether the short-term plans will buy enough time to resolve a mostly long-term problem.
According to an agency document, the NSA's power consumption is nearly three times higher than the average for Defense Department buildings. The main culprit: power demand from the NSA's 24-hour watch centers and supercomputers, as well as a large number of computers in each building.
Some current and former officials said they are frustrated that NSA leaders took so long to address the problem.
The problem was first brought to the attention of then-Director Kenneth Minihan in 1998 as he prepared to upgrade the agency's technology infrastructure. But he chose not to pay for electricity upgrades along with the new technology infrastructure, the senior intelligence official said. Minihan did not respond to requests for comment.
The issue has arisen periodically since then, including the planning for the NSA's modernization programs, but each time leaders chose to set it aside, the official said. As recently as 2004 and 2005, the question of an emerging power deficit came up as the NSA outlined an expansion of its field sites under then-Director Gen. Michael V. Hayden, but no formal plans were made to address electricity problems, the official said.
A spokesman said that Hayden, who now heads the CIA, rejected that accusation and said Hayden took steps to deal with the problem when he was there.
"During his service at NSA, there was major construction at regional (signals-intelligence) operations centers. One goal of that initiative was to ease the burden on the Ft. Meade compound," spokesman Paul Gimigliano said in a written statement. "It's simply wrong to suggest that the director knew of a problem with NSA's electrical infrastructure and ignored it."
The regional centers did help, the senior official acknowledged, but they did not fix the outdated electrical system at headquarters, and some of the efforts were scaled back as Hayden left.
Early in 2006, an internal NSA team produced an extensive study of the power and space shortages that highlighted the urgency of the problem and predicted that the NSA would soon hit a "ceiling" in its electrical capacity, the official said.
Only when it hit that ceiling last summer, the official added, did the agency truly begin mobilizing to address the problem.
siobhan.gorman@baltsun.com