A team of 200 telecommunications experts believes it has narrowed to five the number of possible causes of last week's phone outages, but, a week after the widespread problems struck, it still can't identify the problem with certainty.
John O'Rourke, assistant vice president of switching technology and analysis for Bell Communications Research Inc., which is known as Bellcore, said five likely causes have been identified for the outages that have dogged Bell Atlantic Corp. and Pacific Bell since June 26.
Bellcore is the research arm of the seven regional Bell phone companies.
Mr. O'Rourke, in a telephone news conference that, in keeping with the tone of last week, started late because of a technological foul-up, said the principal theories are that the mysterious outages were caused by:
* Errors in recent software upgrades that are causing equipment tofunction improperly.
* A design flaw in the "signal transfer point [STP]," a computer that sorts and routes calls across a network. Mr. O'Rourke said such a weakness could be causing the STP to become prematurely congested during normal operation.
* An isolated problem in an individual processor within the STP that is causing it to fail.
* An "unexpected and unknown" reaction from connecting DSC equipment with equipment made by other manufacturers. The STPs of both affected Bell companies are made by DSC Communications Corp. of Plano, Texas. But those STPs are linked to hardware and software built by, among others, American Telephone & Telegraph Co. and Northern Telecom.
* Sabotage. That might include the introduction of a computer virus into the system by an outside hacker or an inside job by a disgruntled employee.
Terry Adams, a DSC spokesman, told analysts during a conference call yesterday that he was confident DSC's STP would prove not to be the "culprit."
Mr. Adams intimated that the problem was with other equipment that connects to DSC's computers.
Although it is unlikely, Mr. O'Rourke said Bellcore engineers are also looking at the possibility of an inherent flaw in the basic architecture around which the Bells' sophisticated computer networks are built.
The Bell companies' networks are built around "Signaling System 7 [SS7]," an international signaling standard. SS7 systems, which are constantly being upgraded, use a sophisticated network of computers and electronic relay stations sort and route calls quickly and efficiently.
It is SS7 that allows companies to offer services such as Caller ID and Call Waiting, and it helps cut down on toll fraud.
To find the cause of the recent rash of problems, Mr. O'Rourke said, engineers will attempt to reproduce in the laboratory the events that led up to the outages of the past week.
To that end, three test sites -- one each by AT&T;, DSC and Northern Telecom -- are being used, he said. Engineers at the test sites are working together and exchanging information daily in an effort to get to the heart of the problem as quickly as possible, Mr. O'Rourke said.
"Cooperative test actions will be used to replicate events so we can pinpoint what happened to the live network," Mr. O'Rourke said.
But he conceded that engineers might not be able to reproduce conditions precisely, which means the cause of the problem could continue to elude investigators.
"There is a finite possibility that we may not be successful" in re-creating the same conditions that led up to the outages, he said.
Up to 5 million customers of Bell Atlantic Corp. in Maryland, Washington, Virginia and parts of West Virginia lost local phone service for most of the day June 26. The problem was later traced to a faulty circuit board in a DSC-made STP in Baltimore. The same day, an outage left customers in Southern California without service for three hours. That problem also was traced to DSC equipment.
Monday, a malfunction in a DSC-made STP in Pittsburgh left 1 million Bell Atlantic customers in western Pennsylvania without phone service for most of the day. The same day, an outage in the San Francisco area disrupted service to more than 2 million phone lines for about 5 minutes. That problem was also linked to DSC equipment. Problems in both cities persisted yesterday but were cleared up by evening.
As a precaution, Bell Atlantic began installing a new AT&T-made; STP in the Washington area Monday as a backup to the existing DSC equipment.
The $1 million switch probably won't be operable for at least a month, however, because of the complexity of hooking it up and sending new traffic to it, a Bell Atlantic spokeswoman said.
In all, Bell Atlantic has six pairs of DSC switches in use, and they handle about 40 percent of all calls.
Spokesmen for the two affected Bell companies said yesterday that staffing at their STP sites have been beefed up considerably since the outages. Pacific Bell said DSC equipment is being monitored around-the-clock and that DSC is online 24 hours a day.
Chesapeake & Potomac Telephone Co. of Md. executives said yesterday that the Baltimore STP that housed the faulty circuit board suspected of being the origin of the June 26 problem had been operating for less than one month.
That computer, along with a partner housed in a C&P; office in Pikesville, went on line May 30, said Richard H. Wolfe, manager of the Baltimore and Pikesville STPs.
For a month, the company has been shifting central offices from the Washington and Hyattsville STPs to the Baltimore-Pikesville pair, Mr. Wolfe said.