It took Brooks Cappella about a nanosecond to notice that his BlackBerry wasn't working. He manages the technology operations at a New York hedge fund, Highland Financial, and he was wrestling with some problems on the company's servers when he stopped getting messages late on Apr. 17. Just what he needed to make a difficult day worse. 'Any downtime is something we notice right away,' he says. 'Not having the BlackBerry didn't help.'
Gulp. Surprise turned to disbelief as millions of hyperconnected professionals found themselves in the midst of a BlackBerry blackout that lasted from late in the day of Apr. 17 until the early hours of Apr. 18. As the network for the addictive e-mail device crashed, there were pangs of stomach-churning panic in some cases, simple loneliness in others. Research in Motion (RIMM), the Canadian company behind the BlackBerry, says the e-mail service was down for a little less than 12 hours.
Leaving customers guessing
Though service was restored relatively quickly, it may take a bit longer to regain full confidence in the BlackBerry network. RIM issued a very brief statement on the issue saying the interruption affected customers in North America and consisted of e-mail delivery being 'delayed or intermittent' during the incident. But it didn't address why the network failed or what the company plans to do to avoid similar problems in the future. Analysts and customers were left guessing. 'RIM has been rather uncommunicative today about all this,' says Todd Kort, analyst with market researcher Gartner Group (IT). 'They haven't said anything more than, 'Yes there is a problem and everything has been fixed'.'
This is hardly the first time RIM has faced these kinds of issues. The BlackBerry service experienced outages of varying severity in 2001, 2003, 2005, and 2006. RIM addressed the issue in its most recent annual report, saying that 'continuity of operations planning' had been a priority during the 2005 fiscal year and that the company had added 'additional internal redundancy' in order to improve reliability of its network.
So why more outages‾ RIM's rapid growth may have contributed to the problems. Richard Doherty, research director of technology consultant Envisioneering Group, suspects a network overload of some kind was behind the outage, prompted in part by a spike in network traffic related to the users seeking news about the Virginia Tech shooting and compounded by RIM's stunning subscriber growth. The company added more than a million subscribers in its most recently reported quarter, bringing its worldwide subscriber base to about 8 million.
'Networks can only take so much at once,' he says. 'If 3% of all the people with telephones were to pick up their receivers at the same time, the phone system would collapse. When you build a network you have to plan for peak use.'
The center of the problem may have been at RIM's network operations center, or NOC, in Waterloo, Ontario. Jack Gold, an analyst and consultant based in Northborough, Mass., says that would explain why BlackBerry service on multiple wireless networks all stopped functioning. The devices are available from wireless carriers such as AT&T (T), T-Mobile, Sprint (S), and Verizon Wireless, a joint venture of Verizon (VZ) and Vodafone (VOD).