NaviSite makes a big oversight, knocks 165,000 websites offline for nearly a week

Being that this is the first post for DNSO.net, I won’t spend too much time explaining the purpose behind the site. I understand how blogs work, and after this post drifts off the site’s homepage, I doubt that many visitors will be traveling to the absolute bottom of the archives to read it.

In summary, DNSO.net is all about posting interesting, relevant, and straightforward tech news. It’s all right if your knowledge of technology is mediocre, because these posts won’t include the complicated explanations and lingo seen all too often on other tech blogs. I possess about as much tech knowledge as the next person, but that doesn’t stop me from wanting to know about current events in the industry, and my guess is that you do too.

So in the spirit of my forthcoming posts, and the fact that I may make a mistake or two in my reporting of recent tech news (which I encourage any readers to correct), I decided to launch this blog with a story about someone else’s mistake. And lucky for me, it’s a big one.

During what many would consider a routine data center migration gone disastrously bad, the internet hosting company NaviSite, Inc. knocked 165,000 websites offline for several days last week. The service issues began on Nov. 3 when NaviSite attempted to transfer information between hundreds of servers it acquired from Alabanza, Corp. (another hosting vendor in Baltimore) to its home-based servers in Andover, Massachusetts. Ironically, NaviSite wanted to move the data between the two locations to improve the performance of the sites its serves.

NaviSite originally informed customers that their sites would be down for less than a day because of the data transfer. The company even scheduled the migration on a Saturday to lessen the impact. NaviSite’s initial plan was to move 200 of the 850 servers involved in the transfer by truck (from Baltimore to Andover). The company intended to move the data from the remaining servers virtually. Now I can’t fully explain the issues that ensued, but from what I gather, they had something to do with synchronization glitches.

When the virtual migration between the two locations began to take too long, NaviSite decided to unplug more of its servers and transport them to Andover in trucks as well. Maybe this is just me, but doesn’t this back-up plan seem a little, oh, I don’t know…short sighted? As I graciously mentioned above, I don’t know that much about technology, but unplugging a bunch of computers that weren’t working properly in the first place and driving them across a few state lines in hopes that they’ll work there doesn’t sound like the IT solution of the century. Even if physically moving more servers helped transfer some of the data, it wouldn’t help with the rest of the information that needed to be moved virtually.

In the end, many of the sites that were depending on NaviSite’s services were down for almost a week. In the days of the internet, electronic commerce, and instant gratification, that’s an eternity. It’s impossible to calculate how much revenue was lost because of the outage, and many customers are furious with the lack of response that they received from NaviSite and its customer service department during the botched migration. NaviSite has publicly apologized for the incident, but my guess is that this apology won’t help them much at this point. It’s common knowledge that thousands of disgruntled customers make for a least a few nasty lawsuits…

This entry was posted on Thursday, November 15th, 2007 and is filed under Webhosting matters. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.