Posted on 2009-07-07

Rackspace customers have become frantic about the lack of so called "Fanatical Support" as they have suffered for over a week with power and network outages

Rackspace prides it self on having the best support and infrastructure in its market segment, and charges a fairly high premium for its services, based on this reputation. Recent events however, have shown a lack of redundancy in a specific part of Rackspace's infrastructure, specifically its electrical system. A excitation failure in a generator caused an entire wing of the data center to lose power for a fairly length period, and two maintenance windows were required to effect repairs. Then later a failure in a UPS (that customers were assured had just been recertified) caused a network outage to a bank and a half of servers.

When I worked at Ontario Power Generation, losing power to the data center was actually a very common occurrence, especially considering that there were 8 500mw generators on site, powering a huge portion of southern Ontario. Our electrical system consisted of a specialized switch that automatically tried to get power from two separate feeds from the generator house and then a central UPS bank. Additionally, each server was connected to its own individual UPS to cover small outages and to handle graceful shutdown of the servers during prolonged outages.

Obviously what Rackspace needs to consider is moving away from solely relying on centralized UPS banks, and adding an additional layer of protection with rack level UPSs. Another route is the one Google chose, moving away from centralized UPS solutions all together, and for the sake of efficiency, moving away from the AC to DC to AC to DC conversion you run through when you deliver mains power (AC) into a UPS battery (DC), then deliver it to a server (AC) where the power supply converts it to DC used by the components. Google embeds a UPS battery (DC) directly into each server, and with a $2 change to the motherboard, the server can draw power directly from the battery when AC power is lost.

By: Allan Jude

Tipped By: Jeff Weisbein

Posted on 2009-06-26

Identi.ca the premier instance of the Laconi.ca OpenMicroBlogging platform, attempted to perform an upgrade to 0.8.0 as a final test before releasing the new version, and the site has been pretty much entirely down ever since. The developer's blog blames the problem on their new cloud hosting provider, who seems to have promised more than they can deliver.

Read more

Posted on 2009-06-26

If the news of a pop star dieing is enough to cripple the biggest sites on the internet, what would happen if there was a truly important news item of global interest? This is a matter of the utmost urgency, what if something serious were to happen, and you couldn't find out about it?

Read more

Posted on 2009-06-24

Infrastructure is all the various bits that hold a website up, like the piers of a bridge, or the legs of a chair. For your average blog or forum the most common failure is overloading the database, which happens when unoptimized queries lock tables for too long, or when a single database just cannot handle the volume of queries that are coming in. When any one part of that infrastructure fails, the entire site comes crashing down. In this brief article Security Analyst Allan Jude examines some of the common pitfalls of infrastructure design.

Read more