by Bryon D Beilman
Power and cooling are two issues that come up quite a bit in the IT world.
Most recently the events in SF regarding the Power restored in San Francisco and a Generator Failures Caused 365 Main Outage , show that the loss of power can have crippling effects on today's business. Of course, the scale of this outage was out of many IT manager's hands, but it does emphasize our reliance on power.
Managing power in any small computer room or data center is an important task. Since this Blog is designed to talk about the good as well as the bad, I want to share with you a few power related issues that I have witnessed that caused business disruption.
Company X : All critical applications on non-UPS server
-Company X has critical applications on a Bladeframe system utilizing 208 V, 3 phase power, and designed to have redundant power zones. Providing UPS power for them, however is very expensive. Couple this with the fact that there were about 20 of them are in a lab, and the unit in the very back is the corporate server that hosts customer facing apps, Oracle financials and 80% of what was business critical. There was no UPS, and a hard working air conditioner. There were many instances of losing power and/or AC. Each power outage risked more than just loss of employee's time when they could not work, but the risk of data loss, DB corruption and the reality that a change in power can initiate a hardware failure.
Company Y : Annual Shutdown.
Company Y is situated in a high rise building that has annual power shutdown for the entire building. The building has an annual building wide power outage where they do power maintenance. Even though it is planned, the process of powering down 100+ hosts, disk arrays, and servers, letting them remain idle for 24 hours and then powering them back up should not be a big deal, right? A total of 7 disks decided not to spin back up after they were turned on. This could be viewed as a way to identify problematic hardware but the point here is that changing power states, (and perhaps temperature states) can have a negative impact on a business.
In those two example cases, the business sustained some type of impact due to the loss of power. It is important that businesses , and those responsible for IT, plan for proper power protection as well as perform risk analysis to help make the right decisions. It is rarely straight forward and often difficult to justify, but it is worth the effort.