Backup and Recovery Lesson

By Bryon D Beilman

I thought I would discuss a scenario that came up with one of our clients and consultants. The client was going to move their servers to a new data center. We had to do this with very little notice and help them transition their business critical servers with minimal downtime.

So what do you do before a migration? How do you manage DNS TTL's ? How do you ensure you have backups? What about database backups? Do you know your configuration? When in the process do you change your IP's? Do you know how your servers will fit into the new space? Do you have the vendor support numbers if you need them? There are alot of questions to ask, (and answer) and how much do you prepare? For what level of disaster?

In our scenario, we arrived at the site with two servers, one server was independent and was the most business critical and the other was a server with 5 RAID disks that had 3 virtual machines handling other important but less critical functions. The first server went in, with little effort. A few configuration tweaks a handful of firewall tests and validations and the service was up and operational. The 2nd, and more redundant server, started up and had a kernel panic. It could not find the OS? Hmm, well, this was CentOS, so the first thing was to make sure the controller and all the disks were seated correctly, and then boot into the BIOS to check the hardware RAID status. After seeing all the disks, the boot failed again. The RAID controller indicated that it could not do the check because it was in degraded mode and the rebuild utility said it could not do it because it was not degraded. The /boot partition was visible, but the OS partition (and all the data) was not mountable nor fsck-able (even with alternate superblocks). More analysis followed....

So with a quick OS rebuild, application of configuration files and data from a disk backup, we were back on-line. The backup scenario in this case consisted of a few very basic and free technologies that worked very well. Here is the summary:

Since we handle clients of all sizes, some clients with limited rack space and budget do not have the ability to have full backup servers, media servers and tape libraries, but there are ways to back up data so you can restore it. Rsync is also very efficient and can even be used over a fast WAN, esp for incremental backups.

In the end it all worked out. The consultant had other tools and tricks to use (like an extra disk, network cables etc), but did not have to use them.