Linux Rescue and Debug

By Bryon D Beilman

What would you do if you had to recover a Linux server or workstation? Maybe it was installed by you and perhaps it wasn't. What tools are useful for debugging what went wrong? The discussion below applies to many variants of Linux, but specifically applies to Redhat or CentOS for the specific syntax.

Linux Rescue

Boot into single user mode

I have found this useful if for some reason your server will not boot all the way up and you need to either disable a service before startup, run diagnostics, check files, revert changes ,etc.

As server is booting and the Linux splash screen comes up.
hit e (for edit)
select kernel you want (if there is more than one) with arrow keys
hit 'e'
There will be 3 lines
root (...)
kernel /vmlinuz/../../
/initrd-..
Select the kernel line, hit e
at the end of the line, type "single", then enter
the hit b (for boot)
It will boot the system into single user mode.

While in this mode, you can do a number of things. If you suspect a service is problematic, you can edit/move the /etc/rc3.d/S

If you are not sure what file/service is causing the problem, you can run one at a time, starting alpha numerically (ie S00microcode_ctl, then S01sysstat
./S00microcode_ctl

If along the way, some service hangs or does not start, you can look in that startup file to determine where it is failing (what sub-service) is failing.

In some cases, if it is production and you've made a change to a service or you need to get everything up but the failed service, moving that S aside , boot and then analyze the issue off-line while the core production services are still running.

Booting from a CD

Put the first Linux CD or DVD of the install media into the server.
Make sure BIOS is set to Boot from CD, or change boot order
Note: If you have a special driver (SAN, Network) that is Not Standard and you need to see that resources, you will need to have that on a Floppy/USB.

Boot from the CD, at the prompt type
>linux rescue
or
>linux dd rescue - if you have a driver disk that needs to be used. If you use this, the only difference in the process is that along the way, it will prompt you to "insert your driver disk"

It will ask you for language (English), and Country (US).
It will ask you if you want to start the network, which is useful if you need to NFS mount a recovery resource , or FTP a file from somewhere.

continue
It will attempt to mount your OS
"Your system has been mounted under /mnt/sysimage"

"Press Return to get a shell. If you would like to make your system the root environment, run the command
chroot /mnt/sysimage

Once this process is complete, you will be at a Linux prompt with the OS mounted on /mnt/sysimage. If you are using Logical Volume Manager, It can been a bit tricky, because you need to boot with an LVM-enabled CD. You may need to start LVM in order to see the logical volume managers.
"Do a vgscan, and mount the file system in the usual way"

What is useful about using chroot /mnt/sysimage ?

A few simple tools can help you when you need it most.