When you crashed your EBS instance, restarted, and now you can’t connect via SSH

When you crashed your EBS instance, restarted, and now you can’t connect via SSH


Lets say you did something kinda not-so-smart and you rebooted your EBS (Elastic Block Store) instance, and for some reason it restarts, but won’t let you in via SSH, or worse… It simply will not restart. Ruh-roh!

Yeah, I’ve done that. More than once too. We are all idiots when we’re tired. There’s is nothing more dangerous than a tired sysadmin.

Don’t fret! Pop over to your amazon GUI thingiemahbob, and detach that EBS from your crashed (and stopped) instance. That would be listed under “volumes” in their lovely control panel page there.

Start some other temporary server, and attach this EBS to it. Oh, make sure it’s Ubuntu or some other yummy *nix flavour because I can’t help you in windows.

Once the other server is started and has the instance attached, mount it. You will probably have to find it first. Here’s your new best friend: blkid

/dev/sda1: LABEL="cloudimg-rootfs" UUID="44ca5ce8-7504-45ee-a634-be52fdceb901" TYPE="ext3"
/dev/sdh: LABEL="uec-rootfs" UUID="f9fc3b3b-94aa-4a67-827c-c24941303aa8" SEC_TYPE="ext2" TYPE="ext3"

Lookie ! Crashed instances EBS volume of the root partition is there, woop woop! Now mount that baby!

Make the directory first, else you’ll feel like a right fool later looking at “/vol does not exist”

sudo mkdir -m 000 /vol

Then add it to fstab & link the EBS to that new directory you made

echo "/dev/sdh /vol auto noatime 0 0" | sudo tee -a /etc/fstab

Then finally: mount /vol

Now you can dig around and find the files you need to fix in that broken root partition. Or just save what you need into a new server. Have fun.