So, you were minding your own business chmodding things left and right while logged in to your Ubuntu instance the other day, and now a few days later you get the Permission denied (publickey) response when you try to log in.
Even if you do sudo ssh -v -i YOUR-KEY.pem ubuntu@YOUR-IP, all the verbose flag tells you is that the key is denied. Hmmmm.
Have a think, did you change the permissions of the ubuntu home directory? You did? Oh, well that was the problem then, Ubuntu does not like having the home directory of Ubuntu changed so Ubuntu will not let you log in now. Ooopsie!
What next? You have a dead instance, well, dead in the sense that you can not ssh in to it. No worries, follow these steps:
Log in to your aws console. Stop your broken instance and detach the EBS volume that is the root volume of this instance (this is a good reason to name all volumes, but you can easily find which instance the volume is attached to by looking at the description). wait for it to come to a complete stop.
Start another instance, a temporary one, or if you have one already prepared. You will attach the root volume from your broken instance to this one. Make sure it’s in the same availability zone! us-east-1b and us-east-1d are not the same thing, yo!
Now attach the broken root volume to the new temp instance, like at /dev/sdf
NOTE: Newer kernels are gonna name the partition whatever the hell it wants anyway, so sudo sfdisk -l to find it. It might be /dev/xvdf but lets assume here that it’s /dev/sdf
ssh [your temporary instance]
sudo mkdir -m 000 /vol-a
sudo mount /dev/sdf /vol-a
Voila! You now have your root partition mounted, so head to the /home/ubuntu and set correct permissions. It should be drwxr-xr-x but just to make it simple do chmod 755. While you’re at it, add another user that can log in, just in case.
Nononono, not with the adduser command, silly wabbit, you’re not running this EBS volume right now. You have to get your hands dirty and edit some files in /etc cd /vol-a/etc. You can break some shit by manually editing /etc/passwd, /etc/shadow, /etc/group or /etc/gshadow but hey, we’re living on the edge here. Look at these files, if you don’t know what you’re doing, leave them the fuck alone. Otherwise a line like dabitch:x:1001:1001::/home/dabitch:/bin/bash works well in /passwd.
dabitch:!:: in gpasswd , dabitch:x:1001: in group, in gshadow add dabitch:!::. The /shadow file has the following structure:
Username : Password (encrypted) : Last password change (lastchanged in days) :Minimum # of days password can be unchanged : Maximum# of days password is valid: Warn # of days before warning about password change : Inactive # of days after the password expires that the account is disabled :
Now that you know all that, and you’re using a .pem file with your instance anyway, wing it! dabitch:!:16805:0:99999:7:::
Don’t forget to mkdir /home/dabitch or whatever user name you chose because you are not dabitch, I am. Arity, now …
sudo umount /vol-a
sudo rmdir /vol-a
If you get a “device is busy” it’s because you’re in vol-a, cd / first.
Shut down this temp instance and attach the EBS to your previous instance again. Remember, it’s /dev/sda1 on your instance, that’s where the root partition goes.
Now start your previous broken instance and check if you can’t log in this time. You can! Well, there you go. Now don’t do silly things like changing the /home permissions on Ubuntu again, silly wabbit.
