Recovering from Lost PEM Files: Emergency EC2 Access Restoration

Picture this scenario: You're trying to SSH into your EC2 instance, but you can't find your PEM file. You've searched everywhere—your Downloads folder, your backup drives, your password manager—but it's nowhere to be found. Your heart sinks as you realize you might be locked out of your server forever. But don't panic! While you can't restore a lost PEM file, you can still regain access to your EC2 instance using a clever recovery technique.

Understanding the Problem

Why PEM Files Can't Be Restored

PEM files (Privacy Enhanced Mail format) contain your private key, which is used for SSH authentication. Here's the critical point: AWS never stores your private key. When you create a key pair, AWS generates the key pair and gives you the private key only once. After that, it's your responsibility to keep it safe.

Once a PEM file is lost, it cannot be recovered from AWS or any other source. The private key is gone forever, and you cannot SSH into your instance using the original key pair.

The Impact of Lost PEM Files

When you lose your PEM file:

SSH access is blocked - You cannot connect to your EC2 instance
No recovery from AWS - AWS doesn't store private keys
Potential data loss - You might lose access to important data
Service disruption - Your applications might be running but inaccessible

The Recovery Process: Volume Mounting Technique

Step 1: Create a New Key Pair

First, you need a new key pair to replace the lost one:

# Create a new key pair
aws ec2 create-key-pair --key-name recovery-key --query 'KeyMaterial' --output text > recovery-key.pem

# Set proper permissions
chmod 400 recovery-key.pem

# Extract the public key
ssh-keygen -y -f recovery-key.pem > recovery-key.pub

The new key pair will be used to replace the lost one in the instance's authorized_keys file.

Step 2: Stop the Affected Instance

You need to stop the instance to safely detach its root volume:

# Stop the instance
aws ec2 stop-instances --instance-ids i-1234567890abcdef0

# Wait for the instance to stop
aws ec2 describe-instances --instance-ids i-1234567890abcdef0 --query 'Reservations[0].Instances[0].State.Name'

Make sure the instance is completely stopped before proceeding. You can check the state using the AWS CLI or console.

Step 3: Detach the Root Volume

The root volume contains the instance's operating system and the authorized_keys file:

# Find the root volume ID
aws ec2 describe-instances --instance-ids i-1234567890abcdef0 --query 'Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId'

# Detach the volume
aws ec2 detach-volume --volume-id vol-1234567890abcdef0

Alternative method using AWS Console:

Go to EC2 > Volumes
Find the volume attached to your instance (usually /dev/xvda or /dev/sda1)
Select the volume and click Detach

Step 4: Attach Volume to a Rescue Instance

You need a working EC2 instance to mount the volume and edit the authorized_keys file:

# Launch a temporary rescue instance (or use an existing one)
aws ec2 run-instances --image-id ami-0abcdef1234567890 --instance-type t2.micro --key-name your-working-key

# Attach the volume to the rescue instance
aws ec2 attach-volume --volume-id vol-1234567890abcdef0 --instance-id i-rescue-instance-id --device /dev/sdf

Make sure you have SSH access to the rescue instance using a different key pair that you still have.

Step 5: Mount and Edit the Volume

SSH into the rescue instance and mount the volume:

# SSH into the rescue instance
ssh -i your-working-key.pem ec2-user@rescue-instance-ip

# Create a mount point
sudo mkdir /mnt/recovery

# Mount the volume
sudo mount /dev/xvdf1 /mnt/recovery

# Verify the mount
ls -la /mnt/recovery/

Step 6: Edit the Authorized Keys

Now you can edit the authorized_keys file to add your new public key:

# Navigate to the SSH directory
cd /mnt/recovery/home/ec2-user/.ssh/

# Backup the existing authorized_keys file
sudo cp authorized_keys authorized_keys.backup

# Add your new public key
sudo nano authorized_keys

Add your new public key to the file:

# Copy the content of your new public key
cat recovery-key.pub

# Paste it into the authorized_keys file

Make sure to add the new public key on a new line in the authorized_keys file. Don't remove existing keys unless you're sure they're not needed.

Step 7: Unmount and Reattach

Clean up the rescue instance:

# Unmount the volume
sudo umount /mnt/recovery

# Exit the rescue instance
exit

Reattach the volume to the original instance:

# Detach from rescue instance
aws ec2 detach-volume --volume-id vol-1234567890abcdef0

# Reattach to original instance as root volume
aws ec2 attach-volume --volume-id vol-1234567890abcdef0 --instance-id i-1234567890abcdef0 --device /dev/xvda

Step 8: Start the Instance and Test Access

Start the original instance and test SSH access:

# Start the instance
aws ec2 start-instances --instance-ids i-1234567890abcdef0

# Wait for it to be running
aws ec2 describe-instances --instance-ids i-1234567890abcdef0 --query 'Reservations[0].Instances[0].State.Name'

# Test SSH access with the new key
ssh -i recovery-key.pem ec2-user@your-instance-ip

If everything worked correctly, you should now be able to SSH into your instance using the new key pair.

Alternative Recovery Methods

Method 1: EC2 Instance Connect

If your instance supports EC2 Instance Connect (Amazon Linux 2+ with proper IAM role):

# Use EC2 Instance Connect for temporary access
aws ec2-instance-connect send-ssh-public-key \
  --instance-id i-1234567890abcdef0 \
  --availability-zone us-east-1a \
  --instance-os-user ec2-user \
  --ssh-public-key file://recovery-key.pub

Method 2: Systems Manager Session Manager

If your instance has the SSM agent installed:

# Start a session using Systems Manager
aws ssm start-session --target i-1234567890abcdef0

Method 3: User Data Script

For instances that support user data execution:

# Create a user data script to add your new key
cat > user-data.sh << EOF
#!/bin/bash
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC..." >> /home/ec2-user/.ssh/authorized_keys
chown ec2-user:ec2-user /home/ec2-user/.ssh/authorized_keys
chmod 600 /home/ec2-user/.ssh/authorized_keys
EOF

# Stop instance, modify user data, and start
aws ec2 stop-instances --instance-ids i-1234567890abcdef0
aws ec2 modify-instance-attribute --instance-id i-1234567890abcdef0 --user-data file://user-data.sh
aws ec2 start-instances --instance-ids i-1234567890abcdef0

Prevention Strategies

1. Secure PEM File Storage

# Store PEM files in a secure location
mkdir -p ~/.ssh/keys
mv your-key.pem ~/.ssh/keys/
chmod 700 ~/.ssh/keys/
chmod 400 ~/.ssh/keys/your-key.pem

2. Use Multiple Access Methods

# Create a secondary user with a different key
sudo useradd -m -s /bin/bash backup-user
sudo mkdir /home/backup-user/.ssh
sudo cp /home/ec2-user/.ssh/authorized_keys /home/backup-user/.ssh/
sudo chown -R backup-user:backup-user /home/backup-user/.ssh

3. Implement Key Rotation

# Regularly rotate your key pairs
# 1. Create new key pair
# 2. Add to authorized_keys
# 3. Test access
# 4. Remove old key
# 5. Update documentation

4. Use IAM Roles and Policies

# Attach IAM role for EC2 Instance Connect
aws iam attach-role-policy \
  --role-name EC2InstanceConnectRole \
  --policy-arn arn:aws:iam::aws:policy/EC2InstanceConnect

Best Practices for PEM File Management

1. Backup Strategy

# Create encrypted backups
gpg --symmetric --cipher-algo AES256 your-key.pem
# Store the encrypted file in multiple secure locations

2. Access Control

# Use proper file permissions
chmod 400 your-key.pem
chown $USER:$USER your-key.pem

3. Documentation

# Document your key pairs
echo "Key: your-key.pem, Created: $(date), Instance: i-1234567890abcdef0" >> ~/.ssh/keys/README.md

4. Monitoring

# Set up CloudTrail to monitor key pair usage
aws cloudtrail create-trail --name key-usage-trail --s3-bucket-name your-log-bucket

Common Pitfalls and Solutions

Pitfall 1: Wrong Volume Device

Problem: Mounting the wrong volume or using incorrect device names Solution: Always verify the volume ID and device mapping before mounting

Pitfall 2: Permission Issues

Problem: Incorrect file permissions on authorized_keys Solution: Ensure proper ownership and permissions (600 for authorized_keys, 700 for .ssh directory)

Pitfall 3: Instance State Issues

Problem: Trying to detach volumes from running instances Solution: Always stop the instance before detaching the root volume

Pitfall 4: Rescue Instance Access

Problem: Not having access to a rescue instance Solution: Keep a small, always-running instance for emergency access

Conclusion

Losing a PEM file is a serious situation, but it's not the end of the world. With the right knowledge and tools, you can regain access to your EC2 instance using the volume mounting technique. The key is to act quickly and follow the process carefully.

Remember:

PEM files cannot be restored - AWS doesn't store private keys
Volume mounting works - You can edit authorized_keys through a rescue instance
Prevention is key - Always backup your PEM files and use multiple access methods
Test your recovery process - Practice the recovery procedure before you need it

The most important lesson is to never rely on a single access method. Always have multiple ways to access your instances, and keep your PEM files secure and backed up.

Table of Contents