3 Approaches for Recovering a Linux Instance Key Pair

Understanding How SSH Key Pairs Work in Linux Instances

Before diving into the recovery steps, it is essential to understand how identity verification works on cloud instances. When you launch a Linux virtual machine on AWS (such as Amazon Linux, Ubuntu, or CentOS), AWS installs the public key of the selected key pair in a specific text file within the user directory: /home/username/.ssh/authorized_keys (where "username" is typically ec2-user, ubuntu, admin, or centos). When you attempt to SSH into the instance, the server checks this file to verify that the private key you possess matches the registered public key.

If you delete, lose, or misplace the private key (.pem or .ppk file) on your local computer, SSH connections will be rejected, resulting in a "Permission denied (publickey)" error. Because AWS does not store a copy of your private key for security reasons, it cannot simply be redownloaded. Recovering access requires replacing the public key inside the instance's authorized_keys file, which is exactly what the following three approaches accomplish.

Approach 1: EC2 Instance Connect (Fastest & Zero Downtime)

Amazon EC2 Instance Connect is the absolute easiest way to recover access. When you initiate a connection using EC2 Instance Connect, the AWS API pushes a one-time-use SSH public key to your instance's metadata. This key is valid for only 60 seconds. During this window, the Instance Connect service authenticates you and drops you into a terminal session. This approach requires zero downtime and takes less than two minutes to complete.

Prerequisites for EC2 Instance Connect

To use this method, your instance must meet a few basic requirements:

• The EC2 Instance Connect package must be installed (it comes pre-installed on Amazon Linux 2, Amazon Linux 2023, and Ubuntu 20.04 or later).
• Your security group must permit inbound SSH traffic (Port 22) from the EC2 Instance Connect IP range for your region.
• Your instance must have a public IP address or be accessible via an EC2 Instance Connect Endpoint.

Step-by-Step Execution

Step 1: Navigate to the Amazon EC2 console, click on "Instances" in the left sidebar, and select the checkbox next to your locked-out instance.

Step 2: Click the "Connect" button in the top menu. Under the "EC2 Instance Connect" tab, verify the default username (e.g., ec2-user for Amazon Linux or ubuntu for Ubuntu Server) and click "Connect."

Step 3: A secure, web-based terminal window will open in your browser, establishing a shell session inside your instance. You are now logged in!

Step 4: Once inside, generate a brand new SSH key pair on your local computer. Copy the new public key. On the browser terminal, edit the authorized keys file: nano ~/.ssh/authorized_keys. Paste your new public key into the file, save, and exit. You can now use your new private key to SSH from your local machine.

Approach 2: Volume Exchange using a Rescue Instance (The Manual Standard)

If your instance does not have EC2 Instance Connect configured, or is located in a private subnet, the Volume Exchange method is the standard manual recovery approach. In this scenario, we stop the locked-out instance, detach its root volume, and attach it to a temporary "Rescue" instance. We then mount the volume, inject our new public key, unmount the volume, and move it back to the original instance. This method is incredibly reliable, though it does require stopping the instance (resulting in temporary downtime).

Step-by-Step Execution

Step 1: Stop the Locked Instance. Open the EC2 Console, select the affected instance, go to "Instance State," and select "Stop." Wait for the state to show "Stopped." Important: Do not Terminate the instance! Take note of the original root device name, typically /dev/xvda or /dev/sda1.

Step 2: Detach the Root Volume. Select the stopped instance, click the "Storage" tab, and click the Volume ID link. This will open the Volumes console. Select the volume, click "Actions," and choose "Detach Volume." Confirm the detachment.

Step 3: Launch a Temporary Rescue Instance. In the exact same Availability Zone (AZ) as your original instance, launch a temporary Linux instance (t2.micro or t3.micro is perfect). During launch, configure it to use a new SSH key pair that you have safely stored on your local computer.

Step 4: Attach the Old Volume to the Rescue Instance. Return to the Volumes console, select the detached volume, click "Actions," and select "Attach Volume." In the instance search bar, select your newly launched Rescue instance. Set the device name to something like /dev/xvdf or /dev/sdf, and click "Attach."

Step 5: Mount the Volume on the Rescue Instance. SSH into your Rescue instance using your terminal. Check the attached disks by running lsblk. Create a mount point and mount the disk using the following commands:

sudo mkdir /mnt/rescue
sudo mount -o rw,nouuid /dev/xvdf1 /mnt/rescue

Note: The nouuid flag is essential if you are using XFS filesystems (standard in Amazon Linux 2/2023) because the system will complain about duplicate UUIDs if the rescue instance and attached volume share the same base AMI.

Step 6: Inject the New Public Key. Copy the public key from the Rescue instance (or generate a new public key) and append it to the old volume's authorized keys file:

cat /home/ec2-user/.ssh/authorized_keys >> /mnt/rescue/home/ec2-user/.ssh/authorized_keys

Verify that permissions on the authorized_keys file on the mounted volume are correct (typically 0600 owned by the correct user).

Step 7: Unmount the Volume. Disconnect from the disk path and unmount it safely:

sudo umount /mnt/rescue

Step 8: Reattach the Volume to the Original Instance. Detach the volume from the Rescue instance via the AWS Console. Then, select the volume, click "Attach Volume," and select your original instance. Crucial step: You must change the device name back to the exact root path you noted in Step 1 (usually /dev/xvda or /dev/sda1). If you fail to do this, the instance will not boot.

Step 9: Boot and Connect. Start your original instance and log in using the new private key! You can now safely terminate the temporary Rescue instance.

Approach 3: Create AMI and Launch a New Instance (The Clean Redeployment)

If you prefer not to manage individual block devices and volume mounts, the AMI (Amazon Machine Image) method is an excellent alternative. By creating an image of your stopped instance, you capture a complete snapshot of the root drive. You then launch a new instance from this image, which automatically lets you associate a brand new key pair during the creation wizard. This is highly effective but results in a new Instance ID and a new public IP address.

Step-by-Step Execution

Step 1: Stop the locked-out instance in the EC2 Console to ensure data consistency.

Step 2: Select the instance, then click "Actions" > "Image and templates" > "Create image." Give your image a descriptive name (e.g., Recovered-Webserver-Image) and click "Create image." Wait a few minutes for the AMI status to transition from "Pending" to "Available."

Step 3: Navigate to "AMIs" under the "Images" section in the left navigation pane. Select your newly created AMI and click "Launch instance from AMI."

Step 4: Configure the new instance with the same VPC, security groups, and IAM roles as your original instance. When you reach the "Key pair (login)" configuration section, select a new, valid key pair that you own.

Step 5: Launch the new instance. Verify that you can successfully connect via SSH using the new private key. Once verified, you can copy over any elastic IPs, update DNS records to point to the new instance, and safely terminate the original locked-out instance.

Quick Comparison of Key Pair Recovery Methods

Modern Best Practices: Moving Beyond SSH Key Pairs

While knowing how to recover key pairs is essential, modern cloud architecture aims to eliminate SSH key management altogether. Key pairs present security risks if they are shared, stored in insecure locations, or hardcoded in scripts. Consider adopting the following best practices:

• AWS Systems Manager Session Manager: SSM Session Manager allows you to connect to your instances securely using the AWS Console or AWS CLI without opening inbound Port 22 in your security groups. Connections are authenticated via IAM, logged in CloudTrail, and encrypted out-of-the-box.
• Centralized Identity Management: Integrate your Linux instances with Active Directory, AWS Directory Service, or Okta for credential-less authentication.
• Automated Key Rotation: Implement automated scripts that rotate user SSH keys regularly and store active credentials securely in AWS Secrets Manager.