ec2 key permission denied – "Authentication failed, permission denied"

Sometimes we might encounter the ec2 key permission denied error which happens due to different reasons.

This error can occur if we try to connect using the wrong user name for the AMI. It can also happen if the permissions are incorrect on the instance.

Another scenario is an incorrect SSH public key (.pub) file in the authorized_keys file.

Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to perform related AWS queries.

In this context, we shall look into how to fix this AWS error.

Nature of ec2 key permission denied

"Permission denied (publickey)" and "Authentication failed, permission denied"

In order to fix this error, we verify that we use the correct user name for the AMI.

Similarly, we verify that the instance's permissions are correct and the correct SSH public key is in the authorized_keys file.

To perform these tasks, there are four methods.

1. Use the EC2 Serial Console

It is possible to troubleshoot Nitro-based instance types if we enable EC2 Serial Console for Linux. It helps to troubleshoot boot issues, network configuration, and SSH configuration issues.

We can access it via the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).

Prior to using the serial console, we grant access to it at the account level. Then we create AWS Identity and Access Management (IAM) policies granting access to the IAM users.

Also, every instance using the serial console must include at least one password-based user.

While we run the AWS CLI commands, we may confront errors. In such a case, we should make sure to use the most recent version of the AWS CLI.

2. Use AWS Systems Manager Session Manager to log in to the instance and make corrections

This method updates permissions and injects the SSH public key into the authorized_keys file.

We need to install the SSM Agent to use this method.

i. We open the AWS Systems Manager console.

ii. Then we start a session.

iii. The ls -ld command makes sure the permissions of the files under the home directory are correct.

For example, here, ec2-user is the user name. We can change the user name according to the specific AMI.

$ ls -ld /home/ec2-user/
drwx------ 3 ec2-user ec2-user 4096 Apr  1 08:31 /home/ec2-user/

iv. Then on the local computer, we verify the SSH public key.

v. If there is no sign of the SSH public key in the output, we update the authorized_keys file to allow the SSH key.

For example, here we replace the example key with the SSH public key.

$ echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVogCW5eZogRp+vF6Ut360b0bYyTmqgYaCXOyiW77I916AS5jFL3zsCtONbGn4hnG/UGGWXpLfUV85qpVJb38fskPZNuyZtjGjXM2W7qqbCZ1N9HBb6IPBaL97tmqBi+8rD7mSkoHc40sIV+KxkQSvD6AAFjQruCjxzfGIApnOvuj6IMsVEuFHBx4QhkbCzafxo02D9BZT4+dMy7tmyuC+UiNEQpgfFoszl+4VNFTIPlQQyn6CpUiV/rFXIadXsHqc+UOdVnfEXP+30YL75RHabze/1F5MY6t94AEcmcb05Dq4vwN9IjcxKmwgvxLOXzryytepvHQU+PobBEXAMPLE' >> /home/ec2-user/.ssh/authorized_keys

vi. To correct permissions, on the EC2 instance we run:

$ sudo chown root:root /home
$ sudo chmod 755 /home
$ sudo chown ec2-user:ec2-user /home/ec2-user -R
$ sudo chmod 700 /home/ec2-user /home/ec2-user/.ssh
$ sudo chmod 600 /home/ec2-user/.ssh/authorized_keys

vii. Eventually, we end the session.

viii. Finally, we can connect to the instance via SSH.

3. Run the AWSSupport-TroubleshootSSH Automation procedure

AWSSupport-TroubleshootSSH installs the Amazon EC2Rescue tool.

Then the tool checks for and corrects issues that cause remote connection errors while connecting to a Linux machine through SSH.

4. Use a user-data script to repair SSH permissions and add the correct SSH public key to the authorized_keys file

If the instance is instance store-backed or has instance store volumes containing data if we stop the instance the data is lost.

If the instance is part of an Amazon EC2 Auto Scaling group, it might terminate if we stop. In case the instance is part of an Auto Scaling group, we temporarily remove the instance from the Auto Scaling group before the resolution steps.

Stopping and starting the instance changes the public IP address of your instance. It's a best practice to use an Elastic IP address instead of a public IP address when routing external traffic to your instance.

i. Initially, we open the Amazon EC2 console.

ii. Then we choose Instances from the navigation pane, then select the instance to launch.

iii. We stop the instance.

iv. Later, we choose Actions, Instance Settings, Edit User Data.

v. Then we copy the following user data script into the Edit User Data dialog box and Save:

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
cloud_final_modules:
- [scripts-user, always]
--//
Content-Type:
    text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash
chown root:root /home
chmod 755 /home
chown ec2-user:ec2-user /home/ec2-user -R
chmod 700 /home/ec2-user /home/ec2-user/.ssh
chmod 600 /home/ec2-user/.ssh/authorized_keys
--//

6. Eventually, we verify the SSH public key in the local computer.

7. If in the output, the signature of the SSH public key isn’t present, then we append the correct key to the user data script in step 5. If the signature matches, we skip this step.

For example, we append the SSH public key to the user data script:

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
cloud_final_modules:
- [scripts-user, always]
--//
Content-Type:
    text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash
chown root:root /home
chmod 755 /home
chmod 700 /home/ec2-user /home/ec2-user/.ssh
chmod 600 /home/ec2-user/.ssh/authorized_keys
echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVogCW5eZogRp+vF6Ut360b0bYyTmqgYaCXOyiW77I916AS5jFL3zsCtONbGn4hnG/UGGWXpLfUV85qpVJb38fskPZNuyZtjGjXM2W7qqbCZ1N9HBb6IPBaL97tmqBi+8rD7mSkoHc40sIV+KxkQSvD6AAFjQruCjxzfGIApnOvuj6IMsVEuFHBx4QhkbCzafxo02D9BZT4+dMy7tmyuC+UiNEQpgfFoszl+4VNFTIPlQQyn6CpUiV/rFXIadXsHqc+UOdVnfEXP+30YL75RHabze/1F5MY6t94AEcmcb05Dq4vwN9IjcxKmwgvxLOXzryytepvHQU+PobBEXAMPLE' >> /home/ec2-user/.ssh/authorized_keys
chown ec2-user:ec2-user /home/ec2-user -R
--//