AWS Systems Manager (SSM) Run Command Troubleshooting

I have been working with AWS SSM for couple of months, but I found the troubleshooting document on their website lacks straightforward answers. So I provide the problems that I encountered and the solution based on my experience.

Problem #1: The instance is not visible in AWS Systems Manager Console although documentation says the agent has been installed by default.

Problem #2: The instance is visible, but "Run Command" took too long and even timed out.

Solution:
  1. First thing I would check is whether the instance has a role attached to it.
  2. If so, make sure the role has AmazonEC2RoleforSSM policy attached to it since permission is required for the agent to do health check.
  3. If after all the above has been confirmed, check if the latest SSM agent has been installed and running.
  4. If SSM agent is at the latest and running, check if it is hibernating. The hibernate logic has exponential backoff, so it might not respond for a long time.
  5. If it is hibernating, we can simply restart the agent.
    • On Windows, we can run Restart-Service AmazonSSMAgent PowerShell command.
    • On Linux, we can run sudo restart amazon-ssm-agent shell command.
  6. If all the above fails, it is time to get into the log files.
    • On Windows:
      • %PROGRAMDATA%\Amazon\SSM\Logs\amazon-ssm-agent.log
      • %PROGRAMDATA%\Amazon\SSM\Logs\errors.log
    • On Linux:
      • /var/log/amazon/ssm/amazon-ssm-agent.log
      • /var/log/amazon/ssm/errors.log
  7. If log files doesn't give enough information, we can enable debug logging which will give more information. This requires quite a number of steps, so refer to the reference link below.
Reference: 

Comments

Post a Comment

Popular posts from this blog

AWS EC2 Can't Reach EC2 Metadata Service After Subnet Change

Just another black box day. I had to move an EC2 instance to a different subnet, so I created an AMI out of it and launch it on a different subnet. Everything went well and it has no issue reaching the internet, but apparently not everything went well. The AWS agents such as SSM agent and CodeDeploy agent in the instance stop working. After checking the logs, they can't access the EC2 metadata. Since this is a Windows Server 2019 instance, it also shows that it is not activated, which is strange. On the following article, I found out that my issue was due to the "Gateway Address doesn't match that of the current subnet". https://aws.amazon.com/premiumsupport/knowledge-center/waiting-for-metadata/ Running the suggested command fixed the issue: Import-Module c:\ProgramData\Amazon\EC2-Windows\Launch\Module\Ec2Launch.psm1 ; Add-Routes
Read more

A2 Hosting with .NET Core 2.1

Technology advances so fast and I have a new web application which I built using .NET Core 2.1. However, my current web hosting provider does not support .NET Core and I have to look for a new web hosting. Reading some forums, found out that A2 Hosting plans to install .NET Core 2.1 in their shared hosting on August 2018. So on September 5th, 2018, I signed up for a new account with them. The coupon HOSTINGFACTS that I found from a web hosting review website still works and gave me 53% one-time discount. Everything went smoothly until I found out that in my Plesk, I was unable to go higher than .NET 4.6.2. So, I decided to chat with their customer support. It took a while to get in touch with an agent, which makes sense because they have lots of customers and it was during busy hour. From the chat with their customer support, I found out that .NET Core 2.1 has indeed been installed and what is weird is I don't have to flip the .NET version in Plesk. With that information, I d...
Read more

Xcode CodeSign Incorrectly States Password is Incorrect

I was trying to create an archive in Xcode to prepare the app for testing. It requires a code signing certificate and Xcode tried to reach out to Keychain to obtain it, so it prompted me for a password to my Keychain. My first thought is since the Keychain lives in my Mac, it has to be my Mac's password, so I entered it and it failed. I re-checked the characters, re-entered and it still failed. That is odd considering I did that before without issue. I found out that my Keychain Access app was still active. I have to quit the app and then it proceeds properly. So, the solution is quit the Keychain Access app before entering password for signing.
Read more