Understand Amazon SSM Agent In 2 Minutes

#aws #ssm #devops #cloudopz

🚀 Install SSM Agent on Ubuntu Server instances

To install SSM Agent on Ubuntu Server 20.10 STR & 20.04, 18.04, and 16.04 LTS 64-bit instances (with Snap package)

~ $:/home/ubuntu# sudo snap install amazon-ssm-agent --classic

🚀 Check SSM Agent log

~ $:/home/ubuntu# systemctl restart snap.amazon-ssm-agent.amazon-ssm-agent.service                                                                                                                                                                                                 
~ $:/home/ubuntu# tail -f /var/log/amazon/ssm/amazon-ssm-agent.log                                                                                                                                                                                                                 
        status code: 400, request id: ea74ed4f-70d4-4610-8221-ce7868c3c9fb                                                                                                                                                                                                                
2021-01-08 08:40:20 INFO [amazon-ssm-agent] [SelfUpdate] Initializing self update ...
2021-01-08 08:40:20 INFO [amazon-ssm-agent] Starting Core Agent: amazon-ssm-agent - v3.0.161.0
2021-01-08 08:40:20 INFO [amazon-ssm-agent] OS: linux, Arch: amd64
2021-01-08 08:40:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker is not running, starting worker process
2021-01-08 08:40:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker (pid:11067) started
2021-01-08 08:40:22 ERROR Error adding the directory to watcher: no such file or directory
2021-01-08 08:40:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Monitor long running worker health every 60 seconds
2021-01-08 08:40:22 INFO [ssm-agent-worker] Dial to Core Agent broadcast channel
2021-01-08 08:40:22 INFO [ssm-agent-worker] Dial to Core Agent broadcast channel
2021-01-08 08:40:22 INFO [ssm-agent-worker] Create new startup processor
2021-01-08 08:40:22 INFO [ssm-agent-worker] Start to listen to Core Agent health channel
2021-01-08 08:40:22 INFO [ssm-agent-worker] Start to listen to Core Agent termination channel
2021-01-08 08:40:22 INFO [ssm-agent-worker] [StartupProcessor] Executing startup processor tasks
2021-01-08 08:40:22 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: Amazon SSM Agent v3.0.161.0 is running
2021-01-08 08:40:22 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: OsProductName: Ubuntu
2021-01-08 08:40:22 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: OsVersion: 18.04
2021-01-08 08:40:23 INFO [ssm-agent-worker] Entering SSM Agent hibernate - AccessDeniedException: User: arn:aws:sts::111111111111:assumed-role/SSMAutomation/i-06f3424a03d04c66d is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:eu-central-1:111111111111:instance/i-06f3424a03d04c66d
        status code: 400, request id: f0580f42-a0c8-4038-8242-46e4818a391b

It shows that the SSM agent does not have permission on SSM action such as UpdateInstanceInformation for the instance that it relies on

2021-01-08 08:40:23 INFO [ssm-agent-worker] Entering SSM Agent hibernate - AccessDeniedException: User: arn:aws:sts::111111111111:assumed-role/SSMAutomation/i-06f3424a03d04c66d is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:eu-central-1:111111111111:instance/i-06f3424a03d04c66d

🚀 Attach instance profile for ssm agent permission

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AmazonSSMtoEC2",
            "Effect": "Allow",
            "Action": [
                "ssm:*",
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel",
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        }
    ]
}

Restart the ssm agent for the role affects

2021-01-08 08:44:27 INFO [ssm-agent-worker] Starting SSM Agent Worker: amazon-ssm-agent - v3.0.161.0
2021-01-08 08:44:27 INFO [ssm-agent-worker] OS: linux, Arch: amd64
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] Starting document processing engine...
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] Starting document processing engine...
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] [EngineProcessor] Starting
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] [EngineProcessor] Initial processing
2021-01-08 08:44:27 INFO [ssm-agent-worker] [HealthCheck] HealthCheck reporting agent health.
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] Starting message polling
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] Starting send replies to MDS
2021-01-08 08:44:27 INFO [ssm-agent-worker] [LongRunningPluginsManager] starting long running plugin manager
2021-01-08 08:44:27 INFO [ssm-agent-worker] [LongRunningPluginsManager] there aren't any long running plugin to execute
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [EngineProcessor] Starting
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [EngineProcessor] Initial processing
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] Starting message polling
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] Starting send replies to MDS
2021-01-08 08:44:27 INFO [ssm-agent-worker] [instanceID=i-06f3424a03d04c66d] Starting association polling
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] [EngineProcessor] Starting
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] Launching response handler
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] [EngineProcessor] Initial processing
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] Initializing association scheduling service
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] Association scheduling service initialized
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] Starting session document processing engine...
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] [EngineProcessor] Starting
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] SSM Agent is trying to setup control channel for Session Manager module.
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] agent telemetry cloudwatch metrics disabled
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] Setting up websocket for controlchannel for instance: i-06f3424a03d04c66d, requestId: 8ad8e181-fff5-4aee-81ca-4bbaf6542c3e
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] listening reply.

🚀 Conclusion

For using AWS Systems Manager Run Command, not only set IAM policy for the instance sending SSM command but also the target need IAM policy so that its SSM agent can assume EC2 resource

Mirror

https://github.com/vumdao/ssm-agent

Read More

🌠 Blog · Web · Linkedin · Group · Page · Twitter 🌠

Top comments (10)

Oleksiy Holubyev • Mar 3 '21

Hmm, I have exactly this problem, but I don't understand where I should attach the instance profile. My EC2 instance already has AmazonSSMRoleForInstanceQuickSetup role attached, which has the AmazonSSMManagedInstanceCore policy, which should have all of the listed permissions. But I still get the error mentioned above.

🚀 Vu Dao 🚀 • Mar 4 '21

Attache instance profile which provide ssm:* permisson to the EC2 instance which you got the ssm-agent error

Oleksiy Holubyev • Mar 4 '21

I have that. But I still get the error.

🚀 Vu Dao 🚀 • Mar 4 '21

Can you show you policy and the log

Oleksiy Holubyev • Mar 4 '21

Sure. The role in the instance profile is AmazonSSMRoleForInstancesQuickSetup, the associated policy is arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:DescribeAssociation",
"ssm:GetDeployablePatchSnapshotForInstance",
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetManifest",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:ListAssociations",
"ssm:ListInstanceAssociations",
"ssm:PutInventory",
"ssm:PutComplianceItems",
"ssm:PutConfigurePackageResult",
"ssm:UpdateAssociationStatus",
"ssm:UpdateInstanceAssociationStatus",
"ssm:UpdateInstanceInformation"
],
"Resource": ""
},
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
],
"Resource": ""
},
{
"Effect": "Allow",
"Action": [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
],
"Resource": "*"
}
]
}

and the error is:

Entering SSM Agent hibernate - AccessDeniedException: User: arn:aws:sts::111111111111:assumed-role/AmazonSSMRoleForInstancesQuickSetup/i-instance-id
is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:eu-central-1:111111111111:instance/i-instance-id
status code: 400, request id: