DEV Community

steve
steve

Posted on

How to debug Lambda to ec2 - AWS policy?

#aws #lambda #roles #policies

Example lambda extract

This lambda listens to cloudwatch log and then triggers s3 sync to an ec2 instance id

const sendCommandParams: SendCommandCommandInput = {
    DocumentName: "AWS-RunShellScript",
    Parameters: {
      commands: [`aws s3 sync s3://${BUCKET_NAME} ${DOCKER_VOLUME_PATH} --delete --no-progress`],
    },
    InstanceIds: [ec2InstanceId],
  };
  const sendCommandCommand = new SendCommandCommand(sendCommandParams);
  const response = await ssmClient.send(sendCommandCommand);
Enter fullscreen mode Exit fullscreen mode

Cloudwatch error

ERROR Error: AccessDeniedException: User: arn:aws:sts::$accountID:assumed-role/role-lambdaEC2okOK/lambdaEC2okOK is not authorized to perform: ssm:SendCommand on resource: arn:aws:ssm:$REGION::document/AWS-RunShellScript because no identity-based policy allows the ssm:SendCommand action

EC2 instance error

  • ssh into ec2 instance - debug ssm agent logs

sudo tail -f /var/log/amazon/ssm/amazon-ssm-agent.log

024-11-17 07:59:50.4887 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] SSM Connection channel status is set to ec2messages
2024-11-17 07:59:50.4887 ERROR [ssm-agent-worker] [MessageService] [MGSInteractor] Failed to get controlchannel token, error: CreateControlChannel failed with error: createControlChannel request failed: unexpected response from the service
User: arn:aws:sts::$accountID:assumed-role/role-ec2-instance/$instanceID is not authorized to perform: ssmmessages:CreateControlChannel on resource: arn:aws:ec2:$REGION:$accountID:instance/$instanceID because no identity-based policy allows the ssmmessages:CreateControlChannel action

Add Missing policy on ec2 policy

  • smmessages:CreateControlChannel 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:UpdateInstanceInformation"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2messages:AcknowledgeMessage",
        "ec2messages:DeleteMessage",
        "ec2messages:FailMessage",
        "ec2messages:GetEndpoint",
        "ec2messages:GetMessages",
        "ec2messages:SendReply"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}",
        "arn:aws:s3:::${BUCKET_NAME}/*"
      ]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Update Lambda Policy

Fail 
{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:document/AWS-RunShellScript",
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }
Enter fullscreen mode Exit fullscreen mode
Fail 
{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:document/AWS-RunShellScript",
        "arn:aws:ssm:${REGION}:${accountID}:managed-instance/*",
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }
Enter fullscreen mode Exit fullscreen mode
Fail 
{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }
Enter fullscreen mode Exit fullscreen mode
Fail 
{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:document/AWS-RunShellScript"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }
Enter fullscreen mode Exit fullscreen mode
Fail 
{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:document/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }
Enter fullscreen mode Exit fullscreen mode
Fail 
{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
         "*:*:*:${REGION}:${accountID}:*"
      ]
    }
Enter fullscreen mode Exit fullscreen mode
Fail 
{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:*/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ec2:${REGION}:${accountID}:*/*"
      ]
    },
Enter fullscreen mode Exit fullscreen mode
Success - Unfortunately can't restrict any more 
{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "*"
      ]
    }
Enter fullscreen mode Exit fullscreen mode
Complete policy 
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2messages:AcknowledgeMessage",
        "ec2messages:DeleteMessage",
        "ec2messages:FailMessage",
        "ec2messages:GetEndpoint",
        "ec2messages:GetMessages",
        "ec2messages:SendReply"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetDocument",
        "ssm:GetCommandInvocation",
        "ssm:ListAssociations",
        "ssm:ListInstanceAssociations",
        "ssm:DescribeInstanceInformation"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}",
        "arn:aws:s3:::${BUCKET_NAME}/*"
      ]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Gotchas

The odd bug - when the ec2 instance isn't ready - despite it passing system status check:

ec2 describe-instance-status

  • SystemStatus.Status = "ok"

  • InstanceStatus.Status= "ok"

    InvalidInstanceId: Instances [[i-044df8615ccf4805c]] not in a valid state for account $baccountID

  • Inside Lambda allow an additional 500ms if error?
    Must separate Arns for SSM
    Some policies cannot be restricted on Resource - ie 

  • ec2messages:*

  • ssm:* except ssm:SendCommand

Notes:

Cloud watch logs don't always update with timestamps

  • Go into last cloudwatch log stream and refresh

Lambda needs additional basic permission:

  • arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Need both:*

  • ec2 instance policy to allow ssm communication 
  • Lambda policy policy to allow ssm communication

Ec2 instance security update to allow ssm communication

aws ec2 authorize-security-group-ingress \
    --group-id $securityGroupID \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0 \
    --query="SecurityGroupRules[].SecurityGroupRuleId"
Enter fullscreen mode Exit fullscreen mode

Test lambda on local computer AND in situ, ie when triggered by cloudwatch log or other because it may work in one but not the other

If invoking lambda via cloudwatch log - then lambda needs additional permission:

aws lambda add-permission \
 --function-name $LAMBDA_FUNCTION_EC2_OKOK \
 --statement-id AllowCloudWatchLogsInvoke \
 --action "lambda:InvokeFunction" \
 --principal logs.amazonaws.com \
 --source-arn "arn:aws:logs:${REGION}:${accountID}:log-group:${project_repo}:*"
Enter fullscreen mode Exit fullscreen mode

Make sure to reapply that permission if re-creating lambda from aws cli command

Be sure to add environment variables

aws lambda create-function \
 --function-name $LAMBDA_FUNCTION_EC2_OKOK \
 --zip-file fileb://typescript/lambda-function.zip \
 --handler "index.handler" \
 --runtime nodejs18.x \
 --role $roleArn \
 --environment "Variables={project_repo=\"$project_repo\",REGION=\"$REGION\",LAMBDA_FUNCTION_EC2_OKOK=\"$LAMBDA_FUNCTION_EC2_OKOK\", BUCKET_NAME=\"$BUCKET_NAME\",DOCKER_VOLUME_PATH=\"$DOCKER_VOLUME_PATH\"}" \
 --query "FunctionArn"
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

lindiwe09 profile image

Amazon SQS: The Backbone of Asynchronous Communication

Lindiwe -

er_dward profile image

Karpenter to EKS Auto Mode, worth it?

Internet Explorer -

imsushant12 profile image

Conclusion of My Node.js Journey and a Sneak Peek into My Upcoming AWS Series

Sushant Gaurav -

vng_bach profile image

AWS App Studio

Vuong Bach Doan -