As IoT deployments grow, a set of distinct challenges arise when it comes application consistency across a broad spectrum of devices: efficiently managing update rollbacks, and maintaining comprehensive, auditable change logs can all present significant obstacles. However, these challenges can be effectively managed using AWS IoT Greengrass and the Greengrass Development Kit (GDK).
AWS IoT Greengrass enables the deployment of applications directly to your IoT devices, simplifying the management and control of device-side software. The GDK further enhances this capability by streamlining the configuration and packaging of these applications for deployment, making it easier to get your applications onto your devices.
To create a robust and efficient system, we can also introduce GitOps methodologies. GitOps leverages version control, automated deployments, and continuous monitoring to improve the reliability and efficiency of your deployment processes. By using these methodologies with GitHub Actions, we can automate the deployment process, triggering it with every commit or merge to a specified branch.
In this blog post, we'll explore how these technologies and methodologies can be combined to create a powerful, scalable, and automated IoT deployment system. We'll walk through the setup of the GDK, the definition and deployment of Greengrass components, and the setup of a GitHub Actions workflow to automate the entire process.
Let’s dive into and get started with this setup.
Overview
The AWS IoT Greengrass Development Kit Command-Line Interface (GDK CLI) is an open-source tool designed to streamline the creation, building, and publishing of custom Greengrass components. It simplifies the version management process, allows starting projects from templates or community components, and can be customized to meet specific development needs.
It can also be effectively utilized with GitHub Actions to automate the process of building and publishing Greengrass components. This could be a crucial part of a Continuous Integration/Continuous Deployment (CI/CD) pipeline. Here's a general outline of how this could be done:
-
Install Dependencies: Create a GitHub Actions workflow file (e.g.,
.github/workflows/main.yml
) and start with a job that sets up the necessary environment. This includes installing Python and pip (since GDK CLI is a Python tool), AWS CLI, and the GDK CLI itself. - Configure AWS Credentials: Use GitHub Secrets to securely store your AWS credentials (Access Key ID and Secret Access Key). In your workflow, configure AWS CLI with these credentials so that the GDK CLI can interact with your AWS account.
-
Build and Publish Components: Use GDK CLI commands in your workflow to build and publish your components. For example, you might have steps that run commands like
gdk component build
andgdk component publish
. - Integrate with Other Workflows: If you have other workflows in your CI/CD pipeline (such as running tests or deploying to other environments), you can use the output of the GDK CLI commands as inputs to these workflows.
In this way, every time you push a change to your Greengrass component source code on GitHub, the GDK CLI can automatically build and publish the updated component, ensuring that your Greengrass deployments are always using the latest version of your components.
Furthermore, we can initiate a Greengrass deployment based on the deployment template in the next stage of the pipeline, after a successful build and publish. This can target a specific Thing Group, enabling us to reflect changes across a fleet of devices. With this approach, if we have dev and test branches, each of these can be mapped to selected Thing Groups. This allows us to perform field validation on selected devices, providing an efficient way to test changes in a controlled environment before wider deployment.
For further details on creating GitHub Actions workflows, refer to the GitHub Actions documentation. For more information about using the GDK CLI, please refer to the GDK CLI documentation.
Development Project Setup
Based on the high level overview we can structure our project as such:
project
├── .github
│ └── workflows
│ └── main.yml
├── cfn
│ └── github-oidc
│ ├── oidc-provider.yaml
│ └── oidc-role.yaml
└── components
├── com.example.hello
│ ├── gdk-config.json
│ ├── main.py
│ └── recipe.yaml
├── com.example.world
│ ├── gdk-config.json
│ ├── main.py
│ └── recipe.yaml
└── deployment.json.template
Where we have the following:
-
.github/workflows/main.yml
: Which is the GitHub Actions workflow file where the CI/CD pipeline is defined. The GDK CLI and AWS CLI setup, component building, publishing, and deployment tasks are defined here. -
cfn/github-oidc
: This directory contains AWS CloudFormation templates (oidc-provider.yaml
andoidc-role.yaml
) that are used to set up an OIDC provider and role on AWS for authenticating GitHub Actions with AWS. -
components
: This directory contains the Greengrass components (com.example.hello
andcom.example.world
) that you are developing. Each component has its own directory with:-
gdk-config.json
: This is the configuration file for the GDK CLI for the specific component. -
main.py
: This is the main Python script file for the component's functionality. -
recipe.yaml
: This is the component recipe that describes the component and its dependencies, lifecycle scripts, etc.
-
-
deployment.json.template
: This is a deployment template file for Greengrass deployments. It is used to generate the actual deployment file (deployment.json
) that is used when initiating a Greengrass deployment
GitHub Actions and GDK Deployment Role Setup
The CloudFormation template will be used to create an IAM role (oidc-gdk-deployment
) that provides the necessary permissions for building and deploying Greengrass components using GDK CLI and GitHub Actions. The role has specific policies attached that allow actions such as describing and creating IoT jobs, interacting with an S3 bucket for Greengrass component artifacts, and creating Greengrass components and deployments.
AWSTemplateFormatVersion: 2010-09-09
Description: 'GitHub OIDC:| Stack: oidc'
Parameters:
FullRepoName:
Type: String
Default: example/gdk-example
Resources:
Role:
Type: AWS::IAM::Role
Properties:
RoleName: oidc-gdk-deployment
Policies:
- PolicyName: iot-thing-group
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- iot:DescribeThingGroup
- iot:CreateJob
Resource:
- !Sub arn:aws:iot:${AWS::Region}:${AWS::AccountId}:thinggroup/*
- PolicyName: iot-jobs
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- iot:DescribeJob
- iot:CreateJob
- iot:CancelJob
Resource:
- !Sub arn:aws:iot:${AWS::Region}:${AWS::AccountId}:job/*
- PolicyName: s3-greengrass-bucket
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- s3:CreateBucket
- s3:GetBucketLocation
- s3:ListBucket
Resource:
- !Sub arn:aws:s3:::greengrass-component-artifacts-${AWS::Region}-${AWS::AccountId}
- PolicyName: s3-greengrass-components
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- s3:GetObject
- s3:PutObject
Resource:
- !Sub arn:aws:s3:::greengrass-component-artifacts-${AWS::Region}-${AWS::AccountId}/*
- PolicyName: greengrass-components
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- greengrass:CreateComponentVersion
- greengrass:ListComponentVersions
Resource:
- !Sub arn:aws:greengrass:${AWS::Region}:${AWS::AccountId}:components:*
- PolicyName: greengrass-deployment
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- greengrass:CreateDeployment
Resource:
- !Sub arn:aws:greengrass:${AWS::Region}:${AWS::AccountId}:deployments
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Action: sts:AssumeRoleWithWebIdentity
Principal:
Federated: !Sub arn:aws:iam::${AWS::AccountId}:oidc-provider/token.actions.githubusercontent.com
Condition:
StringLike:
token.actions.githubusercontent.com:sub: !Sub repo:${FullRepoName}:*
Outputs:
OidcRoleAwsAccountId:
Value: !Ref AWS::AccountId
OidcRoleAwsRegion:
Value: !Ref AWS::Region
OidcRoleAwsRoleToAssume:
Value: !GetAtt Role.Arn
The FullRepoName
parameter is used to specify the repository that the GitHub Actions workflow will be running in. This is important for the sts:AssumeRoleWithWebIdentity
action in the AssumeRolePolicyDocument
, which allows GitHub Actions to assume this IAM role for the specified repository.
To deploy this CloudFormation stack, you would use the AWS Management Console, AWS CLI, or an AWS SDK. You would need to specify the FullRepoName
parameter as an input when you create the stack. For example, with the AWS CLI, you would use the aws cloudformation deply
command and provide the template file and the FullRepoName
parameter as inputs:
aws cloudformation deploy \
--template-file cfn/github-oidc/oidc-role.yaml \
--stack-name ga-gdk-role \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides FullRepoName=<your org>/<your repo name>
This setup lays the foundation for the GitHub Actions and GDK CLI to work together to automate the building and deployment of Greengrass components.
In scenario that you would like to use OIDC Provider for GitHub actions (suggested) you would also need to set it up in you AWS account. Please not that this is needed only once per account region:
aws cloudformation deploy \
--template-file cfn/github-oidc/oidc-provider.yaml \
--stack-name oidc-provider
Now we can go and prepare our Greengrass components.
Greengrass Components GDK Setup
Here we focus here on the configuration and implementation of our Greengrass components. These components form the core of our IoT solution, providing the required functionality on our Greengrass core devices.
The gdk-config.json
file is where we configure our Greengrass component.
{
"component": {
"com.example.hello": {
"author": "Example",
"version": "NEXT_PATCH",
"build": {
"build_system": "zip"
},
"publish": {
"bucket": "greengrass-component-artifacts",
"region": "eu-west-1"
}
}
},
"gdk_version": "1.2.0"
}
In the provided example, we have a single component named "com.example.hello". This file specifies the author, the build system (which is set to "zip" here), and the AWS S3 bucket details where the component will be published. The version field is set to "NEXT_PATCH", which means GDK will automatically increment the patch version of the component every time it's built.
The recipe.yaml
file is the recipe for our Greengrass component.
---
RecipeFormatVersion: "2020-01-25"
ComponentName: "{COMPONENT_NAME}"
ComponentVersion: "{COMPONENT_VERSION}"
ComponentDescription: "This is a simple Hello World component written in Python."
ComponentPublisher: "{COMPONENT_AUTHOR}"
ComponentConfiguration:
DefaultConfiguration:
Message: "Hello"
Manifests:
- Platform:
os: all
Artifacts:
- URI: "s3://BUCKET_NAME/COMPONENT_NAME/COMPONENT_VERSION/com.example.hello.zip"
Unarchive: ZIP
Lifecycle:
Run: "python3 -u {artifacts:decompressedPath}/com.example.hello/main.py {configuration:/Message}"
It contains essential metadata about the component, like its name, version, description, and publisher. It also specifies the default configuration, which, in this case, sets the default message to "Hello". The Manifests section describes the component's artifacts and the lifecycle of the component. In this instance, it specifies the location of the component's zipped artifacts and the command to run the component.
Finally, the main.py
file is the Python script that our Greengrass component runs.
import sys
message = f"Hello, {sys.argv[1]}!"
# Print the message to stdout, which Greengrass saves in a log file.
print(message)
This script simply prints out a greeting message. The message is constructed from the argument passed to the script, which comes from the component configuration in the recipe.yaml
file. This setup demonstrates how you can pass configuration to your Greengrass components, as we can see in the “com.example.world” example where we provide “World” as a configuration message.
In addition to the component configuration and scripts, we also define a deployment.json.template
file.
{
"targetArn": "arn:aws:iot:$AWS_REGION:$AWS_ACCOUNT_ID:thinggroup/$THING_GROUP",
"deploymentName": "Main deployment",
"components": {
"com.example.hello": {
"componentVersion": "LATEST",
"runWith": {}
},
"com.example.world": {
"componentVersion": "LATEST",
"runWith": {}
}
},
"deploymentPolicies": {
"failureHandlingPolicy": "ROLLBACK",
"componentUpdatePolicy": {
"timeoutInSeconds": 60,
"action": "NOTIFY_COMPONENTS"
},
"configurationValidationPolicy": {
"timeoutInSeconds": 60
}
},
"iotJobConfiguration": {}
}
This file specifies the deployment configuration for our Greengrass components. The targetArn
is the Amazon Resource Name (ARN) of the Greengrass thing group where we aim to deploy our components. In this example, we're deploying two components, com.example.hello
and com.example.world
, both set to use their latest versions. The deploymentPolicies
section sets the policies for failure handling, component update, and configuration validation. This file is vital as it governs how the deployment of our Greengrass components is handled in the target IoT devices.
Please note that this is a template and we will be using this in our pipeline to replace the ARN accordingly.
Taken together, these files form the basis of a Greengrass component. By modifying these templates and scripts, you can create your own custom Greengrass components with GDK. The next step is to set up GitHub Actions to automate the build and deployment of these components.
GitHub Actions Setup
This GitHub workflow file sets up two jobs, namely publish
and deploy
, which are run when either a push to the main
branch occurs or the workflow is manually dispatched.
1. Publish Job
The publish
job runs on the ubuntu-latest
environment.
jobs:
publish:
name: Component publish
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
ref: ${{ github.head_ref }}
- uses: actions/setup-python@v3
with:
python-version: '3.9'
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: ${{ secrets.OIDC_ROLE_AWS_ROLE_TO_ASSUME }}
aws-region: ${{ secrets.OIDC_ROLE_AWS_REGION }}
- name: Install Greengrass Development Kit
run: pip install -U git+https://github.com/aws-greengrass/aws-greengrass-gdk-cli.git@v1.2.3
- name: GDK Build and Publish
id: build_publish
run: |
CHANGED_COMPONENTS=$(git diff --name-only HEAD~1 HEAD | grep "^components/" | cut -d '/' -f 2)
echo "Components changed -> $CHANGED_COMPONENTS"
for component in $CHANGED_COMPONENTS
do
cd $component
echo "Building $component ..."
gdk component build
echo "Publishing $component ..."
gdk component publish
cd ..
done
working-directory: ${{ env.working-directory }}
We start with checking out the code from the repository and setting up Python 3.9. The AWS credentials are then configured using the aws-actions/configure-aws-credentials@v1
action. The credentials used here are fetched from the GitHub secrets OIDC_ROLE_AWS_ROLE_TO_ASSUME
and OIDC_ROLE_AWS_REGION
. The OIDC_ROLE_AWS_ROLE_TO_ASSUME
secret should contain the ARN of the AWS role that the GitHub Actions should assume when executing the workflow. This is the role we created in the firs step we can obtain it by executing the following:
aws cloudformation describe-stacks --stack-name ga-gdk-role --query 'Stacks[0].Outputs[?OutputKey==`OidcRoleAwsRoleToAssume`].OutputValue' --output text
The OIDC_ROLE_AWS_REGION
secret should contain the AWS region where your resources are located. After that these variables needs to be added under github.com/<org>/<repo>/settings/secrets/actions
.
Next, the Greengrass Development Kit (GDK) is installed using pip. The GDK CLI is used to build and publish any components that have changed between the current and previous commit.
The changed components are identified by looking at the differences between the current and previous commit and extracting the component names. Here it is important that the name of the folder mathces the name of the component.
2. Deploy Job
The deploy
job runs after the publish
job has completed successfully.
deploy:
name: Component deploy
runs-on: ubuntu-latest
needs: publish
permissions:
id-token: write
contents: read
steps:
- name: Checkout
uses: actions/checkout@v3
with:
fetch-depth: 0
ref: ${{ github.head_ref }}
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: ${{ secrets.OIDC_ROLE_AWS_ROLE_TO_ASSUME }}
aws-region: ${{ secrets.OIDC_ROLE_AWS_REGION }}
- name: Deploy Greengrass components
run: |
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity | jq -r '.Account')
export AWS_REGION=${GREENGRASS_REGION}
# Thing Group is the name of the branch
export THING_GROUP=${GITHUB_REF#refs/heads/}
CHANGED_COMPONENTS=$(git diff --name-only HEAD~1 HEAD | grep "^components/" | cut -d '/' -f 2)
if [ -z "$CHANGED_COMPONENTS" ]; then
echo "No need to update deployment"
else
envsubst < "deployment.json.template" > "deployment.json"
for component in $CHANGED_COMPONENTS
do
version=$(aws greengrassv2 list-component-versions \
--output text \
--no-paginate \
--arn arn:aws:greengrass:${AWS_REGION}:${AWS_ACCOUNT_ID}:components:${component} \
--query 'componentVersions[0].componentVersion')
jq '.components[$component].componentVersion = $version' --arg component $component --arg version $version deployment.json > "tmp" && mv "tmp" deployment.json
done
# deploy
aws greengrassv2 create-deployment \
--cli-input-json file://deployment.json \
--region ${AWS_REGION}
echo "Deployment finished!"
fi
working-directory: ${{ env.working-directory }}
It follows a similar structure to the publish
job, starting with checking out the code from the repository and configuring AWS credentials using the same secrets.
In the deployment step, it first identifies any changed components in a similar way as in the publish
job. If no components have changed, it does not proceed with deployment. If there are changed components, it prepares a deployment.json
file from the template, replacing placeholders with the actual values. It then gets the version of the changed components from AWS Greengrass and updates the deployment.json
file with these versions.
Finally, it creates a deployment using the aws greengrassv2 create-deployment
command, providing the deployment.json
file as input and setting the region to the one specified in the AWS_REGION
environment variable.
Here it is important to note that Thing Group is taken from the name of the branch THING_GROUP=${GITHUB_REF#refs/heads/}
as that way we can have different branches related to different Thing Groups as discussed above. In case the thing group is not create you can use the following command:
aws iot create-thing-group --thing-group-name main
At this point, when ever there is a new commit on the main branch a process will kick start and issue a deployment to the specified group of devices.
Conclusion
In this blog post, we've taken a deep dive into AWS IoT Greengrass V2, focusing on the usage of the Greengrass Development Kit (GDK) and how it can be automated with GitHub Actions. We started by setting up the GDK, explored the various components and how they interact, then we moved on to setting up a GitHub Actions workflow to automate the entire process.
By leveraging AWS IoT Greengrass, the GDK, and GitHub Actions, you can create a powerful, scalable, and automated IoT solution. Whether you're managing a small group of IoT devices or a large fleet, this approach offers a robust and efficient way to handle your IoT application deployments.
That's all for this blog post. We hope you found it informative and that it helps you on your journey to creating and managing IoT solutions with AWS IoT Greengrass. Happy coding!
All the above code and setup can be referenced here:
https://github.com/aws-iot-builder-tools/greengrass-continuous-deployments
If you have any feedback about this post, or you would like to see more related content, please reach out to me here, or on Twitter or LinkedIn.
Top comments (0)