DEV Community

Cover image for Using AWS S3 to Store DevStream State
DevStream profile image Tiexin Guo
Tiexin Guo for DevStream

Posted on • Originally published at blog.devstream.io

Using AWS S3 to Store DevStream State

#devops #devstream #aws #s3

In our latest release v0.6.0, using AWS S3 to store DevStream state is supported.

In this blog, we are going to demonstrate the usage of AWS S3 to store DevStream's state files.

Terminology

State: if you don't already know about DevStream state, please read this doc and this blog helps, too.

Backend: where to actually store the state file. It can be either local or s3 at the moment.

Configuration

In the main config file, we have a section to configure the state. Currently, local and S3 are supported.

Local example:

varFile: variables-gitops.yaml

toolFile: tools-gitops.yaml

state:
  backend: local
  options:
    stateFile: devstream.state
Enter fullscreen mode Exit fullscreen mode

S3 example:

varFile: variables-gitops.yaml

toolFile: tools-gitops.yaml

state:
  backend: s3
  options:
    bucket: devstream-remote-state
    region: ap-southeast-1
    key: devstream.state
Enter fullscreen mode Exit fullscreen mode

More on configuring state here.

In short, we can use the "backend" keyword to specify where to store the state: either locally or in an S3 bucket. If S3 is used, we need to specify the bucket, region, and the S3 key as well.

Config File Examples

In this demo, we use the following configs:

config.yaml:

varFile: variables-gitops.yaml

toolFile: tools-gitops.yaml

state:
  backend: s3
  options:
    bucket: devstream-test-remote-state
    region: ap-southeast-1
    key: devstream.state
Enter fullscreen mode Exit fullscreen mode

variables-gitops.yaml:

githubUsername: IronCore864
repoName: dtm-test-go
defaultBranch: main

dockerhubUsername: ironcore864

argocdNameSpace: argocd
argocdDeployTimeout: 5m
Enter fullscreen mode Exit fullscreen mode

tools-gitops.yaml:

tools:
- name: github-repo-scaffolding-golang
  instanceID: default
  options:
    owner: [[ githubUsername ]]
    org: ""
    repo: [[ repoName ]]
    branch: [[ defaultBranch ]]
    image_repo: [[ dockerhubUsername ]]/[[ repoName ]]
Enter fullscreen mode Exit fullscreen mode

Getting Started

Before reading on, now is a good time to check if you have configured your AWS related environment variables correctly or not.

For macOS/Linux users, do:

export AWS_ACCESS_KEY_ID=ID_HERE
export AWS_SECRET_ACCESS_KEY=SECRET_HERE
export AWS_DEFAULT_REGION=REGION_HERE
Enter fullscreen mode Exit fullscreen mode

For more information, see the official document here.

Apply

Then let's run dtm apply:

tiexin@mbp ~/work/devstream-io/devstream $ ./dtm apply
2022-05-30 17:07:59 ℹ [INFO]  Apply started.
2022-05-30 17:07:59 ℹ [INFO]  Using dir <.devstream> to store plugins.
2022-05-30 17:07:59 ℹ [INFO]  Using s3 backend. Bucket: devstream-test-remote-state, region: ap-southeast-1, key: devstream.state.
2022-05-30 17:08:00 ℹ [INFO]  Tool (github-repo-scaffolding-golang/default) found in config but doesn't exist in the state, will be created.
Continue? [y/n]
Enter a value (Default is n): y

2022-05-30 17:08:08 ℹ [INFO]  Start executing the plan.
2022-05-30 17:08:08 ℹ [INFO]  Changes count: 1.
2022-05-30 17:08:08 ℹ [INFO]  -------------------- [  Processing progress: 1/1.  ] --------------------
2022-05-30 17:08:08 ℹ [INFO]  Processing: (github-repo-scaffolding-golang/default) -> Create ...
2022-05-30 17:08:12 ℹ [INFO]  The repo dtm-test-go has been created.
2022-05-30 17:08:29 ✔ [SUCCESS]  Tool (github-repo-scaffolding-golang/default) Create done.
2022-05-30 17:08:29 ℹ [INFO]  -------------------- [  Processing done.  ] --------------------
2022-05-30 17:08:29 ✔ [SUCCESS]  All plugins applied successfully.
2022-05-30 17:08:29 ✔ [SUCCESS]  Apply finished.
Enter fullscreen mode Exit fullscreen mode

As we can see from the output, the S3 backend is used, and it also shows the bucket and key you are using, and in which region this bucket lives.

Checking the State File

After apply, let's download the state file from S3 and check it out:

tiexin@mbp ~/work/devstream-io/devstream $ aws s3 cp s3://devstream-test-remote-state/devstream.state .
Enter fullscreen mode Exit fullscreen mode

And if we open the downloaded file, we will see something similar to the following:

github-repo-scaffolding-golang_default:
  name: github-repo-scaffolding-golang
  instanceid: default
  dependson: []
  options:
    branch: main
    image_repo: ironcore864/dtm-test-go
    org: ""
    owner: IronCore864
    repo: dtm-test-go
  resource:
    org: ""
    outputs:
      org: ""
      owner: IronCore864
      repo: dtm-test-go
      repoURL: https://github.com/IronCore864/dtm-test-go.git
    owner: IronCore864
    repoName: dtm-test-go
Enter fullscreen mode Exit fullscreen mode

which is exactly the same as if we were using the local backend to store state.

Like this quick tutorial? Then I suggest to read more of our latest DevOps blogs:

Top comments (0)

Read next

notharshhaa profile image

100 AWS DevOps Interview Questions and Answers

H A R S H H A A -

lovestaco profile image

Why We Chose NGINX + HashiStack Over Kubernetes for Our Service Discovery Needs

Athreya aka Maneshwar -

rootstack profile image

Tools for effective DataOps implementation

Rootstack -

gkarwchan profile image

Enhancing Your Git Workflow with Git Credential Manager Core

Ghassan Karwchan -