In this series, we saw how to create and manage our secrets in git and restrict their access to team members.
What if I want do use my secret from my CI for Continuous Deployment? This is exactly what we will see in this article, based on GitLab CI π¦.
DISCLAIMER: Every CI system should be compatible, you just need to have access to CI secrets and be able to fetch them during execution.
π€ Creation of the CI identity
To be able to decrypt secrets within Continuous Integration, we should create an identity for our CI. Alice, Bobby and Devon will have a new friend named CI π€ !
Devon, in charge of the CI, will create a new key-pair for this new special user. He will use the gpg --full-gen-key command and answer questions like if it was for a real human.
Then, he will extract a backup of the private/public key with the command gpg -o ci.key --armor --export-secret-keys ci@domain.fr. This key is in a textual format, which is easy to use as an environment variable.
NOTE: This key and the passphrase should be stored securely, in a keypass, vault or something else, because they represent the identity of your CI. By design, it will be able to decrypt all secrets from the repository.
We also need to extract the public key separately to distribute it to every team members. Without it, they won't be able to encrypt a secret and include the CI key in the process.
To do so, Devon will use the command gpg -o ci.public.key --armor --export.
π Import public key of CI
Each team member should import the public key extracted by Devon in the previous step. Here, Booby will use the gpg --import ci.public.key command to perform this.
At the end, he, and every other team members should have in their key list (gpg --list-keys) the public key of CI.
π Update keys of every secret
CI is a new team member, as any other human. The team has to let this new "person" access and decrypt every secret. So we will use the sops updatekeys <secret> on every secret.
This requires a correctly configured .sops.yaml file. For more information about this, you can read the second part of this series.
π¦ GitLab CI Configuration
Now, the CI is ready to be configured. Devon, in charge of this, will import into GitLab secret pane (in Settings > CI/CD > Variables) the content of the ci.key created previously and the passphrase defined for it.
NOTE: The variable Type is defined to File for the KEY. For more information about this, look at the official GitLab documentation about file variable
Then, Devon will define a GitLab CI job able to read the value from the encrypted secret. For our example, this will just do a cat on the standard output. Of course, you can do what you want with those values in your CI pipeline.
We need to take care of the image used in our Continuous Integration. We should be able to access the gpg command. In this example, we will install it.
deploy int:
image: google/cloud-sdk # <1>
before_script:
- apt-get update && apt-get install -y curl gnupg # <2>
- curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /usr/local/bin/sops # <3>
- chmod +x /usr/local/bin/sops
- cat $KEY | gpg --batch --import # <4>
- echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp) # <5>
script:
- sops -d int.encrypted.env > int.env # <6>
- cat int.env
- Define a job image based required for our deploy phase... it should fit team needs, this is just for example here.
- We install
gpgandcurlthanks toapton debian distribution. Should be adapted if you are using another distribution. - We use
curlto downloadsopsbinary from GitHub. Take care to get the latest version π. - Import the
ci.keyinto ourgpgtoolchain. - Provide the
PASSPHRASEto thegpgtoolchain to be totally "not interactive". - Then, the CI can decrypt the
int.encrypted.envfile and use it where we want in our deploy phase.
Running with gitlab-runner 13.0.0 (c127439c)
on docker-auto-scale 0277ea0f
Preparing the "docker+machine" executor
Using Docker executor with image google/cloud-sdk ...
Pulling docker image google/cloud-sdk ...
Using docker image sha256:5d096c48b3b4bab72df240dcf94940be3e0397606bb4cc94143f2a12189dba1f for google/cloud-sdk ...
Preparing environment
Running on runner-0277ea0f-project-19225532-concurrent-0 via runner-0277ea0f-srm-1591443365-4e9ad7c3...
Getting source from Git repository
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/davinkevin/sops-blog-post-repository/.git/
Created fresh repository.
From https://gitlab.com/davinkevin/sops-blog-post-repository
* [new ref] refs/pipelines/153546622 -> refs/pipelines/153546622
* [new branch] master -> origin/master
Checking out 84edc518 as master...
Skipping Git submodules setup
Restoring cache
00:02
Downloading artifacts
00:01
Running before_script and script
$ apt-get update && apt-get install -y curl gnupg
Hit:1 http://deb.debian.org/debian buster InRelease
Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
Get:3 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Hit:4 https://packages.cloud.google.com/apt cloud-sdk-buster InRelease
Get:5 http://deb.debian.org/debian sid InRelease [146 kB]
Get:6 http://security.debian.org/debian-security buster/updates/main amd64 Packages [201 kB]
Get:7 http://deb.debian.org/debian sid/main amd64 Packages.diff/Index [27.9 kB]
Get:8 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-05-2003.49.pdiff [23.8 kB]
Get:9 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0208.52.pdiff [16.2 kB]
Get:10 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0803.34.pdiff [12.5 kB]
Get:10 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0803.34.pdiff [12.5 kB]
Fetched 543 kB in 2s (243 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
dh-python libperl5.28 libpython3.7 libpython3.7-minimal libpython3.7-stdlib
perl-modules-5.28 python3.7-minimal
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
dirmngr gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server
gpgconf gpgsm gpgv libbrotli1 libcurl4
Suggested packages:
dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie xloadimage
scdaemon
The following NEW packages will be installed:
libbrotli1
The following packages will be upgraded:
curl dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client
gpg-wks-server gpgconf gpgsm gpgv libcurl4
13 upgraded, 1 newly installed, 0 to remove and 132 not upgraded.
Need to get 8559 kB of archives.
After this operation, 1241 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian sid/main amd64 gpg-wks-client amd64 2.2.20-1 [507 kB]
Get:2 http://deb.debian.org/debian sid/main amd64 dirmngr amd64 2.2.20-1 [740 kB]
Get:3 http://deb.debian.org/debian sid/main amd64 gnupg-utils amd64 2.2.20-1 [889 kB]
Get:4 http://deb.debian.org/debian sid/main amd64 gpg-wks-server amd64 2.2.20-1 [500 kB]
Get:5 http://deb.debian.org/debian sid/main amd64 gpg-agent amd64 2.2.20-1 [641 kB]
Get:6 http://deb.debian.org/debian sid/main amd64 gpg amd64 2.2.20-1 [894 kB]
Get:7 http://deb.debian.org/debian sid/main amd64 gpgconf amd64 2.2.20-1 [532 kB]
Get:8 http://deb.debian.org/debian sid/main amd64 gnupg-l10n all 2.2.20-1 [1035 kB]
Get:9 http://deb.debian.org/debian sid/main amd64 gnupg all 2.2.20-1 [749 kB]
Get:10 http://deb.debian.org/debian sid/main amd64 gpgsm amd64 2.2.20-1 [627 kB]
Get:11 http://deb.debian.org/debian sid/main amd64 gpgv amd64 2.2.20-1 [608 kB]
Get:12 http://deb.debian.org/debian sid/main amd64 libbrotli1 amd64 1.0.7-6.1 [267 kB]
Get:13 http://deb.debian.org/debian sid/main amd64 curl amd64 7.68.0-1 [249 kB]
Get:14 http://deb.debian.org/debian sid/main amd64 libcurl4 amd64 7.68.0-1 [321 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 8559 kB in 0s (22.3 MB/s)
(Reading database ... 85863 files and directories currently installed.)
Preparing to unpack .../00-gpg-wks-client_2.2.20-1_amd64.deb ...
Unpacking gpg-wks-client (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../01-dirmngr_2.2.20-1_amd64.deb ...
Unpacking dirmngr (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../02-gnupg-utils_2.2.20-1_amd64.deb ...
Unpacking gnupg-utils (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../03-gpg-wks-server_2.2.20-1_amd64.deb ...
Unpacking gpg-wks-server (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../04-gpg-agent_2.2.20-1_amd64.deb ...
Unpacking gpg-agent (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../05-gpg_2.2.20-1_amd64.deb ...
Unpacking gpg (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../06-gpgconf_2.2.20-1_amd64.deb ...
Unpacking gpgconf (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../07-gnupg-l10n_2.2.20-1_all.deb ...
Unpacking gnupg-l10n (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../08-gnupg_2.2.20-1_all.deb ...
Unpacking gnupg (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../09-gpgsm_2.2.20-1_amd64.deb ...
Unpacking gpgsm (2.2.20-1) over (2.2.12-1+deb10u1) ...
Preparing to unpack .../10-gpgv_2.2.20-1_amd64.deb ...
Unpacking gpgv (2.2.20-1) over (2.2.12-1+deb10u1) ...
Setting up gpgv (2.2.20-1) ...
Selecting previously unselected package libbrotli1:amd64.
(Reading database ... 85881 files and directories currently installed.)
Preparing to unpack .../libbrotli1_1.0.7-6.1_amd64.deb ...
Unpacking libbrotli1:amd64 (1.0.7-6.1) ...
Preparing to unpack .../curl_7.68.0-1_amd64.deb ...
Unpacking curl (7.68.0-1) over (7.64.0-4+deb10u1) ...
Preparing to unpack .../libcurl4_7.68.0-1_amd64.deb ...
Unpacking libcurl4:amd64 (7.68.0-1) over (7.64.0-4+deb10u1) ...
Setting up libbrotli1:amd64 (1.0.7-6.1) ...
Setting up gnupg-l10n (2.2.20-1) ...
Setting up gpgconf (2.2.20-1) ...
Setting up libcurl4:amd64 (7.68.0-1) ...
Setting up curl (7.68.0-1) ...
Setting up gpg (2.2.20-1) ...
Setting up gnupg-utils (2.2.20-1) ...
Setting up gpg-agent (2.2.20-1) ...
Installing new version of config file /etc/logcheck/ignore.d.server/gpg-agent ...
Setting up gpgsm (2.2.20-1) ...
Setting up dirmngr (2.2.20-1) ...
Setting up gpg-wks-server (2.2.20-1) ...
Setting up gpg-wks-client (2.2.20-1) ...
Setting up gnupg (2.2.20-1) ...
Processing triggers for libc-bin (2.30-8) ...
$ curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /usr/local/bin/sops
$ chmod +x /usr/local/bin/sops
$ cat $KEY | gpg --batch --import
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key FFEFFE9451381735: public key "continuous-integration <ci@domain.fr>" imported
gpg: key FFEFFE9451381735: secret key imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: secret keys read: 1
gpg: secret keys imported: 1
$ echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
$ sops -d int.encrypted.env > int.env
$ cat int.env
secret=5Tz2QNxki789YFDa
Running after_script
00:01
Saving cache
00:02
Uploading artifacts for successful job
00:01
Job succeeded
Like we see here (and in the GitLab job view), every time the pipeline is triggered, it will re-install every software, from gpg to sops, which can be time-consuming and source of error.
π¦ GitLab CI Optimization
Devon will create another project just for CI tooling image, and he will prepare the image of the CI to be able to have gpg and sops installed. Here is for example, the Dockerfile:
FROM tutum/curl AS downloader
RUN curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /opt/sops && \
chmod +x /opt/sops
FROM google/cloud-sdk as final
COPY --from=downloader /opt/sops /usr/local/bin/sops
RUN apt-get update && apt-get install -y gnupg --no-install-recommends
Then, he will activate a schedule to build this image every day and publish it in their own docker registry. Thanks to this, he can improve the previous .gitlab-ci.yml:
deploy int:
image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops # Image created by the previous Dockerfile π³
before_script:
- cat $KEY | gpg --batch --import
- echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
script:
- sops -d int.encrypted.env > int.env
- cat int.env
You can see, the job execution is much straightforward now:
Running with gitlab-runner 13.0.0 (c127439c)
on docker-auto-scale ed2dce3a
Preparing the "docker+machine" executor
Using Docker executor with image registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
Using docker image sha256:ec7e941ae9f66656c9c0b7b4ce61b229eeb215492c65ebfe21c69026aa166013 for registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
Preparing environment
00:05
Running on runner-ed2dce3a-project-19225532-concurrent-0 via runner-ed2dce3a-srm-1591446538-6f33846d...
Getting source from Git repository
00:03
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/davinkevin/sops-blog-post-repository/.git/
Created fresh repository.
From https://gitlab.com/davinkevin/sops-blog-post-repository
* [new ref] refs/pipelines/153552640 -> refs/pipelines/153552640
* [new branch] master -> origin/master
Checking out 625b7479 as master...
Skipping Git submodules setup
Restoring cache
00:01
Downloading artifacts
00:02
Running before_script and script
00:04
Authenticating with credentials from job payload (GitLab Registry)
$ cat $KEY | gpg --batch --import
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key FFEFFE9451381735: public key "continuous-integration <ci@domain.fr>" imported
gpg: key FFEFFE9451381735: secret key imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: secret keys read: 1
gpg: secret keys imported: 1
$ echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
$ sops -d int.encrypted.env > int.env
$ cat int.env
secret=5Tz2QNxki789YFDa
Running after_script
00:02
Saving cache
00:01
Uploading artifacts for successful job
00:02
Job succeeded
NOTE: You can use this method with any docker registry, not only the one from GitLab π¦.
Of course, the before_script should be copy/paste for every job requiring access to secrets. Devon will refactor this to optimize the Don't Repeat Yourself aspect:
.auth: &auth | # <1>
function gpg_auth() {
cat $KEY | gpg --batch --import > /dev/null
echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp) > /dev/null
}
gpg_auth
deploy alice:
image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops
before_script:
- *auth # <2>
script:
- sops -d dev_a.encrypted.env > dev_a.env
- cat dev_a.env
deploy int:
image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops
before_script:
- *auth # <3>
script:
- sops -d int.encrypted.env > int.env
- cat int.env
- A
YAMLanchor used to set a custom shell function and calling it right away. - Usage of the anchor as a
before_scriptstep in the job. - Re-use of the
authanchor in another job.
You can see deploy alice and deploy int logs in the GitLab UI.
Of course, this is just an extract of the real .gitlab-ci.yaml, which includes linting, test, build & cie π.
Conclusion
Here, we learn how to configure our CI identity and configure the GitLab CI π¦ to be able to decrypt secrets. We also see how to optimize it in a GitLab environment leveraging usage of home made docker image.
You can find the source code of this article, files and scripts in this GitLab repository. The CI is triggered in this GitLab repository
Top comments (0)