Prerequisites
This is a continuation of my previous article: Monorepo using Lerna, Conventional commits, and Github packages. The prerequisites of that previous article are assumed to understand this one, so you may want to read it first for extra context.
If you feel stuck at any point during the article or you simply want to get the code right now, it can be found in the repository.
Context
Continuous Integration (CI)
Continuous Integration is a practice in software development that consists in integrating the code as frequently as possible. Before integrating the code, it's common to execute a series of checks such as running tests or compiling/building the project, aiming for detecting errors the earlier the better.
A common practice is to automatically execute these checks when opening a new Pull Request or even pushing code to the repository to force that all these checks pass before we can safely integrate the changes into the codebase.
Github actions
Github actions are a Github feature that allows developers to execute workflows when certain events happen in our repositories, such as pushing code or closing a Pull Request(often used in Continuous Integration scenarios). Github actions are free of charge for public repositories.
These workflows are organized in jobs
, steps
and actions
in a nested fashion, and are triggered by one or more events
. Each workflow is a single file written in the YAML language.
What are we going to build?
We are going to automate the versioning and publication of the packages in our monorepo using Lerna (with conventional commits) and Github actions.
We are going to implement two different Github workflows:
1 - Checks
workflow: When we open a new Pull Request or push changes to a pull request that is open, it will run a set of checks that we consider essential for integrating the changes into our codebase.
2 - Publish
workflow: Whenever a Pull Request is merged, we'll execute a workflow that will version and publish our packages. It will behave slightly different depending on the destination branch:
- When merged against the
development
branch, it will publish beta versions of the changed packages (suitable for QA or testing). - When merged against the
main
branch, it will publish final versions (ready for production).
We will start from an existing monorepo that already contains two javascript packages that I created for this previous article.
The following picture illustrates the workflows that we will implement in Github actions terminology:
Hands-on
Part 1 - Checks workflow on PR open/modified
Github expects workflows to be located under the ${projectFolder}/.github/workflows
, so let's create a new Github branch and add our first workflow checks.yaml
inside that directory (you can create workflows from the Github UI too):
The project structure looks like this:
/
.github/
workflows/
checks.yaml
[...]
Now, let's start working on the workflow. Open the checks.yaml
file in an editor and add the following attributes:
name: Checks # Workflow name
on:
pull_request:
types: [opened, synchronize] # Workflow triggering events
-
name
: The name of the workflow. -
on
: The listener of the event(s) that will trigger this workflow. In our case, it will be triggered every time that a Pull Request gets opened or modified.
Next, we will add a job to the workflow and configure the type of instance that Github will spin up for running it with the runs-on
attribute:
name: Checks
on:
pull_request:
types: [opened, synchronize]
jobs: # A workflow can have multiple jobs
checks: # Name of the job
runs-on: ubuntu-latest # Instance type where job will run
This job will contain several steps:
-
Checkout
: Get the code from the repository where the workflow is defined. -
Setup NodeJS
: Setup NodeJS with a specific version. -
Setup npm
: Since we will install dependencies from our private registry (in Github packages), we have to add it to the npm config. -
Install dependencies
: Install the needed npm packages. -
Run tests
: Execute tests, if any.
In a real-world project it's likely that we run other steps such as checking syntax using a linter, building the project or running any other check/process that we consider essential to mark the changes as valid before integrating them into the codebase.
Custom vs public actions
For some of the mentioned steps we will write the commands from scratch but for others, we will take advantage of existing public actions that have been created by the community and are available in the Github marketplace.
The public actions use the uses
keyword and the custom commands (single or multiple lines) use the run
one.
Let's implement the first two steps of the build
job:
name: Checks
on:
pull_request:
types: [opened, synchronize]
jobs:
check:
runs-on: ubuntu-latest
steps:
- name: "Checkout" # Download code from the repository
uses: actions/checkout@v2 # Public action
with:
fetch-depth: 0 # Checkout all branches and tags
- name: "Use NodeJS 14" # Setup node using version 14
uses: actions/setup-node@v2 # Public action
with:
node-version: '14'
The
Checkout
step will download the code from the repository. We have to add thedepth: 0
option so Lerna can properly track the tags of the published packages versions and propose new versions when it detects changes.In the
Use NodeJS 14
step we are configuring NodeJS to use the version 14 but we could even execute it for multiple versions at once using a matrix.
Let's commit and push this version of the workflow to Github and open a Pull Request afterward (if you don't have a development
branch already created, create one from main
because we'll open the pull request against it).
Once the Pull Request has been opened our workflow will be executed. Open a browser and navigate to the "Actions" section of the repository to see the execution result:
If we click on it, we can see the execution details, and by clicking on any of the jobs (in our case, the checks
job) we will be able to see the status and outputs of each of its steps:
Let's add the next step: Setup npm
. In this step, we'll add our Github packages registry to the .npmrc
file so npm can find the packages published in our Github packages registry.
One or multiple commands can be executed in every step action. In this case, we'll run a couple of npm set
commands in the same action:
name: Checks
on:
pull_request:
types: [opened, synchronize]
jobs:
checks:
runs-on: ubuntu-latest
steps:
- name: "Checkout"
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: "Use NodeJS 14"
uses: actions/setup-node@v2
with:
node-version: '14'
- name: "Setup npm" # Add our registry to npm config
run: | # Custom action
npm set @xcanchal:registry=https://npm.pkg.github.com/xcanchal
npm set "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}"
Workflow environment variables
In the previous snippet, you'll have noticed the secrets.GITHUB_TOKEN
. This environment variable is added by Github and can be used to authenticate in our workflow when installing or publishing packages (know more).
A part from that one, Github adds other variables such as the branch name or the commit hash, which can be used for different purposes. The complete list is available here.
Next, we'll add another step: Install dependencies
. In this step action we'll install the root dependencies in production mode (see npm ci command) as well as running lerna bootstrap
for installing the dependencies for each of our packages and create links between them.
name: Checks
on:
pull_request:
types: [opened, synchronize]
jobs:
checks:
runs-on: ubuntu-latest
steps:
- name: "Checkout"
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: "Use NodeJS 14"
uses: actions/setup-node@v2
with:
node-version: '14'
- name: "Setup npm"
run: |
npm set @xcanchal:registry=https://npm.pkg.github.com/xcanchal
npm set "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}"
- name: Install dependencies
run: | # Install and link dependencies
npm ci
npx lerna bootstrap
Commit and push the changes and see how the "Pull Request synchronized" event triggers our workflow, which now contains the last steps that we added:
Before adding our last step Running tests
we need to make a change in our date-logic
and date-renderer
packages, modifying the npm test script. Since we haven't implemented any actual test yet, we'll simple echo "TESTS PASSED" when that command is executed.
Modify the test script in the package.json
of the date-logic
package and push the changes to the repo. Then, repeat the same process for the date-renderer
.
# package.json
"scripts": {
"test": "echo TESTS PASSED"
}
# commit and push
$ git add .
$ git commit -m "feat(date-logic): echo tests"
$ git push
After pushing the new test command to our packages we can add the Running tests
step to our workflow.
name: Checks
on:
pull_request:
types: [opened, synchronize]
jobs:
checks:
runs-on: ubuntu-latest
steps:
- name: "Checkout"
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: "Use NodeJS 14"
uses: actions/setup-node@v2
with:
node-version: '14'
- name: "Setup npm"
run: |
npm set @xcanchal:registry=https://npm.pkg.github.com/xcanchal
npm set "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}"
- name: Install dependencies
run: |
npm ci
npx lerna bootstrap
- name: Run tests # Run tests of all packages
run: npx lerna exec npm run test
Push the changes to the repository and see the execution results in the Github actions section:
Congrats! we completed our first job and half of this tutorial.
Part 2 - Publish workflow on PR merged
Create a publish.yaml
file under the workflows
repository with the following content. You'll notice that we added a new branches
attribute to the event listeners. With this configuration, we're telling Github that only executes this workflow when a Pull Request is merged either against development
or main
branch.
name: Publish
on:
pull_request:
types: [closed]
branches:
- development
- main
Now, we'll add a job named publish
to this workflow, the runs-on
attribute and a new one that we haven't used yet: if
. This attribute is used to evaluate an expression to conditionally trigger the job if it evaluates to true or false (it can be used in steps too).
According to the on
attribute that we configured, this workflow will trigger on every "Pull Request closed" event against development
or main
, but what we actually want is to execute it ONLY when the Pull Request has been merged (not discarded). Therefore, we have to add the github.event.pull_request.merged == true
condition to the job:
name: Publish
on:
pull_request:
types: [closed]
branches:
- development
- main
jobs:
publish:
if: github.event.pull_request.merged == true # Condition
runs-on: ubuntu-latest
Now, let's replicate the same first three steps that we added in the checks workflow (Checkout
, Use NodeJS 14
and Setup npm
)
name: Publish
on:
pull_request:
types: [closed]
branches:
- development
- main
jobs:
publish:
if: github.event.pull_request.merged == true
runs-on: ubuntu-latest
steps:
- name: "Checkout"
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: "Use NodeJS 14"
uses: actions/setup-node@v2
with:
node-version: '14'
- name: "Setup npm"
run: |
npm set @xcanchal:registry=https://npm.pkg.github.com/xcanchal
npm set "//npm.pkg.github.com/:_authToken=${{ secrets.GITHUB_TOKEN }}"
Finally, we will add the final (and interesting) step: Publish and version
. Let's analyze in detail the step attributes and the commands inside the action:
- Since Lerna will be in charge of publishing new versions of the packages, we have to set the
GH_TOKEN
environment variable with our Personal Access Token as the value, so Lerna has the required permissions. - We have to add a couple of Github configuration lines to specify the username and email credentials, so Lerna can make commits and create tags for the new versions in the repository. For that, we'll take advantage of the
github.actor
variable available in the environment. - In the if/else statement we're checking the
${{ github.base_ref }}
variable to see if the destination branch of the PR isdevelopment
. In that case, we will send the--conventional-prerelease
and the--preid
flags to the Lerna version command to generate beta versions. Otherwise (it only can bemain
because we restricted at the workflow level that it must be one of these two branches), we will use the--conventional-graduate
argument to generate final versions. Last but not least, the--yes
flag autoconfirms the version and publish operations (otherwise Lerna would prompt for manual confirmation and the CI would fail).
name: Publish
on:
pull_request:
types: [closed]
branches:
- development
- main
jobs:
publish:
if: github.event.pull_request.merged == true
runs-on: ubuntu-latest
steps:
- name: "Checkout"
uses: actions/checkout@v2
with:
fetch-depth: 0
- name: "Use NodeJS 14"
uses: actions/setup-node@v2
with:
node-version: '14'
- name: "Version and publish" # Interesting step
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
git config user.name "${{ github.actor }}"
git config user.email "${{ github.actor}}@users.noreply.github.com"
if [ ${{ github.base_ref }} = development ]; then
npx lerna version --conventional-commits --conventional-prerelease --preid beta --yes
else
npx lerna version --conventional-commits --conventional-graduate --yes
fi
npx lerna publish from-git --yes
Let's commit the new workflow to the repository and merge the Pull Request afterward, so it gets triggered. If we inspect the output of the Version and publish
step we can see a lot of information about the two steps that Lerna executed:
1) When running the lerna version
command, it detected the changes in the packages and proposed new beta versions (notice the -beta.1
prefix) that were auto-accepted. After that, it pushed the version tags to the Github repo:
2) When running the lerna publish from-git
command, it analyzed the latest Github tags to determine the versions that had to be published and published the packages to the Github package registry.
So now we have some testing versions in our Github packages registry:
We'll assume that they have been through testing and that are marked as ready for production. Let's create a new Pull Request from development
against master
, merge it and see how the same Publish
job is executed, but this time Lerna will publish the final versions:
Conclusion
We have seen how powerful a couple of Lerna commands can be (in conjunction with a proper conventional commits history) for the Continuous Integration workflows of our monorepos.
By automating these steps we can forget about having to manually decide the versions for all of our packages and thus, avoiding human errors. In this case, we used Github actions for doing it but any other tool such as Jenkins or CircleCI would work too.
Next steps
- Configure Github branch protection rules to block the Pull Request merge button if the
checks
workflow failed. - Set up a commit syntax checker (e.g. commitlint) to avoid human mistakes that could impact the versioning due to an inconsistent commit history.
Follow me on Twitter for more content @xcanchal
Buy me a coffee:
Top comments (16)
thanks, great post! One comment though, I would look into replacing the custom token with
GITHUB_TOKEN
to follow best practices security wise. according to their docs"GitHub Packages allows you to push and pull packages through the GITHUB_TOKEN available to a GitHub Actions workflow."
docs.github.com/en/packages/managi...
Great suggestion Ben. I was doubtful about wether the
GITHUB_TOKEN
would allow authenticating against package registries and I see it does!What an awesome series, clarified lot of Lerna sticking points for me.
Questions-
1) lets say you keep add 3 merges to
development
without pulling tomain
so how does the versions work out?
a) after 1st merge (fix) in development 2.1.0-beta.0 --> 2.1.0-beta.1 ? or 2.1.1-beta.0?
b) assuming 2.1.1-beta.0, then 2nd merge (fix) in development 2.1.2-beta.0
c) then 3rd merge (fix) in development 2.1.3-beta.0
Now when we merge to
main
then it becomes 2.1.3? hence directly jumping from2.1.0 to 2.1.3 in
main
?Any good reading resources on semantic relases apart from the dry semVer doc??
2) Do you see if there is a need to changes fetch depth in the YAML - github.com/lerna/lerna/issues/2542
How can we add conventional commits for a git commit message which involves ay changes in 3 packages - pkg1, pkg2, pkg3 ?
Scenario
Previously,
pkg1 - 1.0.0
pkg1 - 1.0.0
pkg1 - 1.0.0
git commit -m "
feat(pkg1): added button color
fix(pkg2): fixed typo
fix(pkg3): fixed header
"
Now,
pkg1 - 1.1.0
pkg1 - 1.0.1
pkg1 - 1.0.1
Is that so? Also how to add breaking changes in the multiple package change commit where 1 of the packages have a breaking change, others do not.
Thanks!!!
Hi! I would commit the changes of each package separately (one commit for each package)
I'm facing this issue:
Any idea?
Had a query on the usage of the packages after they are published-
let's say
step1) changes merged to
dev
--> publishes --> 1.0.1-beta.0step2) the changes from
dev
are now merged tomain
--> publishes --> 1.0.1Now can an end user
install
both1.0.1-beta.0
&1.0.1
?Yes, they are published and available in the registry unless you delete them.
Thanks for post, great help! Question for me is how do you upgrade project version? After you merge to master, it removes
-beta
postfix, but how do you upgrade package version from2.1.0
further to2.1.0-beta.0
? And how can it determine if to change major, minor or patch number?Hi SFilisnky. Lerna parses the conventional commit history, which include of the scope change, to determine the major, minor or patch version (see conventional commits specification). For the -beta removal, it’s the other way around. Let’s say you are in 2.1.0 and you merge a patch PR against development. Then, Lerna will bump to 2.1.1-beta.0 and when merging against master it will remove the suffix, leaving the final 2.1.1 version. Does that make sense?
Thank you for the great post! Got everything working except one thing that is bugging me:
When using
--conventional-prerelease
, lerna creates the correct changelogs(with the correct commit history)but when I use
--conventional-graduate
, lerna only puts this into my changelog:Note: Version bump only for package
Also this only happens when running the command in github action, outside of it, it's fine!
Do you know why? Thanks!
Is this still a workable solution? I followed your steps but on the version/publish step I get the error:
EUNCOMMIT Working tree has uncommitted changes, please commit or remove the following changes before continuing:
because the version command has updated the package.json and is therefore not able to carry on.
Hi Some Edge Cases I wanted to discuss
So we have master and dev branch merging to dev branch npm ci works perfectly and versioning is performed and beta is applied. But now if you try to merge this branch into master your lockfile is outdated npm ci would throw error cant perform installing node_modules.
PS: using pnpm as package manager
This is a great post, thanks. When I create a new Pull Request from development against master (main), in my case, I end up with merge conflict. The issue is that the development branch has beta versions and the main has major versions. So there is a conflict everywhere the version is defined. This can't be the desired behavior but I can't see a way around it... Maybe I misunderstand something about the workflow? Any Ideas about where I am going wrong?
have you found a better branching approach to solve that problem?