We started with a monorepo at my current company and have been using circle almost since the beginning. It was tough. It required a lot of boilerplate in our config and the necessity to get every developer to generate a circle API key and add it to git. It never felt that great but at least it worked. In April CircleCI released the dynamic configuration API and this allowed us to refactor our monorepo support into something we think is pretty great. We've gone from tolerating Circle to enjoying it, and now we've released the code as an open source project and provided the functionality as an orb so that any project can benefit.
The project is circletron, the code is hosted on github and the orb is here. It currently supports lerna monorepos but we plan to support other monorepos in the future.
Motivating example
Typically a circleci configuration exists in a single file within a repository at .circleci/config.yml
. For a monorepo a very minimal example may look something like this:
version: 2.1
jobs:
validate-everything:
steps:
- checkout
- run: npm run general-validation
test-subpackage-a:
steps:
- checkout
- run: cd packages/package-a && npm run test
test-subpackage-b:
steps:
- checkout
- run: cd packages/package-b && npm run test
test-subpackage-c:
steps:
- checkout
- run: cd packages/package-c && npm run test
publish-subpackage-c:
steps:
- checkout
- run: cd packages/package-c && npm run publish
workflows:
validate-everything:
jobs:
- validate-everything
subpackage-a:
jobs:
- test-subpackage-a
subpackage-b:
jobs:
- test-subpackage-b
subpackage-c:
jobs:
- test-subpackage-c
- publish-subpackage-c
Obviously there are many problems with this, for a start all the CI is defined in one file at the root of the project. Worse, each of the jobs may take some time to complete and on every commit to the project, circleci will run every single job no matter whether there are any changes to the respective subpackages or not. It may not look so bad in this tiny example but the bigger the configuration gets, the worse it is to deal with.
Using circletron to solve this
With circletron
the .circle/config.yml
is always the same:
version: 2.1
setup: true
orbs:
circletron: circletron/circletron@3.0.1
workflows:
trigger-jobs:
jobs:
- circletron/trigger-jobs
The trigger job step will take many individual circle.yml
distributed within the project and combine them into a single configuration which will be issued via the continuation API. It will modify the configuration to ensure that jobs that are not necessary are no longer run, in a way that is friendly to CI branch protection rules.
The single configuration file can now be split up across the monorepo, with one optional circle.yml
in the root of the project and where the CI for each subpackage lives in the directory for that subpackage.
In this instance the circle.yml
in the root of the project will host configuration not specific to any subpackage:
version: 2.1
jobs:
validate-everything:
steps:
- checkout
- run: npm run general-validation
workflows:
validate-everything:
jobs:
- validate-everything
The jobs here are run on every commit and it's also a good place to set version
. It's also a good place to provide commands
that can be used in subpackage specific circle.yml
files.
Now lets look at packages/package-a/circle.yml
:
jobs:
test-subpackage-a:
steps:
- checkout
- run: cd packages/package-a && npm run test
workflows:
subpackage-a:
jobs:
- test-subpackage-a
Everything related to this package all in one place. Even better, when the PR contains no changes within packages/package-a
the jobs for this subpackage will be skipped. You will probably want to use branch protection rules to ensure that when the test-subpackage-a
job fails the PR will be blocked so omitting this job entirely would not be ideal. Omitted jobs remain in pending
state permanently, blocking the PR. circletron replaces unneeded jobs with a simple job using the busybox:stable
docker image that echoes "Job not required"
and issues a success error code.
Dependencies and unconditional jobs
What if the code in packages/subpackage-c
uses code from packages/subpackage-a
? In this case the jobs within this package should run when the code in subpackage-a
changes, even if the code in the subpackage itself doesn't change.
There may also be some instances where a job should run on every push.
The configuration at packages/subpackage-c/circle.yml
shows how to add dependencies and create jobs which run unconditionally:
dependencies:
- subpackage-a
jobs:
test-subpackage-c:
steps:
- checkout
- run: cd packages/package-b && npm run test
publish-subpackage-c:
conditional: false
steps:
- checkout
- run: cd packages/package-c && npm run publish
workflows:
subpackage-c:
jobs:
- test-subpackage-c
- publish-subpackage-a
In this instance test-subpackage-c
will only be run when changes to package-a
or package-c
are detected and publish-subpackage-c
will run on every push.
Target branches
CircleCI does not pass the target branch to workflows or jobs so it becomes necessary to help circletron out. By default circletron considers the branches main
, master
, develop
and any branch starting with release/
as a target branch. The latest commit from the branch commit history which belongs to one of these branches is considered to be the branch-point. For pushes to one of the branches above, all of the jobs are run. This can be changed via the configuration file at .circleci/circletron.yml
:
targetBranches: ^(release/|main$|master$|develop$)
Any branch which matches this regex is considered a target branch, the above configuration shows the default.
circletron is available to use now.
Top comments (0)