Uclusion had what should have been a simple problem. We needed to deploy our Lambda, DynamoDB and React based application to dev, stage and production environments without owning a lot of our own code to do so. The code is spread out over many repositories where each repo exposes a different set of back end APIs or holds the front end code. The back end APIs are testing by automated integration tests and the front end by a combination of automated and manual QE. In addition sometimes upgrade scripts must be run before deploying.
We expected to adopt some out of the box solution in CircleCI but instead ended up having to write a lot of code to tag the repos that passed the integration tests as they go from one environment to the next. Here is a partial excerpt from that code:
from utils.constants import rest_api_backend_repos
import sys
def get_latest_release_with_prefix(releases, prefix):
latest = None
latest_date = None
for release in releases:
if release.tag_name.startswith(prefix):
created_at = release.created_at
if latest is None or created_at > latest_date:
latest = release
latest_date = created_at
return latest
def find_latest_release_with_prefix(repo, prefix):
releases = repo.get_releases()
latest_release = get_latest_release_with_prefix(releases, prefix)
if not latest_release:
print("Couldn't get a release with prefix " + prefix + " for " + repo.name)
sys.exit(3)
return latest_release
def get_latest_releases_with_prefix(github, prefix, repo_name=None, is_ui=False):
if repo_name:
repos_to_search = [repo_name]
else:
repos_to_search = rest_api_backend_repos if not is_ui else ['uclusion_web_ui']
candidates = []
for repo in github.get_user().get_repos():
if repo.name in repos_to_search:
latest_release = find_latest_release_with_prefix(repo, prefix)
candidates.append([repo, latest_release])
if len(candidates) != len(repos_to_search):
print("Some repos are missing tags")
sys.exit(5)
return candidates
def get_tag_for_release(repo, release):
for candidate in repo.get_tags():
if candidate.name == release.tag_name:
return candidate
return None
def get_tag_for_release_by_repo_name(github, repo_name, release):
for repo in github.get_user().get_repos():
if repo.name == repo_name:
return get_tag_for_release(repo, release)
return None
def clone_release(repo, old_release, new_name):
# get the tag for the release
tag = get_tag_for_release(repo, old_release)
if not tag:
sys.exit(4)
sha = tag.commit.sha
repo.create_git_tag_and_release(new_name, 'Blessed build tag', new_name, 'Blessed', sha, 'commit')
def get_master_sha(github, repo_name):
for repo in github.get_user().get_repos():
if repo.name == repo_name:
head = repo.get_git_ref('heads/master')
return head.object.sha
return None
def release_head(github,dest_tag_name, prebuilt_releases, repo_name=None, is_ui=False):
sha_map = {}
for entry in prebuilt_releases:
repo = entry[0]
release = entry[1]
tag = get_tag_for_release(repo, release)
if tag:
sha_map[repo.name] = tag.commit.sha
if repo_name:
repos_to_search = [repo_name]
else:
repos_to_search = rest_api_backend_repos if not is_ui else ['uclusion_web_ui']
for repo in github.get_user().get_repos():
if repo.name in repos_to_search:
head = repo.get_git_ref('heads/master')
sha = head.object.sha
if sha != sha_map[repo.name]:
print("Will clone head of " + repo.name + " to " + dest_tag_name)
repo.create_git_tag_and_release(dest_tag_name, 'Head Build', dest_tag_name, 'Head', sha, 'commit')
else:
print("Skipping " + repo.name + " because head has already built")
def clone_latest_releases_with_prefix(github, source_prefix, dest_tag_name, repo_name=None, is_ui=False):
print("Cloning releases")
candidates = get_latest_releases_with_prefix(github, source_prefix, repo_name, is_ui)
for candidate in candidates:
repo = candidate[0]
release = candidate[1]
print("Will clone " + release.tag_name + " in repo " + repo.name + " to " + dest_tag_name)
clone_release(repo, release, dest_tag_name)
and then more code to run the upgrade scripts.
The situation does not seem any better in Github Actions because blogs like this are being written and feature requests like this have a horde of beggars.
So how did our out of the box Serverless deployment dream get deferred? Should we, as SEED suggests, use a monorepo? HELL NO.
Uclusion of course has code that is shared between back end services and we put those in Lambda layers (which sadly have their own order of deployment and upgrade scripts). But there is no way we went to all the trouble of using Lambdas just to have them turn to spaghetti in a giant shared repository. There has to be a better way then jumping from the frying pan into the fire.
Could the problem be that we don’t have a complex enough branching strategy? For instance the creators of Serverless configuration YAML (which we use) describe here a three AWS account dev, stage and prod structure like ours but also say:
Promoting from staging to production
If everything in staging looks ok and ready to promote to production, we’ll use our git workflow to merge the desired commit from master to the prod branch. Since we already configured branch deployments for the prod branch, it will automatically run the tests and deploy to the prod stage.
Let’s break down this paragraph though. “If everything in staging looks ok” — that’s got to be passing integration tests that run across all of your services. There is no way any one service has sufficient testing to promote it on its own and at the same time require staging with all of the other services. But then what is the “desired commit” that you are supposedly merging? Your integration tests ran across many commits over many repositories unless you use a monorepo or again aren’t really staging anything at all.
So in conclusion I can’t offer you any out of the box solution to what should have been a generic solved problem. I can only give you some small comfort that you are not alone.
Top comments (0)