CI/CD for interdepartmental teams

Introduction

The topic of CI/CD is often considered the domain of DevOps, infrastructure, and development teams. I would argue this thinking misses the great opportunity that is including the entire organisation in the conversation around software delivery lifecycle (SDLC). From QA, to product managers, to of course operators, security, all the way through to the top of engineering and technology leadership. Especially as organisations increasingly rely on software, not only in tech industry but in most other industries as well, it makes sense to spread the DevOps thinking and the use of tools such as CI/CD to more folks, more teams, and more departments.

Incidentally, that’s what the DevOps movement is all about, and its natural evolution. Bringing the principles of early and continuous collaboration across departments to their entire organisations.

This article argues for the use of CI/CD across various company departments - to anyone involved in - or adjacent to - the software delivery process, and covers the principles and tools that will benefit them, and points you to how to implement this using CircleCI in your organisation.

Who this article is for

People who will benefit the most from this article are the ones implementing pipelines themselves - be it DevOps and Software engineers. While this article is not a tutorial, yet it contains ideas and snippets for implementing the topics discussed within.

As we’re talking about how CI/CD can benefit most of the organisations, this article will be approachable to people with many different backgrounds. From developers to operations teams, all the way to folks dealing with delivery and governance - delivery managers, security teams, engineering leadership.

The software delivery lifecycle from the CI/CD perspective

Software delivery lifecycle, from the perspective of the CI/CD processes works kind of like this - the developers on the team write and push code changes, have them reviewed, and then automatically tested or otherwise validated, built, and ultimately deployed to a target environment using the CI/CD system, at which point the running software begins to be monitored by the operations teams.

What technologies are used, where it is deployed, how it is monitored, and by whom, will all depend on the organisation; From Go microservices running on Kubernetes, to static websites with lambda functions, to iOS applications on the App Store, and everything in between.

But of course there are many more things we do. Software doesn’t begin in an editor, it more often begins as a story somewhere on a backlog or a ticket in a bug tracker. That’s when we actually start thinking about what needs to be built, and how it should work, and for whom (and whether it ultimately does work or not), so why not use CI/CD to do more than just build, validate, and deploy?

Expanding the SDLC for the entire organisation

A modern CI/CD platform can work beyond just building testing and deploying code, and instead it can tie together many more people and teams.

In this section we will look at some of these capabilities, and how connecting to external tools can make us focus more on the software we're building and ultimately build better software.

QA, Product, and before work gets made

QA engineers work best with constant access to latest builds to run tests on, as well as run and automate their their own tests. They are often also existing CI/CD users, who write test code and commit to the codebase.
We can create build outputs either directly in CircleCI with store_artifacts and store_test_results as in the example below, as well as of course through third party tools such as AppStore Connect, and deploying to test environments.

  jobs:
    integration-test:
      docker:
        - image: ...
      steps:
        - checkout
        ...
        - run-test ...
        - store_artifacts:
            path: tested_app_binary
        - store_test_results:
            path: test-results

Beyond the CI/CD platform itself, QA engineers also work a lot with product management and ticketing software - be it validating the requirements, logging defects, or confirming that updated products do not indeed have the defects anymore.
A CI/CD tool can integrate with a PM tool such as Jira, so that all your builds and deploys are automatically available and referenceable in it. This of course doesn’t only apply to QA engineers but also for product managers.

You do so by first connecting CircleCI with Jira, and then making sure that you label each commit with the right ticket name - like AB-234. Finally, as the last step of your deployment or build job - you can invoke the Jira orb's notify command, with which you can mark deployments and builds, as in this example:

orbs:
    jira: circleci/jira@1.3.1
workflows:
  build:
    jobs:
      - build:
          post-steps:
            - jira/notify 

Notifications

You can also notify your QA engineers and product managers of new releases waiting for their verification through the tools the teams already use - like Slack.
You can install a CI/CD tool as an application in your Slack workspace, which will also let it to post on your behalf. There are many ways to use it, frpm informing you when certain workflows and jobs are like deployments completed, tests are failing, and also when your workflow is waiting on an action - such as an approval or review by a QA engineer, product manager, and also security engineer, or a delivery manager. Below is an example of a generic deployment notification:

jobs:
  deploy:
    docker:
      ...
    steps:  
      - perform_deployment
      - slack/notify:
            event: pass
            template: success_tagged_deploy_1

Besides Slack there are of course numerous other tools organisations might use, from other instant messaging products to more conventional text messages and e-mails. All of these tend to have APIs which enable similar integrations to be built.

Security, Delivery management, and approvals

We have mentioned a few new roles; Security engineers are crucial for all companies to include in their SDLC processes. In an automated CI/CD pipeline, if a security team member doesn’t approve of a deployment after reviewing all necessary documentation and performing the checks, then the deployment should not complete.

CircleCI offers two very powerful concepts for ensuring access control - approval jobs and contexts.
Approval jobs introduce a manual step in your otherwise automated pipeline, which must be confirmed by an authenticated user in order for the pipeline to proceed. This is very useful for deployments for example.
Contexts in CircleCI can be applied to individual jobs, and can be restricted to specific user groups when for example connected to GitHub.
When using both of them, we can apply contexts to the subsequent job in the workflow after the approval job, in addition to adding user groups to the context itself.
This means that if a non-authenticated person tries to approve this step, the subsequent step will fail with an unauthorized message. I have written a whole blog post on the topic of implementing access control: https://circleci.com/blog/access-control-cicd/.

You can of course combine both with what we covered previously - Jira and Slack, to flag and notify the folks required to act on a deployment when it happens.

Going a few steps further even, once the relevant team member is notified and sent a link to the relevant pipeline or job, why not give them everything they need to make a decision? We expect this it as part of reviewing pull and merge requests after all.
For every testing job, security scan, and other checks we execute these tools can usually gather reports - why not make all reports available in a single report job. That’s possible by passing reports between jobs using a workspace:

jobs:
  security_scan:
    ...
    steps:
      - ...
      - perform-security-scan
      - persist_to_workspace:
          root: workspace
          paths:
            - security-output
  generate_reports:
    ...
    steps:
      - attach_workspace:
          at: /tmp/workspace
      - store_artifacts:
          path: /tmp/workspace/security-output
      - store_artifacts:
          path: /tmp/workspace/some_other_report_output

In this way you can leverage the power of a workspace to create a single point where all the relevant reports are collated. This can be a lightweight job which can save a busy team a lot of time and effort. Besides security engineers the same steps can be taken to inform delivery managers for example, and the manual approval jobs can be chained one after another, so the review steps are taken in order.

Ops, deployment reporting, and going beyond the individual dev team

Now the software is approved and being deployed, we can notify the operations teams of a new deployment.
That means that if any defects have been published, we can trace that back to the problematic deployment, revert that feature, and understand the root cause of breakage. Depending on what tools your operations teams use to monitor the software in production, you can call that API using either an orb or directly, which lets you tie a CircleCI pipeline with a deployment event.

Finally let’s take a few steps back and realise that we’ve only been talking about a single team and the relevant stakeholders. Apart from very small companies (whose engineers are too busy to read this article anyway) that single team doesn't exist in a vacuum, but works with adjacent teams working on different parts of software, sometimes even integrating one another. So not only is there a single software development cycle, there are many.

This means that implementing the above steps for a single team helps just that - the single team, but far from the wider organisation. People responsible for working across teams, should take on that work to ensure best practices are followed throughout the organisation. This could be the DevOps team that implement the integrations with security, PM, or ops processes, or the head of engineering looking to find way for their teams of engineers to learn and adopt approaches that work from each other. Looking at insights across multiple teams you could consider the pipeline throughput, mean time to recovery, and other metrics covered in our 2022 State of Software Delivery report to understand and identify the best practices within your own organisation, and areas where you can apply these relatively small improvements for the greatest impact in most teams.

Conclusion

We’re not in the world anymore where CI/CD is the sole domain of the development teams.
The role of DevOps teams is to not only work across the Dev organisations but with the wider teams, promoting the virtues of automation and how it benefits other parts of the business. The DevOps and CI/CD engineers are in fact uniquely positioned to bring that automation to more and more people and roles within the org.

In this article we have looked at a few ways CI/CD can be used to reach more parts of the company, from sending a slack message to your security engineers telling them of a required review, to letting your QA team know that a new bug fix has been deployed and is ready for testing in their ticketing application, to helping your ops team know that a new deployment has indeed started and could potentially have an impact on the users they are seeing.

If this article has helped you in a way, I would love to know, also if you have any questions or suggestions about it, or ideas for future articles and guides, reach out to me on Twitter - @zmarkan or email me.