Continuous Integration Should NEVER be rushed

Continuous integration is a wonderful thing. The historical process of moving code around from person to person in various different version control systems before running a manual build is just that. History.

But now the relatively young discipline of DevOps is gaining traction with the introduction of various cloud services and on-prem server solutions that boast the ability to bring Continuous Integration to your dev team's workflow.

What are we waiting for! Chuck all of the code into an Azure DevOps / Github / GitLab repo STAT! I've put a pipeline together which will build all our code for us and even test a bit too! We're all set right?

Wrong.

Now, I am not a DevOps engineer at all. I am a software engineer, and I happen to think that the two roles differ significantly in a lot of areas. So I am certainly not qualified to tell you how you must implement your CI pipelines. However, I can (as someone on the dev side of the issue) tell you how not to do it. (Or at least how you can best prepare before you even start) (This is also not an attack by any means on DevOps engineers, most of whose talent and knowledge easily eclipses mine! This is mostly for smaller companies looking to start from scratch with DevOps)

Don't just do 'what Microsoft or Netflix does.'

It's an easy mistake to make. Microsoft build Azure DevOps so surely we have to run our CI process exactly like them right? Well not necessarily. The obvious fact is, you're either Microsoft or you're not. You will most likely have very different ways of working to Microsoft, who have spent decades perfecting their development workflows for the many different products they produce.

There are plenty of articles out there outlining how different companies like Microsoft and Netflix run their workflows. While these are valuable insights and can certainly be a source of inspiration, it doesn't change the fact that you most likely have very different needs.

• You may have a smaller team. A workflow designed to cater for hundreds of commits to a single branch per day might be overkill if you have a team of 5 developers working on one product.
• Legacy software such as codebases running VB6 or heavily ActiveX driven code is not a good fit for a lot of pipeline solutions, particularly Azure Pipelines.
• You may have different levels of reporting requirements from your leadership team, changing the number of approvals for each process which in turn can even affect the kinds of branches you use.
• Your software may need to be tested in a wide number of environments for cross-platform compatibility or account for extremely specific situations / environmental configurations. This can call into question whether you use a containerized approach to CI.
• You may want to enforce automation for code standards such as linting

Some of the companies who's workflows you may find it easier to simply emulate have very particular policies when it comes to shipping code. It has been rumoured for example that some sites like GitHub will ship directly to production, opting to fix with patches as issues are found by the community. If it's true, it must be working for GitHub, but can you honestly say that would work for your highly sensitive banking app?

The above points are only a fraction of the many idiosyncracies found within software teams and companies, which leads me to my next point.

There is no 'One Size Fits All' Approach to CI

I hope I've conveyed in the above that it is naive to think that there is one, universal standard for CI. A recent research paper conducted in partnership between the National University of Defense Technology, China, Carnegie Mellon University and the University of California looked into the application of standardized CI frameworks specifically with the containerization behemoth Docker and reached the conclusion that:

Most of our survey findings were confirmed in the data, but some were not, emphasizing the power of mixed methods to produce holistic findings. Our findings indicate that developers face trade-offs when choosing between different CD workflows with respect to config-urability, simplicity, requirements, performance, stability, developer experience, etc., and we were able to distill some implications for different stakeholders

The sentiment is also put across quite nicely by consultancy Version1.com:

There is no “one size fits all” when it comes to DevOps implementation and processes. It should vary depending on what you’re doing, what type of organisation and software applications you use, deploy or manage...

Build a Process with Developers, not for them

The introduction of any new process can rock even the sturdiest of boats, but if there is anyone who stereotypically despises change it is the Software Developer.

I may be being presumptuous, but I know from experience that if a process is introduced that affects the way you deliver, you want to be consulted.

This need for involvement doesn't stem from an infantile need to have our own way but instead comes from a desire for collaborations and a yearning for good communication between Development and DevOps teams.

It's win-win to work together to implement CI. Your developers fully understand their process, what works well and the technical hurdles that may arise when automation is applied. Putting this together with the expertise of the DevOps engineers who understand the capabilities of the myriad CI tools at your disposal and you are on your way to winning the war, together.

We're also forgetting the QA guys. In a lot of organisations, they are the last people to handle code before it goes out into the world. It might be a good idea for them to be in the loop from the beginning so that you can better design the process that ultimately prepares the product for their sign-off.

Take it Slow

I have limited experience with CI tools but I have worked mostly with Azure DevOps and JetBrain's TeamCity. I like them both, but they have one challenge in common; the sheer amount of potential configurations.

Of course, there are going to be so many different ways to configure a CI solution, seeing as there are so many different ways to build your code. What I would ultimately advise anyone looking to start their treacherous journey into the realms of release automation is to walk before you can run.

Maybe try automating just one of your low-impact repos first and have a gameplan for how you can switch back to your original configuration if something goes terribly wrong. Maybe you could run a side-by-side comparison of your current process along with your proposed shiny new one, to see if you have any 'if it aint broke don't fix it' areas to retain.

The whole point of CI, in general, is to make you ship code quickly whilst maintaining a high level of quality. In many cases, rushing to use a new process will certainly allow you to ship code quicker, but will it be stable or riddled with bugs? It's very hard to say and for me, jumping blindly into this with production code, without a test or even staging run before it reaches customers is too high a risk.

I totally believe in the old maxim that says 'shipping is a feature.' I also think that your product becoming drastically more unstable because you rushed into a sexy new process should not be.

Remember, slow and steady builds the pipeline.