Jenkins to Github Action Workflows Migration: Applying the DRY principle

Basic knowledge on Jenkins and Github Actions is required.

Jenkins has been my DevOps team's CI/CD tool of choice for years, however we recently had to say goodbye and switch to GitHub Action Workflows. During the planning stage of our migration, we discovered a lot of redundancy in the Jenkins pipelines, so we devised a strategy to address this by utilizing the DRY concept.

This blog post explains the application of the "Don't Repeat Yourself" (DRY) software development principle, which helps us minimize the effort needed for the migration.

DRY is a key concept in programming that emphasizes the importance of reducing redundancy. Duplicating code is not just ugly it is bad practice because it would make your code hard to maintain. If you need to fix one part on your code, you have to do it on all other parts where it is repeated and often times that can cause issues whenever the fix is not done properly.

Keeping it DRY

Before writing any GitHub workflow, we took a closer look first into our Jenkins pipelines to identify areas we could apply the DRY principle. 

Duplicate Stages Across Multiple Pipelines

Here you can see an example of two separate Jenkins pipelines for deployment that accept the same parameter and have very similar stages. 

Pipeline A: DB Deployment

pipeline {
    agent any

    parameters {
     string(name: 'JIRA_TICKET', description: '', defaultValue: '')
    }

    stages {
        stage('Get JIRA Ticket Details') {
            steps {
                script{
                  echo 'Fetch Ticket Details from JIRA API'
                }
            }
        }

       stage('DB Deployment') {
            steps {
                script{
                   echo 'Logging in to DB'
                   echo 'Running Deployment Script'
                   echo 'Parsing logs for any ORA errors'
                }
            }
        }

       stage('JIRA Update') {
            steps {
                script{
                   echo 'Update JIRA Ticket with Deployment Result'
                }
            }
        } 
    }
}

Pipeline B: App Deployment

pipeline {
    agent any

    parameters {
     string(name: 'JIRA_TICKET', description: '', defaultValue: '')
    }

    stages {
        stage('Get JIRA Ticket Details') {
            steps {
                script{
                    echo 'Fetch Ticket Details from JIRA API'
                }
            }
        }
        stage('App Deployment') {
            steps {
                script{
                  echo 'Logging in to Server'
                  echo 'Running Deployment Script'
                }
            }
        } 
        stage('Trigger Post Deploy and HealthChecks') {
            steps {
                script{
                  echo 'Running post deploy and healthchecks'
                }
            }
        } 
        stage('JIRA Update') {
            steps {
                script{
                   echo 'Update JIRA Ticket with Deployment Result'
                }
            }
        } 
    }
}

The first and last stages, Stage Get JIRA Ticket Details and Stage JIRA Update, are similar and are quite repeated in other pipelines since we have a dependency on getting JIRA ticket field values to identify what to deploy and also keep track of deployment status.

Pipelines triggered frequently from other pipelines

We also identified which pipelines are triggered from other pipelines frequently, which are usually parameterized pipelines. Taking the above's example, we also have other pipelines that should trigger the same set of steps in Pipeline A. 

The Solution

To avoid duplication when we migrate to Github Workflows, below was our solution:

For similar stages across pipelines → Use Composite Actions

For pipelines triggered frequently from other pipelines → Use Reusable Workflows

In our case, we choose these solutions, which Github already offers, to achieve our goals below:

1. Simplified Migration
   Migrating existing Jenkins pipelines often involves translating Jenkinsfile scripts into Github Action Workflows. Composite actions and reusable workflows help manage complexity by isolating and reusing common logic

2. Reusability
   Mirrors Jenkins's shared libraries or reusable pipelines, easing the transition.

3. Maintainability
   Using shared logic in one place simplifies maintenance and troubleshooting.

Using Composite Actions ✨

Composite actions allow you to turn a step or set of steps into an action and reuse it on multiple GitHub workflows. In Jenkins, this is similar to a Jenkins Shared Library.

Example Composite Action

Here is an example composite action created for the Get JIRA Ticket Details stage above.

This composite action handles getting information for a build given a JIRA Ticket number and importing the information fetched from an API as environment variables which can be used multiple times across other workflows.

On composite-repo repository, under a folder called get-jira an action.yaml like below is created:

name: 'Get JIRA Ticket Details'
description: 'Fetch JIRA Ticket details'
inputs:
  jira_ticket:
    description: JIRA Ticket
    required: true
  jira_user:
    description: JIRA Username
    required: true
  jira_pass:
    description: JIRA Password
    required: true

runs:
  using: 'composite'
  steps:
    - name: Get JIRA Ticket Details
      id: get_jira
      shell: bash
      run: | 
       curl -u ${{ inputs.jira_user }}:${{ inputs.jira_pass }} \
       "$jira_api/${{ inputs.jira_ticket }}" > jira.json

       summary=$(cat jira.json | jq -r '.fields.summary')
       status=$(cat jira.json | jq -r '.fields.status')
       artifacts=$(cat jira.json | jq -r '.fields.artifact')

       echo "SUMMARY=$summary" >> $GITHUB_ENV
       echo "STATUS=$status" >> $GITHUB_ENV
       echo "ARTIFACTS=$artifacts" >> $GITHUB_ENV

Here’s how the composite action above is used on a workflow. Note that get-jira is the name of the action and v1 is the branch name or tag on composite-repo repository.

name: Deployment Pipeline
on: 
  workflow_dispatch:
    inputs:
      JIRA_TICKET:
        description: JIRA Ticket number
        required: true
        type: string

jobs:
  Deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Get JIRA Ticket Fields
      uses: composite-repo/get-jira@v1
      with: 
        jira_ticket: ${{ inputs.JIRA_TICKET }}
        jira_user: ${{ secrets.JIRA_USER }}
        jira_pass: ${{ secrets.JIRA_PASS }}

Now that we have this composite action, we can add and make changes to this and all other workflows which uses this are automatically updated.

We also made other actions we can reuse aside from this one and and we ended up having like a tool box full of tools we can just use on any of our workflows. 🔧 🔨

Lessons Learned when creating Composite Actions 🎉

• A mono repo containing commonly used composite actions with the below structure can be created.

  |-action-1
  |  |-action.yaml
  |-action-2
  |  |-action.yaml

• Use branches or tags to version the composite actions for easy maintenance and testing.

• Avoid combining very different sets of steps into the same composite action, as it will become complicated to use and may require multiple optional parameters and conditions.

• Document actions for easy use. Tools like npalm/action-docs CLI can be used to automate the creation of documentation.

Using Reusable Workflows ✨

For Jenkins pipelines triggered frequently from other pipelines, we created reusable workflows.

Unlike Composite actions which can be run on steps, reusable workflows are run directly as jobs and can even use the matrix strategy option to trigger it multiple times.

In our case we have one main workflow which triggers multiple DB Deployment workflows for each JIRA ticket and environment, so we created a reusable workflow for that purpose.

Example Reusable Workflow: db_deploy.yaml

 name: DB Deployment
 on:
    workflow_call:
      inputs:
         JIRA_TICKET:
           description: Jira Ticket
           required: true
           type: string
         ENVIRONMENT:
           description: Environment
           required: true
           type: string

 jobs:
   Oracle-Deploy:
     runs-on: ubuntu-latest
     steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Get JIRA Ticket Fields
        uses: composite-repo/get-jira@v1
        with: 
          jira_ticket: ${{ inputs.JIRA_TICKET }}
          jira_user: ${{ secrets.JIRA_USER }}
          jira_pass: ${{ secrets.JIRA_PASS }}

      - name: Get DB Details
        id: get-db
        uses: composite-repo/get-db-details@v1
        with: 
          environment: ${{ inputs.ENVIRONMENT }}

      - name: DB Deployment
        uses: composite-repo/db-deploy@v1
        with:
          dbhost: ${{ steps.get-db.outputs.dbhost }}
          dbport: ${{ steps.get-db.outputs.dbport }}
          dbuser: ${{ secrets.DB_USER }}
          dbpass: ${{ secrets.DB_PASS }}
          dbscripts: ${{ env.ARTIFACTS }}

Here’s how the reusable workflow above is used on a workflow. Since we have multiple deployments for multiple JIRA tickets, a reusable workflow is used on a job with matrix strategy.

jobs:

  Oracle_Deploy:

    strategy:

      matrix: 

       jira: ${{ fromJSON(needs.jobname.outputs.tickets) }}

    uses: workflows-repo/.github/workflows/db_deploy.yaml@v1

    with:

      JIRA_TICKET: ${{ matrix.jira.ticket }}

      ENVIRONMENT: ${{ matrix.jira.environment }}

Lessons Learned when creating Reusable Workflows: 🎉

• Reusable workflows and composite actions are quite similar. Use reusable workflows to reuse an entire workflow with multiple jobs and steps, and use composite actions for small tasks that can be reused in a step.

• Too much reuse can lead to complications. Package related steps and jobs only into one reusable workflow and, for other purposes, create a separate one.

• Reusing of workflows is allowed up to 4 levels only. If your main workflow does not need to wait for the triggered reusable workflow to complete, you can use the github CLI gh workflow run <workflow name> on a step to trigger it using the workflow_dispatch event instead.

Conclusion

Composite Actions and Reusable Workflows can be used to avoid duplication and ease the migration from Jenkins to Github Actions.