GitHub Actions: a New Hope in YAML Programming Wasteland

The main superpower of a programmer is her ability to automate almost everything. That's where GitHub Actions shines... if you can write with confidence a GitHub Workflow in YAML to solve your particular problem. That's a big if. But if you can't, there is still hope.

Let me introduce you to an open-source project started by Piotr Krzemiński and to which I contributed a lot in the last months 👨🏻‍💻

What does github-actions-kotlin-dsl?

github-actions-kotlin-dsl(*) is a library that allow you to generate a GitHub Workflow for Kotlin Actions in a type-safe way.

(*) I know that's quite a mouthful. I personally call it github-actions.kts

1) Edit the script in IntelliJ IDEA Community Edition which is free and awesome
2) Put the script in the folder .github/workflows, make it executable
3) Install Kotlin
4) github-actions-kotlin-dsl is the library you use to build a GitHub Workflow object
5) At the end, you call workflow.writeToFile()
6) which generates .github/workflows/$NAME$.yaml which you can then run on GitHub Actions.

You can see underlined in blue that the main components of a YAML workflow have a striaghtforward Kotlin equivalent: workflow() Push() job() uses() run() GithubActionNameVxxx()

To actually run your first Kotlin Workflow, read the friendly documentation

At that point, you may ask yourself:

Wait but why 🤔 ?

GitHub Actions is a Wonderful Service But...

The main superpower of a programmer is her ability to automate almost everything.

Since that's precisely what GitHub Actions is for, all good right?

Unfortunately most people struggle to write a GitHub workflow solving their particular problem: And that's because YAML is a bad programming language - see below.

So instead we resort to copy/paste programming, trying to find someone that has already spent the time writing a workflow to solve a problem hopefully similar to ours.

Don't get me wrong, copy/paste programming is better than nothing, and I have written myself a guide to get a CI up and running in no time for Java/Kotlin developers

How do I setup GitHub Actions for my Gradle or Android project?

Jean-Michel 🕵🏻‍♂️ Fayard ・ Sep 16 '20

#actionshackathon #tutorial #kotlin #android

But we are missing out if we can't automate our particular problems.

So what's the issue?

Github's YAML is a Bad Programming Language

Average YAML developer trying to get things done, loosing ten minutes or more between each iteration.

YAML in general is just a different way to write JSON.

With a different set of issues...

But GitHub has extended YAML to something we should recognize as a programming language, according to the principle of Duck Typing.

*🐥 GitHub YAML as a programming language: 🐥 *

• it allows to write arbitrary scripts
• it runs on abritrary VM on Azure
• ...with an arbitrary number of jobs and commands
• ...with a shitload of plugins written in a arbitrary language, and which can be configured in all sort of ways
• ...with variables and secrets
• ...with for loops - a.k.a. strategy matrix
• ...with boolean logic - run this job only if the previous was successfull, run this action only on error
• ...with powerful GitHib expressions - a.k.a an eval function
• ...

If it walks like a duck and it quacks like a duck, then it must be a duck.

The issue here is that YAML was never designed for this

Therefore support in your IDE is quite poor. They do try to help but it's nowhere as good as with an actual programming language. Also the reason that YAML looks fancier than JSON is that its rules are less simple, but which means you are more likely to screw up.

Most importantly:

GitHub Action's edit-compile-run feedback loop is super slow

What you can't see in the picture above is that the GitHub Actions YAML developer might be wasting 10 minutes or more for each iteration of her script!

So is there a way out of the YAML wasteland?

Type-Safe Workflows

In YAML, an incorrect syntax looks very much like a correct one.

on:
  cron: '7 42 * * 7'
  schedule: '7 42 * * 7'  
  cron:
    schedule: '7 42 * * 7'
  schedule:
    cron:  '7 42 * * 7'    
  schedule:  
   - cron: '7 42 * * 7'

Which syntax is the correct one?

This time there is no debate. The simple existence of a type system makes all sort of tiny mistakes disappear!

The library also guarantees that whatever content it produces is valid YAML. No more waiting for the CI to produce a MalformedYamlError...

In addition there are runtime checks that will fail on you now and quickly, instead of failing slowly and later.

name: Hello: World!  
on:  
  pr:  
  schedule:  
   - cron: '7 42 * * 7'  
jobs:  
  "git checkout":  
    runs-on: "ubuntu_latest"  
    steps:  
      - id: step-0  
        name: First: git checkout
        uses: actions/checkout@v3

Can you find the 7 errors in the workflow above?

Let's have a look at the Kotlin equivalent

val workflow = workflow(  
    name = "Hello: World",  // 1. runtime error: semicolon not allowed
    on = listOf(  
        Schedule(listOf(
            // 2. named arguments 07:42 instead of 42:07 in the YAML
            // 3. runtime error: Field 'dayWeek' outside of range 0..6
            Cron(minute = "42", hour = "7", dayWeek = "7"))
        ),  
        PR(), // 4. compile error: should be PullRequest() 
    ),  
    sourceFile = __FILE__.toPath(),  
) {
    job(
    id = "git checkout",  // 5. runtime error: space not allowed in a job id
    runsOn = ubuntu_latest // 6. compile error: should be UbuntuLatest
    ) {  
        // 7. runtime error: semicolon not allowed here
        uses(name = "First: Check out", action = CheckoutV3())  
    }  
}

By just compiling and running the script we can prevent plenty of small errors that would have wasted each 10 minutes of our time on the CI.

Type-Safe Actions

Have a look a how this action is configured:

steps:  
  - id: step-0  
    name: git check out  
    uses: actions/checkout@v4  
    with:  
      branch: main  
      fetch-depth: 0

There are three issues here:

• branch looks like a valid parameter, but it isn't
• I have no idea what the magic value for fetch-depth means
• v4 doesn't exist yet

Let's see the Kotlin version:

• Here we see clearly that branch is wrong, and auto-complete tells us to use ref instead
• F1: Documentation allows us to see the documentation, and the special values have a name.
• CheckoutV4 wouldn't even compile

At the time I write those words, we have 89 supported action wrappers

Type-Safe Expressions

GitHub Actions also have a powerful syntax for expressions.

And I think you get it now: we havre a type-safe alternative for them too:

Read more at: https://krzema12.github.io/github-actions-kotlin-dsl/user-guide/type-safe-expressions/

Migrate Automatically Your Existing Workflows

So now I can see what you are thinking:

This is pretty cool, and I wished I had that when I got started.
But now I have YAML workflow that work,
and I don't want to restart from scratch.

I hear you.
I had the same concern.

And then a diabolic idea was born in my mind:

Right so we can have a Kotlin script that generates its YAML version.
But could we take the existing YAML and generate the Kotlin script that would generate itself? A bit like a child who would give birth to its parent?

As it turns out it's possible, and after some intense hacking, the script-generator was born. It allows you to automate your migration to Kotlin.

You run something like:

$ ./gradlew :script-generator:run --args /path/to/.github/workflows/build.yaml
Kotlin script written to build.main.kts 
Run it with: ./build.main.kts 
The resulting YAML file with be available at build.yaml

You can then make sure the new YAML is equivalent to the old one by doing a semantic diff with https://yamldiff.com/

I won't go in all the details: read the friendly documentation

Behind the scenes: the Wrapper Generator

When I discovered the project, there were maybe a dozen of actions that were supported by the library.

My main concern was: how do you scale that up?

There are dozens and dozens of GitHub Actions. And they have new release adding or deprecating parameters all the time.

The answer: the maintainer Piotr Krzemiński and I introduced automatic generation of the action wrappers.

• The wrapper generator... takes an action like Vampire/setup-wsl@1
• It downloads its action.yml file (required by GitHub), which contains all the inputs and outputs parameters along with their description
• From those two files, we generate a wrapper for this action the file vampire.SetupWslV1
• This generation is done with the KotlinPoet library. That's also what powers the script-generator from the previous paragraph.

Creating this wrapper generator was a lot of work but it is worth it:

At the time I write those words, we have 89 supported action wrappers

Behind the scenes: GitHub Actions

The other thing that allows us to scale up with our limited time to the challenge is: GitHub Actions itself.

We are the first users of our library, and leverage the power of GitHub Actions:

• to check whether some existing wrappers needs to be udpated (new parameter for example).
• to check wheter a new major version is available for an existing wrapper (V2 to V3).
• to automate the release of our library.
• as the CI on our PRs

For the motivated, our actions are available at: https://github.com/krzema12/github-actions-kotlin-dsl/tree/main/.github/workflows

This really hits home for me: what a superb asset GitHub Actions can be if you can feel confident that you can write a workflow to solve your particular problem.

FAQ

What if I Don't Know Kotlin?

I don't think the choice of programming language matters, as long that it's not Bash. I mean that any modern programming language with good IDE support would be up to the task.

We are doing only basic stuff here. We build a Workflow object by calling workflow() then job then run("my command") or uses(MyActionWithParameters(....))

The only thing is that you need to install IntelliJ IDEA Community Edition but it's free and awesome.

Any drawbacks?

The YAML and the Kotlin will get out of sync if you don't run the Kotlin script before committing it, if you forget to commit the YAML file, or if you edit the YAML file directly. We have a consistency check to fail early when that's the case. Just use workflow.writeToFile(addConsistencyCheck = true). The trade-off is that the CI will be something like 15s slower.

GitHub's YAML has a huge number of features so you may stumble upon one not supported by the library yet. We have a type-unsafe alternative to compensante those gaps. You can also look at open issues or create your own.

IntelliJ does not support Kotlin Scripts as well it does normal Kotlin programming. Hopefully this gets better with time.

Compiling and running a Kotlin script is slower than you are used to with an interpreted language like Python or JavaScript. But it's still two orders of magnitude faster than having to commit, push and wait 10 minutes on the CI.

If you use it in a team, you have to make sure that your colleagues undertand that the YAML file was generated by the Kotlin script. Mind you the first YAML line tells it: # This file was generated using Kotlin DSL (.github/workflows/hello_world_workflow.main.kts).

You are welcome to send them a link to my article :)

I'm a GitHub Action maintainer, can I help?

Yes!

The one thing missing from the action.yml spec is the type of each input parameter. We are currently adding this information in our repository but it would be much better if it could live in your own repository in a action-types.yml file as some nice earlly adopters already do.

Please see https://github.com/krzema12/github-actions-typing

Links

• GitHub: https://github.com/krzema12/github-actions-kotlin-dsl
• Documentation: https://krzema12.github.io/github-actions-kotlin-dsl/

The title for this article was stolen from Michel's Pardo talk Kotlin: a new Hope in Java 6 Wasteland which made want to switch to Kotlin back in the days.

The End 🔚

That's all I have friends. Thanks a lot of you made is this far.

I'm especially happy to be back writing articles, something I didn't manage to do since almost one year.

I would love to hear from your feedback if you give it a try.