Simplify your automation scripts with a low-code task-runner

#showdev #productivity #devops #python

pypyr automation task-runner is a free open-source tool that lets you script sequential task workflow steps in a pipeline using yaml. You get conditional execution, loops, error handling & retries without writing custom code.

pypyr runs pipelines from a clear straightforward CLI. You can also use this as a library programatically from your own code - the pypyr API to run a pipeline is undemanding and clearly documented!

Why should you use a task-runner?

You know how your automation scripts usually start off with a couple of innocent lines of bash, or maybe python?

And then they get out of hand?

And then before you know it you've got a big clumsy monster that you really hope doesn't break because you're too scared to touch it?

And then you have to code & test the repetitive not-all-that-interesting plumbing of:

sequencing tasks
parse & validate cli input arguments
parse config files, write output files
handling errors
retrying commands on error
running a group of commands in a loop
invoking the cli from python or invoking python from the cli

Streamline automation with a task-runner

pypyr task-runner helps with all that. With pypyr you create your automation workflows in a human-friendly yaml pipeline.

Add loops, error-handling & automatic retries to your command & program sequences, without having to write & troubleshoot code to do so.
Create your own cli app, with your own custom command line arguments, without having to do all that arg parsing & validation yourself.
Control-of-flow where you conditionally execute or skip steps based upon input expressions & switches.
Easily work with config files like json, yaml or plain old text to parse & substitute values.
No-code to low-code way of sequencing commands & programs, while still having the power to drop into full-on Python if you need to express something more complicated.

Add automation to existing utilities without writing code

Here is an example pipeline to retry a curl command up to four times until it's successful, at 0.5 second intervals:

The reason I first wrote pypyr, is because when I'm writing automation scripts, I find myself jumping between bash/shell and a more advanced language like Python to get things done efficiently.

For example. . . sometimes a quick sed from the cli is quicker to implement, sometimes more advanced string parsing in Python is necessary - sometimes both things are necessary at different parts in the same automation sequence.

Some tools you need to invoke to automate your tasks only have CLIs, other tools only have APIs. . .

The famous Unix philosophy of building programs that do one thing and do them well has given us a lot of battle-tested amazing utilities that have been around for decades (think curl, grep, awk, sed. . .)

Why not use these, rather than reinvent the wheel? So the point of pypyr is to make it easier for you to stitch together those awesome utilities, do some of your own scripting in between them, and generally automate it all without the usual repetitive pain points.

Why use a task-runner if you could just code it all yourself instead?

The obvious hard-core developer response is:

yeah, I just code that myself, why bother with this?

And yes, of course, you totally can code all of the surrounding, boring, repetitive, not-all-that-interesting bits of automation code for retries and err handling etc. yourself. . .

But I'm sure you'd rather keep yourself busy with doing cool new algorithms or functional code, rather than get mired down in automation tasks!

Free & Open-source

The source code is here: https://github.com/pypyr/pypyr/.

If you want to contribute, even just a GitHub star is useful!

I wrote the 1st code for this years ago when I found myself having to solve the same sort of problems each time I wrote some sort of automation script. . . Over the years it's grown as I've found more & more uses for it and expanded upon it, and because it was open-source from day one pypyr has been blessed with great enhancements from superb contributors over the years who brought ideas, enhancements and fixes that I wouldn't even have thought of. Big shout out to you all!

The new pypyr technical documentation website source-code itself is also open-source, and packaged as a re-usable Hugo static site generator theme.

100% Test Coverage

From the beginning I've made an effort to keep 100% test coverage over both unit & integration tests, and this (although annoying to keep to that sometimes. . .) has been invaluable in preventing regressions & providing steady stability of the tool as it evolved, but let me not break my arm patting myself on the back here. . . this implementation detail is probably not all that interesting unless you're into code.