DEV Community

Cover image for PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows
Rafael Pierre
Rafael Pierre

Posted on

PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

PyJaws enables declaring Databricks Jobs and Workflows as Python code, allowing for:

  • Code Linting (e.g. with Flake or Ruff)
  • Formatting (e.g. with Black)
  • Parameter Validation
  • Modularity and reusability

In addition to those, PyJaws also provides some nice features such as cycle detection out of the box.

Folks who have used Python-based orchestration tools such as Apache Airflow, Luigi and Mage will be familiar with the concepts and the API if PyJaws.

PyJaws leverages some existing libraries in order to allow for modularisation, reusability and validation, such as:

  • Click - for providing a rich CLI functionality
  • Pydantic - for efficient parameter validation
  • NetworkX - for Graph and Cycle Detection features
  • Jinja2 - for templating

Check it out: https://github.com/rafaelpierre/pyjaws

Top comments (0)