How we Learned to Stop Worrying and Love Data Movement with Airbyte

Faros Community Edition (CE) is an open-source engineering operations platform that connects the dots between all your operational data sources for a single-pane view across the software development life cycle. Airbyte powers Faros CE.

In this post, we will explain:

1. Why Faros CE needs ELT
2. Why we chose Airbyte
3. How we use Airbyte and contribute back
4. How we let our users unleash even more of Airbyte’s power

1. Why Faros CE needs ELT

Faros CE provides API, BI, and automation layers for engineering operations data (from Pull Requests to Incidents). With Faros CE, you can answer questions about bottlenecks in your CI/CD pipelines, issues in your onboarding processes, progress towards your goals, with data that is fresh, connected, queryable, and most importantly, all in one place.

But before any of this can be achieved, data needs to be ingested into the Faros CE platform from many different sources (GitHub, Jira, CircleCI, PagerDuty, …) in order to properly connect and analyze that data. Faros CE decided to leverage Airbyte for that ELT.

2. Why we chose Airbyte

We didn’t want to spend our energy working on data movement. Instead, we want to focus on our data schema and its useful applications for our community. We hence had to pick a data integration platform.

There are 3 reasons Airbyte stood out:

• Airbyte has a vibrant community; just check those stats out (powered by Faros CE)!
• Airbyte has become the de-facto standard for data movement
• Airbyte already has a rich catalog of connectors that we can leverage (GitHub for example)

3. How Faros CE integrates Airbyte and how we contribute back

To integrate Airbyte, we first created the Faros destination connector. It knows how to send data from sources into the platform, and takes care of mapping the various entities being extracted against our rich, connected data schema. Because of the great Airbyte specification, that code is compact and focused on the value Faros CE provides (i.e. that mapping).

Second, we contributed to source connectors important to Faros CE (Jira) or new source connectors that didn’t exist yet (Jenkins, Customer.io, Harness, VictorOps, PagerDuty). Since we use Typescript, we decided to contribute our Typescript CDK. It is our way to strengthen the community while benefiting from it!

Finally, whether it is through docker-compose (local) or plural.sh (cloud), Airbyte is deployed as part of Faros CE, with pre-configured connections. Our users only need to enter credentials to their source systems, and can have data ingested in the platform in a matter of minutes!

4. How we let our users unleash even more of Airbyte’s power

Our users have direct access to the Airbyte instance that launched as part of their Faros CE platform. They have therefore access to its full feature set! In particular, they can create custom sources or contribute new sources to the Airbyte community with minimal effort!

Airbyte is the bedrock on top of which Faros CE is built. The product and its community enables us to focus on our core value. We are proud members of the Airbyte community.