Supermetrics retrieves data from over a hundred services like Google Analytics API and Facebook Graph API so our users can conveniently combine and analyze their scattered marketing data in one place. We add new data sources all the time, which means we're constantly integrating with new APIs.
Building these integrations, or connectors, used to take up a lot of development time. But now that we have created a new technology called Codeless Connectors, we have accelerated connector development tremendously.
In this article, I explain how we built connectors in the past and how we streamlined the process with Codeless Connectors.
Lead time down from two months to two weeks
Simply put, a connector is an API integration. It's a solution that fetches data from various sources and processes it to match the user's request before sending it to a destination like Google Data Studio, Microsoft Excel, or Snowflake.
In the past, building a new connector has been a development-intensive process. Each connector was developed separately as a coded connector with a particular API. It could take up to two months to get these coded connectors launched to our customers.
With Codeless Connectors, connector development takes, on average, only a couple of weeks. To fully understand where this huge efficiency came from, we must look at the old connector development process.
Coding connectors was slow and repetitive
Before adding a new connector to our products, we had to implement the target API first. Each Coded Connector build began with the same list of steps regardless of its data source. They included simple components like defining the field list but more development-intensive processes, too, like coding the logic for authentication and report types.
We needed to determine logic for what data to fetch and where to fetch it, as well as validation that the things we got were exactly what we wanted and potential retry logic. These components repeated in each connector, and with Coded Connectors, there wasn't a good way to unify this logic. We then needed to write a lot of boilerplate code before implementing the API. The boilerplate was a time sink and a repetitive manual chore for developers.
It wasn't just the boilerplate that took up developers' time. Each component required manual, unit, and integration testing and peer reviews before deployment to production. Even small changes like fixing a typo in a field's description required the full process, from a pull request and a peer review to making a deployment to production. And if something had gone wrong, reverting changes also required a separate deployment.
While the review and deployment process didn't take too much time for a single component, a connector contains more than one component. When you went through this process for loads of components, the time it took started to add up, making lead times for connectors longer.
Enter Codeless Connectors
We reimagined our approach to connector building to speed up product development. The result was a piece of entirely new technology for running and building connectors.
Codeless Connectors, a shared codebase for connectors
The Supermetrics Codeless Connectors is our new platform for running connectors. Instead of coding each connector separately, we have all connectors sharing the same codebase.
The Codeless Connectors relies on reusable components and allows us to eliminate the repetitive boilerplate and reuse the code for multiple connectors. The shared codebase allows us to build features only once and make them available to all connectors running on the platform. It works for bug fixes, too. Fix a bug in one connector, and you fix it in all of them.
Having generic classes means we can't program the connector behavior in the classes themselves. Instead, we need a different way to provide the execution logic for the connector. And here's where the Connector Builder comes in.
Connector Builder, a no-code configuration wizard
We create new Codeless Connectors with the Supermetrics Connector Builder, a no-code tool that helps us create and manage configurations for each connector.
The connector configuration is a specialized JSON file that gives our classes instructions on how to operate. Elements that used to be a part of the connector code, like authentication logic, data fetching, and response validation, are now part of the connector configuration. In a sense, what makes a connector a connector is now in the configuration instead of the codebase.
We store these configurations in a database. When the connector is in use, we load the configuration from the database and use it to instruct the Codeless Connectors code to perform the things any connector would do.
While writing a configuration JSON from scratch is possible, it might not be the best approach. The Connector Builder is a tool to help create and manage configurations, and it can function as a step-by-step configuration wizard that allows you to create the JSON by filling in the required fields. The result is a workable configuration for your chosen connector.
The Connector Builder also makes it easy to duplicate configurations for new versions. And if you need to make under-the-hood changes to the JSON, it offers you a handy JSON editor view where you can see the entire configuration and make changes as you wish. To help you make sure the changes you make are valid, the Connector Builder has a built-in validator that helps you spot errors early.
Configuration eliminates unnecessary deployments
The Connector Builder is an enormous time saver as it allows us to safely bypass the heavy test-review-deploy process for most changes. Storing the configurations in a database allows us to deploy code less often. Since the configuration dictates connector behavior, we can make changes without deployments.
For example, adding a new report type doesn't require us to code it in. Since the report type classes already exist, we can avoid a potentially lengthy code review process.
We still want to review the configurations, but those reviews are speedier and more straightforward than code reviews. We also have an automated validation in place, which ensures all inputs conform to a schema.
Developing a new report type in the old way could take up to two weeks. With the new configuration approach, it only takes about an hour or two. And since a connector usually includes multiple report types, you can imagine how the time saved adds up.
The no-code solution saves developer bandwidth
Anyone who reads and understands API documentation can build a connector with the Connector Builder. The UI-assisted configuration creation allows non-developers to create connectors simply by following the wizard's instructions and the target API's documentation.
We now rely on a team of Connector Specialists responsible for building new connectors. Instead of coding each connector by hand, the development team can focus on maintaining and developing the codebase for the Codeless Connectors and the Connector Builder.
Adding new features to the Codeless Connectors still requires developer resources from time to time. But as all the code is shared, we only need to implement a new feature once, and it instantly becomes available for all Codeless Connectors. Even with new features, the time spent on development per connector is still marginal compared to how it used to be.
Next, our team is looking to evolve connector building further through Hybrid Connectors. They would combine the easy customization of Coded Connectors with the reusability and fast development of the Connector Builder.
– – – –
Would you like to work with us?
Explore developer careers at Supermetrics 🚀
Top comments (0)