It has been a year and a half since I embarked on the journey of bringing my long-standing idea to life. This idea had been lying dormant on my list of projects, but I finally decided to develop a database gateway tailored for building data-driven applications. Yes, you read it correctly: a database gateway. Allow me to explain briefly.
The entire project has been carried out in stealth mode, with only a select few witnessing the product demo. During the initial phase, I dedicated three intense months to delve into various database technologies, meticulously examining their internals and understanding the specific needs of users. This extensive reading, research, and investigation allowed me to gain valuable insights and prioritize what truly matters. As a developer myself, I focused on addressing the challenges and demands that I encounter daily, ensuring that the final product caters to my needs effectively.
After spending quite some time on the initial PoC and then the MVP, I had the opportunity to participate in the Nordic PgDay 2023, where I gathered substantial validation for my idea. Throughout the event, I diligently took note of (almost) forty distinct pain points, which I identified while attending talks, workshops, and engaging with professionals possessing diverse levels of expertise. Among the attendees were PostgreSQL maintainers, developers, DBAs, and enthusiasts. I intend to share my findings in a separate post. However, for now, let’s shift our attention to exploring the core aspects of the idea itself.
The Problem
In the vast database world, we encounter numerous layers and actors, but it appears that some areas receive less attention compared to other segments of software engineering and development. One particular aspect of concern is the realm of APIs.
When it comes to HTTP-based APIs, such as REST, gRPC, and GraphQL, we find a plethora of well-established technologies that effectively expose data and functionality to the outside world. An extensive array of supporting tools, including frameworks, webservers, documentation, testing, mocking, versioning, monitoring, security, gateways, load-balancers, and analytics, have evolved to complement these APIs. On the other hand, when we delve into the database realm, we observe that some tools are not as mature as their counterparts. This disparity can be attributed to the unique nature of the databases and varying degrees of attention received by different aspects and parties.
An exemplary illustration of well-developed API-related tools lies in API gateways. These software components sit between clients and servers, offering a range of functionalities like security, monitoring, analytics, load-balancing, caching, and scalability support. Over the years, numerous API gateways have matured and become highly efficient. They excel at intercepting traffic and performing their magic to enhance the API experience.
It is precisely at this point where the concept of a database gateway emerges. Inspired by the success of API gateways, a database gateway serves as an intermediary, bridging the gap between users and the database. Just as API gateways offer a wealth of benefits for APIs, a well-designed database gateway could potentially address the current gaps and bring significant improvements in the database realm. By emulating the magic of API gateways for databases, we can explore new possibilities and unlock greater potential in data management and utilization.
Introducing GatewayD: The Database Gateway Solution
The solution to address the existing database challenges involves the implementation of a database gateway, strategically positioned between clients and databases. With its traffic-intercepting capabilities, this software piece performs its magic to bring forth a range of benefits. From ensuring security and monitoring to enabling analytics, load-balancing, caching, scalability, and various other use cases, the database gateway emerges as a versatile tool.
GatewayD, akin to an API gateway for databases, embodies the very essence of this solution, aiming to tackle the issues mentioned earlier. Comprising several key components, as shown in the below high-level architecture, GatewayD sets out to revolutionize the database landscape:
GatewayD Core (CLI): At the heart of the architecture lies the GatewayD core, positioned between clients and databases, effectively intercepting traffic. Serving as a database-agnostic TCP proxy, it is meticulously crafted in Go and designed to be extensible through plugins. This core component has the exceptional capability to serve multiple databases seamlessly, simplifying the management of plugins through automated installation and management.
GatewayD Plugins: GatewayD boasts an array of dynamic plugins that enhance its functionality. Crafted in Go, these plugins are loaded at runtime, catering to diverse use cases. The following section will provide further insights into the diverse possibilities these plugins offer.
GatewayD SDK: To facilitate the development of plugins, the GatewayD SDK provides a collection of Go utility libraries. These robust tools enable developers to create plugins efficiently, fostering innovation and versatility.
GatewayD Plugin Template(s): Emphasizing efficiency and simplicity, GatewayD offers template projects. Developers can rapidly bootstrap and develop Go-based plugins using these templates. Leveraging the power of gRPC, a GatewayD plugin can be written in any language supporting gRPC. Presently, Go is fully supported, and a Python PoC template is also developed.
Unleashing the Power of GatewayD: Addressing Critical Database Issues
In a manner akin to API gateways, GatewayD empowers users through its plugin system, offering the flexibility to develop customized solutions to tackle pressing database challenges. By harnessing the potential of these plugins, GatewayD aims to address various crucial issues faced in the database world:
High Total Cost of Ownership (TCO): GatewayD seeks to alleviate the total cost of database ownership significantly. By automating tasks and reducing complexity, it curtails expenses related to maintenance, staff, on-premises hardware, cloud infrastructure, migration, and other associated costs. A streamlined approach helps optimize resource utilization and enhances operational efficiency, ultimately leading to cost savings.
Vendor Lock-In: Promoting vendor neutrality, GatewayD aims to support multiple databases, allowing seamless migration from one database to another. By facilitating a smooth transition between databases, it liberates users from vendor lock-in, empowering them with the freedom to choose the most suitable database solution for their needs.
Security and Privacy Enhancement: GatewayD puts a strong focus on bolstering database security and privacy. With capabilities to detect and prevent SQL injection attacks, encrypt data in transit, anonymize and manage data, among other features, it ensures that sensitive information remains well-protected, safeguarding against potential threats and unauthorized access.
Scalability Boost: Emphasizing the importance of scalability, GatewayD strives to enhance the performance of databases. Through scalable solutions, including horizontal scaling, it empowers databases to handle increasing workloads, ensuring smooth operations as demands grow.
Reducing Inherent Complexities (Black-Box Mentality): GatewayD is dedicated to simplifying the complexities often associated with databases. Its innovative approach automates tasks for developers, DBAs, and SREs, streamlining the database management process and making it more accessible to all stakeholders. By lifting the veil on the black-box mentality, GatewayD empowers users to have greater control and understanding of their databases.
GatewayD aims to revolutionize the database ecosystem, providing a powerful solution to elevate the efficiency, reliability, and extensibility of database operations. With its dynamic plugin architecture and extensive capabilities, GatewayD opens the door to a new era of streamlined database management and enhanced data utilization.
The GatewayD Features
GatewayD core boasts a wide range of features, making it a versatile and powerful TCP proxy:
Connection Pooling and Proxying: GatewayD efficiently pools connections to database servers, effectively proxying them to clients. Additionally, it manages connections to clients and proxies them to database servers. This intelligent pooling mechanism optimizes resource utilization and enhances performance.
Database-Agnostic: GatewayD is designed to be database-agnostic, offering seamless compatibility with various databases. PostgreSQL is the current focus, yet we have plans to support MySQL-based databases. This inherent flexibility enables easy extension to support different database systems, enhancing its adaptability.
Plugin-Based and Extensible: GatewayD’s innovative plugin-based architecture allows for limitless possibilities. Developers can craft custom plugins to expand the functionality of GatewayD, tailoring it to meet unique requirements and use cases. This extensibility ensures GatewayD remains future-proof and adaptable to evolving needs.
Multi-Tenancy Support: GatewayD excels at serving multiple databases simultaneously, providing a multi-tenant environment. This capability fosters efficient resource sharing, allowing users to manage multiple databases with ease.
Observability: GatewayD prioritizes observability, enabling users to gain insights into its internal workings and the functioning of its plugins. Comprehensive observability features, including logs, metrics, and tracing, empower users with valuable data to monitor and analyze GatewayD’s performance effectively.
So far I’ve implemented two plugins, and I am working on a few more. These plugins are developed for PostgreSQL, but they can be extended to other databases as well:
gatewayd-plugin-cache (OSS): Transparent caching in Redis for PostgreSQL with automatic cache invalidation based on incoming queries and TTL
gatewayd-plugin-cache-advanced (Enterprise): Transparent caching in Redis for PostgreSQL with advanced features like WAL-based invalidation
gatewayd-plugin-sql-ids-ips (Enterprise): Detecting and preventing SQL injection attack for PostgreSQL using deep learning and other techniques
The Competitive Landscape
In the realm of database gateway solutions, several notable competitors hold their ground, each with its unique use cases and advantages. Below, I’ll provide factual information on some prominent contenders that I’ve investigated:
- Connection Poolers:
PgBouncer: A lightweight connection pooler specifically designed for PostgreSQL. It excels at pooling connections to PostgreSQL but is limited to this database only. PgBouncer operates as a single-threaded solution and lacks extensibility and plugin support. Despite these limitations, it remains a popular choice for efficiently managing PostgreSQL connections, and providing valuable features.
Pgpool-II: Functioning as a middleware, Pgpool-II operates between PostgreSQL servers and database clients. It effectively pools connections to PostgreSQL, offering a range of other features. Similar to PgBouncer, it lacks extensibility and plugin support, and its scope is confined to PostgreSQL.
2. Database Proxies:
ProxySQL: As a high-performance SQL proxy, ProxySQL provides robust features but is not extensible and lacks plugin support. It exclusively supports MySQL, making it an excellent choice for MySQL users seeking enhanced performance and capabilities.
MaxScale: This database proxy supports multiple database servers, including MySQL and MariaDB. Notably, MaxScale stands out due to its extensibility, allowing for customization through C/C++ development. Its versatility makes it a preferred choice for users with heterogeneous database environments.
Acra: Positioned as a comprehensive database security suite, Acra offers encryption, access control, data masking, and intrusion detection features. Although it lacks extensibility and plugin support, it provides valuable security enhancements for both PostgreSQL and MySQL.
3. General Purpose Proxies:
- HAProxy: Renowned for its reliability, HAProxy provides high availability, load balancing, and proxying for TCP and HTTP-based applications. While not equipped with traditional plugins, HAProxy supports extensibility through Lua scripting. Furthermore, its database-agnostic nature allows seamless integration with various databases.
Let’s not overlook the remarkable contributions of Vitess and Citus, both of which have significantly enhanced MySQL and PostgreSQL’s scalability to unprecedented levels. These powerful extensions have revolutionized the way we handle data, opening up new horizons for scaling our databases like never before.
How-Tos
Install and Use GatewayD
GatewayD is available as a Docker image and can be installed using Docker. It can also be installed using the binary releases for Linux, macOS and Windows. It can also be built from source. Please refer to the installation guide for more information.
A typical use case is to use GatewayD as a transparent cache for PostgreSQL, which is explained in this guide.
GatewayD is configured using a YAML files. There are two configuration files: one for the core and one for the plugins. Please refer to the configuration guide for more information.
Develop Plugins
GatewayD plugins can be developed using the GatewayD Go plugin template. A Python plugin template is in the works. Please refer to the plugin developers guide for more information.
Contribute
GatewayD is an open-source project and contributions are welcome. Please refer to the contributing guide for more information. Proposals are also welcome.
Conclusion
I hope I could explain the idea behind GatewayD and how it can be used to solve some of the problems we face in the database world. I am working on a few more plugins and will share them in the coming weeks and months. I am also working on more features for GatewayD core. I’ll share more information about them in the coming weeks. The project is still in beta, and we need beta testers to help find issues.
GatewayD Cloud is also in the works and will be available soon-ish. It is a managed service for GatewayD that can be used to deploy and manage GatewayD instances. It can also be used to manage plugins and other related services. Stay tuned!
Top comments (0)