Greetings, fellow developers! I hope you're all doing well. Today, I'm thrilled to delve into the realm of multi-tenant architecture and share my i...
For further actions, you may consider blocking this person and/or reporting abuse
There are many problems to this and one of the most important, it will create huge issues when you needs to scale and need to do some sort of load balancing.
You are keeping these database connections in an LRU Cache (in-memory in the same machine), if I am gonna add a load balance and balance my request using 4 instances what's the chance that my users request will always go to the same instance which actually have the connection in the cache.
Most of the cases you will end-up opening a new database connections, yeah you can use sticky sessions and make sure the request does not go to a different instance based on some sort of identifier, like tenantId or something like that but in this case you won't be able to balance the request properly in all the instances.
Will points out one problem and you will surely understand that, you really can't scale with this model.
Well, thanks for your comments and insights. First, I would like to clarify that this is just a beginner's guide and should not be viewed as a complete solution. This model can easily handle 1 million requests or users; after that, you will have to start designing strategies to handle the load, as is the case with every software. But you first have to release the software and get customer feedback. You can't make the software complex from the get-go to handle 10 million requests, which you won't get.
As for the scaling solution, what I have done previously is move the connections cache to a separate machine. Now, I have made my backend truly stateless, and it can just request the connection from the other machine whenever it needs. But I repeat again, you should consider these problems after you truly hit the bottleneck, which may take time. And if you do reach the bottleneck, which means your software is earning you money, you can dedicate more effort since you will be getting an incentive for that.
Interesting article, I have worked on similar project in different language. I'm always interested in understanding how other people design the multi tenant architecture.
Can you explain about the "truly stateless backend" part ?
In the current backend architecture, a state is maintained as it keeps track of connections using an LRU (Least Recently Used) cache. If another instance of this backend is spun up, it will manage its own LRU cache. To address this issue, I separated the repository part of the code and created a separate Node.js ( later migrated to golang for speed ) project for it. This repository process now manages its own LRU cache and is responsible solely for running queries sent from the "stateless" backend, which contains the business logic.
This approach allows me to spin up multiple instances of the business logic process without concerns about managing connections. In a production environment, I only require two instances of the repository process. When I mentioned "truly stateless" I was referring to the backend handling the business logic.
I'd also appreciate hearing about your approach to this problem. What logic did you follow, and do you have any valuable insights? There's always room for improvement, and I believe there may be better solutions that I haven't yet discovered.
When you mention 1 million request/users using this way, would love to know a few things:
What's your request frequency (and how many requests/sec you are handling)?
What was the average latency for request processing?
Now, I am considering your repository process implementation, which is great but still it's hard to scale out your server for database request processing, you still can scale up but.
It all comes down to one very important thing and that's how many connections you have opened for the database and how many you are actually wasting.
We build multi-tenant system with sort of a hybrid approach, we design the system in a way where it understand 2 ways of communication so we can separate tenant based on their requirements and let other system work however it is.
I will explain it now: let's imagine we got a tenant (we are into B2B, business have their own taste for security based on the money), if tenant is interested in fully dedicated system so it's just easy now, but when it comes to shared we keep the users in the same database (protected using Row Level Security) and whenever required we can move them a few tenants to any other instance (we aren't handling every tenant directly from the same instance), based on the tenant id we decides which server to hit based on the centralized config which is managed differently (you can think on an INIT request via an API)
It's sort of an Application level sharding by tenant.
This solution helps us to scale out and scale up (horizontally or vertically respectively).
Thank you for sharing how your organization has handled multi-tenancy. The product I was building was a B2B CRM app, and the client's requirement was to have separate databases for each tenant, rather than separate code bases. Otherwise, I would have simply written code to manage a single connection. Your provided method can be a viable option for those without constraints unlike mine, as it offers easier management and scalability.
Regarding your questions:
First, it's important to establish that our backend primarily involved CRUD (Create, Read, Update, Delete) operations, with no intensive computations. All tasks were I/O operations. In its peak state, the monolithic architecture of the backend could handle up to 1000 requests per second (RPS) with an average latency of 300 milliseconds. Regarding database connections, we created five connections for each tenant, though unfortunately, I didn't measure the extent of connection wastage at that time.
I think your blog is a great source for understanding what really goes into building a multi tenant system initially, but what I observed in my experience is building multi tenant systems are tough and mostly depends on the type of operations you are going to handle and how are you partitioning the tenants later on in the architecture.
Actually there are many tradeoff and that's one of the reasons we provide both of the options to our clients and it being a very simple way to handle multiple tenants becomes cost effective and also scalable.
In the world of software development, trade-offs are an integral part of the process, especially when building software intended for use by millions of users. These trade-offs encompass various aspects, ranging from the choice of programming language to the selection of frameworks and databases.
For instance, when opting for Node.js, you might trade execution speed for the benefits of asynchronous programming. Conversely, languages like Golang might require trading speed for improved developer experience, particularly when defining request bodies for each request. (This is one reason I've turned to gRPC.)
Similar trade-offs apply to Rust and many other technology choices. Therefore, conducting a comprehensive analysis before embarking on software development is essential. This analysis should not only consider your own preferences but also the needs and priorities of stakeholders involved.
Ultimately, the key takeaway is that individuals and teams embarking on software development projects should carefully weigh their requirements and constraints. Decisions made at the outset can have a profound impact on the software's performance and long-term maintainability. Therefore, thoughtful consideration of these factors is crucial before commencing any software development endeavor.
This is an amazing article and I am really happy I could validate my implementation.
The scaling up caught my eye. Your approach to moving the connections cache to a separate machine is intriguing, and I'm interested in understanding how you implemented this in practice, especially given the common constraints in Javascript environments. Typically, in Javascript, we use Plain Old JavaScript Objects (POJOs) for data transfer, which aren't designed to maintain state or handle complex behaviors that might be needed for a distributed cache system. Moreover, when scaling up and using multiple containers or instances, maintaining a consistent state across them can be challenging. Could you elaborate on how you managed to effectively transfer and synchronize the connections cache across different machines while ensuring consistency and reliability?
Hey sorry for the late reply what I meant y taking connections to seperate machine is that all the db related code was moved to another machine where I cached the result of db operations as well so a single machine just for querying
Thank you Ram. And i speculate, if we were to scale that single machine, we'd prefer horizontal scaling.
Correct
Nice article on multiple database solutions! If you are interested in achieving multi-tenancy using a single database, you should check out how to do it with Prisma:
Multi-tenant implementation approaches with Prisma and ZenStack
JS for ZenStack ・ Mar 18
What do you guys think about this approach. Multi-tenant using node and mongoDB