This is far from an article against distributed systems, and advanced engineering topics, this is a letter about overengineering. Product Design can solve most of your pains.
Ok, that title sounds really like clickbait, but it isn't.
What do your users need?
The first time you started to think about implementing any kind of distributed mechanism in your company, you probably thought that because your manager said that some feature is slow, users are complaining about slow loadings, features eventually consistent...
And then you go ahead and implement a huge collision-free distributed tree with 300 workers on a self-implemented job scheduler that shares info between their processes.
The results? Your system becomes faster by 20/30%, but your manager is still unsatisfied, you have wasted 2 months, and your page still slow. And then when you and your teammates are developing new features you realize a lot of problems with concurrency, changing the same data, weird database locks that nobody knows how to solve...So, your request went from 60 secs to something around 50 and a whole lot of new bugs were introduced. So, you still don't solve what your customer needs and shipped a few new bugs 😀.
Another downside of these approaches that highly increases the throughput on the code layer is that your services (Like Databases, Cache, and Message Queues) need to support that, even if it just pays more, as long as you go with your amount of data, you'll see how your data modeling is performing, maybe pay more for a large machine won't solve your problems. You'll have a large trash data in the disk of old logs, you'll need to understand your most expensive queries and try to create indexes. If you just increase the throughput of your server something known as backpressure will occur in your system.
How to solve that with Product Design
You can make some tweaks to how the customer interacts with your application, for example:
In that example below the user is waiting for the whole flow complete to get their response, you have a lot of layers for that get success:
- Your API
- Your Database
- External Payment Provider
- Email Service
- All the Routing/DNS stuff around your application
So, even if you manage to decrease the latency by 50%, your user will wait 30 secs, and this is a long time.
How you can improve?
Ps: I put a redis here, but can be any other job scheduler.
What will change on your project? Imagine that instead of your user waiting 30 seconds for a confirmed/refused UI on their screen, he can get a Pre-Approved or Refused other in a few seconds.
You can use the simplest option than implement a new service. Keep in mind that every running system can be a job scheduler, you can just use nodejs worker threads, Redis, or even your DB as a job scheduler, check PGBoss for example.
You can do that instead of waiting for the whole flow to return your response to the browser:
- Handle the Main HTTP Request
- Verify errors that always will refuse the purchase, such as invalid credit card numbers.
- Save on the DB the order with the actual status, which can be pre-approved or refused.
- Trigger a Job to make the purchase on your external provider.
- Handle the Job
- Call the external provider
- Update the status on your DB
- Trigger an email sending the confirmation of the order for your customer.
- On your frontend
- You show a message that a email will be sent
- Or do a Short Polling (something around 5/10secs to get the new status of the order.)
Conclusion
I hope you understood my idea with this article, the main keywords are:
- Simplify things
- Beware of backpressure
- Don't create more entropy in your codebase
- Opine on Product Design
Top comments (0)