I am Dumb
Well, whenever we work in our local system everything works as butter. That is why we call "No better place than 127.0.0.1" but WAKE UP TO THE REALITY
Well things not always work in production as expected. Mostly when you are running multiple instances of your application.
🚀 As you can see that if multiple instances of our application are running and let's say that our client make a request to mark a user as a paid user in our DB.
- Client will request our server
- Request will reach at our load balancer
- And one of the instance will get the request and will make a write query into our DB
It seems alright right? No problem till now right.
Well, yes till now there is no problem. But what if we want to write a business logic like:-
- fetch the user from the DB
- check if user is a free user or already paid
- if free, then mark it paid and save in the db
- if paid, send "Already paid" in the response.
⚡️ As we know that (assume we are using MySQL here) MySQL DBs are ACID compliant which means any query will be atomic and isolated. which means MySQL query will be run in atomic way, either it will pass or fail. But it will not quit in between.
🤔 But there is one issue here. Think, Think....
- Step 1: We are fetching user (Atomic transaction)
- Step 2: Running some business logic in the code
- Step 3: Updating the MySQL record if user not paid (Atomic transaction)
What will happen if at Step 2, one more request comes to cancel the payment, and then that query runs first and marks user as free, then step 3 runs and user marked as Paid.
🕺🏻 Hurray, User got access to our Products without even paying.
Locking
✅ Here comes the saviour, Locks
🔐 Lock is a structure that allows only one thread at a time to enter a critical section (block of code that should not be accessed by multiple workers i.e. threads)
Hence, we will acquire lock before and release after the completion of the operation:-
- Step 0: try-acquire() lock
- Step 1: If acquired, We are fetching user (Atomic transaction)
- Step 2: Running some business logic in the code
- Step 3: Updating the MySQL record if user not paid (Atomic transaction)
- Step 4: release() the lock
😅 Problem
Now, here comes the problem which is if we will use some in memory lock data structure or any memory based lock it will be eligible for one instance for our application. what about the other instances running the same code and updating in the DB?
Well here comes the concept of Distributed locking
🔓 Distributed Locking
Here lock acts as a centralised service, where if one instance of our service acquires the lock then others can't on the same key.
WHAT KEY COULD BE HERE IN PAYMENT SERVICE?
🔓 For a user making payment key could be the combination of = "PAYMENT_" + user_id + amount
And this will be unique per user. And this key will remain same in case of user making payment or cancelling payment. Hence when one is happening other action can't proceed because both actions will try to acquire on same key.
💭 What what the heck is Key, Acquire lock, release lock. And most importantly how redis is being in use?
🎈 Using Redis to implement Distributed Locking
Using a single instance of Redis:-
But here are the few problems with a single redis instance:-
- A single instance may fail & lock acquired may not be released
- If two instances are used (master-replica) when one client will acquire lock on one instance
- master has to communicate the same with replica to sync. This communication itself is an async communication
🚀 So, if lock is acquired on master, and while communication to replica if master goes down before syncing with replica. Replica will become master where Lock on the same key will be available to acquire that was acquired on the master earlier.
Two instances of our services will be able to acquire the lock on redis even having two instances (master-replica).
Using Redlock Algorithm:-
Acquiring Lock:- We will try to acquire lock on multiple redis instances with lock expiration time
Validation of Lock:- lock will be considered as acquired if major redis instances got lock acquired for the client
Releasing Lock:- When releasing the lock, all instances releases the lock
And yes that's it.
❤️ Thanks for the read, and subscribe to our newsletter for more such articles:- https://www.serversidedigest.com/
For more information:-
- Jedis in Java:- https://redis.io/docs/latest/develop/connect/clients/java/jedis/
- Redis Client in Golang:- https://github.com/redis/go-redis
Top comments (15)
Why just not use DB lock?
It is taken as an example bruh
Bad example "bruh". The point of DB transactions is that you can do:
Step 0: start transaction
Step 1: fetch data
Step 2: do business logic
Step 3: write data and commit transaction
Fetch and write become one atomic transaction together.
Or do the same with user locks as mentioned in another comment.
Performance impact: Long-held database locks can lead to contention, slowing down queries and increasing latency.
Reduced DB load: By offloading locking to Redis, you reduce contention on the database and improve overall system performance.
And also using distributed locks for every critical sections is more preferred than using resource level locks in the code
Actually redis is also a db in the sense it stores KV of any type.
You can perform the same but then you have to add a code logic on client side for resource locking and sync which is also possible.
Using redis gives you that for free as its curse has become a favor of being single thread actually helps with critical operations
First, thing is, you can actually use db for lock, as specified earlier: dev.mysql.com/doc/refman/8.4/en/lo...
Secondly: You could try to do update user with "transaction" and watch approach: redis.io/docs/latest/develop/inter...
And finally, as you have multiple redis instances, perhaps some hybrid solution, e.g sharding + lock would work well too.
You need to define some sort of hash function, which would help you to identify to which server to connect.
E.g, in simplest approach l you calculate some sort of crc, normalise that to your number os servers,
That way all components will try to acquire lock on the same redis server
In Laravel I tried cache locks and it were failing in production, now implemented database transaction with mysql lock for update on query with retry mechanism and no exception for the last few days.
Yep that works also
This is absolutely awesome, using db rise at times is always confusing and a little bit stress.
You have really put in do much there.
Nice one
Redlock is not safe and should not be used. To understand why, please see this thorough analysis
If you need distributed consensus, look elsewhere!
I believe It may take care by DB transaction isolation levels like dirty read and others and each server request will acquire DB lock until DB operation may not completed. another query can anyone let me know the use case to use redis/distributed cache locking?
Performance impact: Long-held database locks can lead to contention, slowing down queries and increasing latency.
Reduced DB load: By offloading locking to Redis, you reduce contention on the database and improve overall system performance.
And also using distributed locks for every critical sections is more preferred than using resource level locks in the code
I believe locking at distributed cache level, it would be for longer time as it may include cache level ops + network communication + Db operation. While DB lock shall have for DB operation only. Apart from that Db lock provide lot's of other features like locking timeout in case of Db connection down/Db down. I am not sure how distributed level locking is more optimized then DB locking. Whether Distribute cache level locking have all the mandatory features like DB locking which may required during any failure.
If you already have a sql db, why not simply use transactions and conditional writes?
DB is taken as an example for the critical section man