The problem
Some time ago, me and my team were asked to migrate several Elasticache databases (Redis) from EC2-Classic (yep, I know π€·ββοΈ) to VPC. All these Databases were being used in production services so, taking an snapshot and restoring it on a VPC was not a good idea due to the time that takes doing the whole process.
Using AWS Online Migration was not an option too, since Online migration is designed for data migration from hosted Redis on Amazon EC2 or on-premise self-hosted Redis to ElastiCache for Redis and not between ElastiCache for Redis clusters.
So we kept looking for alternatives until we found a nice gist with a script that almost fit our needs. The only fail there was that it wasn't work ok on encryption in transit clusters.
The solution
So we did some changes on the script to fit our use-case and et voila! We finaly got a working version to move the data.
Demo
From a terminal with access to both Elasticache clusters (very important!):
- Clone the repo:
git clone git@github.com:joseapeinado/ElasticacheMigration.git
- Prepare the virtual env and install dependencies
cd ElasticacheMigration
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
- Profit!!!
SRC_CLUSTER=src_cluster.spznk9.ng.0001.use1.cache.amazonaws.com
DST_CLUSTER=dst_cluster.spznk9.ng.0001.use1.cache.amazonaws.com
python3 migrate-redis.py $SRC_CLUSTER --src_port=6379 $DST_CLUSTER --dst_port 6379 --src_db 2 --dst_db 2
Connecting to Redis instances...
Counting keys to migrate...
19222 keys: 100% |###############################################################################################################################################################################################################################################| Time: 0:00:05
('Keys disappeared on source during scan:', 0)
('Keys already existing on destination:', 19195)
Use cases
Copying just one Database to other existing cluster
The script copies data between clusters in a per-database basis, so it is ready to perform this migration out-of-the-box.
Copy all databases from one cluster to other
The only caveat here is the need of run the migration script once per database.
Copy database data from one DB to other DB within the same cluster.
Advantages doing this migration
The more obvious advantage I think is the time that could take using the script versus:
- Stop writes on Redis cluster
- Take a snapshot
- Launch a new cluster from the snapshot
- Resume writes on the new cluster
This process can take at least 15 minutes or so.
Contribute
Any improvements will be more than welcomed π
https://github.com/joseapeinado/ElasticacheMigration/
This is my very first tech post, so any advice would be appreciated.
And I hope the migration script help you like it helped me π
Top comments (2)
Sweet, tho you didn't include a link the repo itself (just a reference to it on the code).
Are you accepting PRs? I could add a docker env to those python commands.
Nice job :D
Thanks for your advice Santiago, I've added the link to the repo. And please, propose a PR!