DEV Community

Jose Peinado
Jose Peinado

Posted on

Moving data between Elasticache Databases

The problem

Some time ago, me and my team were asked to migrate several Elasticache databases (Redis) from EC2-Classic (yep, I know πŸ€·β€β™‚οΈ) to VPC. All these Databases were being used in production services so, taking an snapshot and restoring it on a VPC was not a good idea due to the time that takes doing the whole process.
Using AWS Online Migration was not an option too, since Online migration is designed for data migration from hosted Redis on Amazon EC2 or on-premise self-hosted Redis to ElastiCache for Redis and not between ElastiCache for Redis clusters.
So we kept looking for alternatives until we found a nice gist with a script that almost fit our needs. The only fail there was that it wasn't work ok on encryption in transit clusters.

The solution

So we did some changes on the script to fit our use-case and et voila! We finaly got a working version to move the data.

Demo

From a terminal with access to both Elasticache clusters (very important!):

  1. Clone the repo:
git clone git@github.com:joseapeinado/ElasticacheMigration.git
Enter fullscreen mode Exit fullscreen mode
  1. Prepare the virtual env and install dependencies
cd ElasticacheMigration
python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode
  1. Profit!!!
SRC_CLUSTER=src_cluster.spznk9.ng.0001.use1.cache.amazonaws.com
DST_CLUSTER=dst_cluster.spznk9.ng.0001.use1.cache.amazonaws.com

python3 migrate-redis.py $SRC_CLUSTER --src_port=6379 $DST_CLUSTER --dst_port 6379 --src_db 2 --dst_db 2
Connecting to Redis instances...
Counting keys to migrate...
19222 keys: 100% |###############################################################################################################################################################################################################################################| Time: 0:00:05
('Keys disappeared on source during scan:', 0)
('Keys already existing on destination:', 19195)
Enter fullscreen mode Exit fullscreen mode

Use cases

Copying just one Database to other existing cluster

The script copies data between clusters in a per-database basis, so it is ready to perform this migration out-of-the-box.

Copy all databases from one cluster to other

The only caveat here is the need of run the migration script once per database.

Copy database data from one DB to other DB within the same cluster.

Advantages doing this migration

The more obvious advantage I think is the time that could take using the script versus:

  1. Stop writes on Redis cluster
  2. Take a snapshot
  3. Launch a new cluster from the snapshot
  4. Resume writes on the new cluster

This process can take at least 15 minutes or so.

Contribute

Any improvements will be more than welcomed πŸ˜ƒ
https://github.com/joseapeinado/ElasticacheMigration/

This is my very first tech post, so any advice would be appreciated.
And I hope the migration script help you like it helped me πŸ™‚

Top comments (2)

Collapse
 
ingcsmoreno profile image
Carlos S. Moreno

Sweet, tho you didn't include a link the repo itself (just a reference to it on the code).

Are you accepting PRs? I could add a docker env to those python commands.

Nice job :D

Collapse
 
bambuco profile image
Jose Peinado

Thanks for your advice Santiago, I've added the link to the repo. And please, propose a PR!