Backing Up and Restoring Redis

Originally posted here: https://koalatea.io/redis-backup-restore/

Intro

Redis uses two methods for persistence, snapshotting and append only file. Both have different use cases and can be used separately or in conjunction. In this article, we will learn how to backup data and work with Redis persistence.

The Persistence Restore Flow

Redis makes it fairly simple to back up your data. The general method is as follows:

• Turn on snashotting or AOF
• Redis will write the data to a file (or two files)
• Backup this file (maybe a cron job to copy this file to S3)
• If your redis server fails, grab a back up, place in the correct directory, start redis
• Redis will read the file and restore what is backed up

I mentioned the above because I am used to backing up databases such as psql where you usually import the database. However, redis just reads the file in correct directory.

Setting up Redis

Let's begin by setting up redis. We will use docker compose for this, so be sure to install docker for desktop.

Here is the file we will use:

version: "3.2"
services:
  redis:
    image: "redis:alpine"
    command: redis-server /usr/local/etc/redis/redis.conf --requirepass sOmE_sEcUrE_pAsS
    ports:
      - "6379:6379"
    volumes:
      - ./redis-data:/data
      - ./redis.conf:/usr/local/etc/redis/redis.conf
    environment:
      - REDIS_REPLICATION_MODE=master

Notice that we set up two volumes.

volumes:
    - ./redis-data:/data
    - ./redis.conf:/usr/local/etc/redis/redis.conf

The first line will connect to where redis puts its data directory. This is documented on the official redis image.

The second line will allow us to connect our config to the docker file. So, be sure to download a config from the website above and put the config in your current directory.

We also use the following command to tell redis to use our config. Again, this is documented in the official image.

command: redis-server /usr/local/etc/redis/redis.conf

Now we can start our redis instance by running

docker-compose up

Snapshotting

Let's begin by looking at snapshotting in Redis. We have a few configuration options.

save 60 1000
stop-writes-on-bgsave-error no
rdbcompression yes
dbfilename dump.rdb

You can find each of these and more in the configuration file. Configs can be found here: https://redis.io/topics/config. Here is the direct link to Redis 6 config: https://redis.io/topics/config. If you search for "SNAPSHOTTING" you will see the section below.

One of the main configs we will use is save 60 1000. Now, the redis config file does a great job of documenting the config, so we won't cover them all. But, let's run through an example of using the save. The above directive tells redis to save every 60 seconds if 1000 keys have changed. So, you can see that we may never have a snapshot if 1000 changes are not saved. This is important to remember.

Let's add the following save command. This will instruct redis to save every 10 seconds if 1 key has changed.

save 10 1

With this, let's restart the docker container.

docker-compose down
docker-compose up

Now, let's login to our docker instance and change a key. Click the CLI in docker desktop and you should see the cli screen below.

Next, type redis-cli to connect to your redis.

Then, let's see set a key value. This can be creating or updating a key.

set new-key "value-for-my-key"

Now, we wait about 10 seconds, and we should see the following logs.

This is redis saving our snapshot to the disk. You should also see the following dump.rdb example in the redis-data folder.

Append Only

Now that we know about snapshotting, we can move on to Append Only. Append only is defined by its name, and tells redis to append logs to a file. This can become a large file, but is a nice event log. This is a great pattern for you event sourcing nerds.

The following configs are available to us.

appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

We can find this in our redis config similarly to snapshotting. We can search for "APPEND".

To enable this mode, change the following line:

appendonly no

to

appendonly yes

Then restart your docker service. Then, open up your cli in docker like we did before. Next, add a new key, and feel free to run other coammnds as well.

set funtimes "all day"

You should now see a new file in your redis-data directory called appendonly.aof. This will contain some information regarding key changes for redis to use.

Note that it doesn't have anything before we turned on the setting. This is one of a few reasons why you might want to use snapshots and append only together.

Run manual backup

Let's take a slight detour to see how we can manually backup our data. If we want to manually trigger backups, say from a cron script, we have to commands we can use.

• SAVE
• BGSAVE

Ideally, you should only use BGSAVE so that redis runs the save in the background. To test this out, open up your redis cli and run the BGSAVE command. You should see save logs in your docker terminal.

Delete and Restore

We are now becoming familiar with backups. Next, we will move on to replicating a redis fail over.

Our first step to backup our data. For this, I will just copy my two db files over to my desktop, but feel free to do what you want.

Next, let's stop the docker server and delete the volume.

Restart the server, and check that our keys are indeed missing.

docker-compose up

scan 0

Next, drop in our db files and restart the server.

And now we see that our keys have been restored!