Day 09

10 examples in 11 days

Sync Local to Remote over SSH and Limit Bandwith

Today is going to be a simple but useful example. I will explain how to limit the bandwidth.

Let me set a very common scenario. Sometimes we need to upload a very large file or multiple files to a remote instance. We know that this task may take a long time to transfer up to our remote instance, this is ok if we are the only one who is using the internet or if we don't care having rsync consuming all of our bandwidth. But if we are in a shared internet connection or with a limited bandwidth, we don't want to cause a downtime to our connectivity.

The solution is to limit the bandwitdh, this way rsync can transfer everything correctly and we can still work in something else without loosing connectivity performance.

The option to do this is --bwlimit=KBPS. KBPS stands for KiloBytes Per Second. Let's check what the man page says about this option:

--bwlimit=RATE

This option allows you to specify the maximum transfer rate for the data sent over the socket, specified in units per second. The RATE value can be suffixed with a string to indicate a size
multiplier, and may be a fractional value (e.g. "--bwlimit=1.5m"). If no suffix is specified, the value will be assumed to be in units of 1024 bytes (as if "K" or "KiB" had been appended).
See the --max-size option for a description of all the available suffixes. A value of zero specifies no limit.

For backward-compatibility reasons, the rate limit will be rounded to the nearest KiB unit, so no rate smaller than 1024 bytes per second is possible.

Rsync writes data over the socket in blocks, and this option both limits the size of the blocks that rsync writes, and tries to keep the average transfer rate at the requested limit. Some
"burstiness" may be seen where rsync writes out a block of data and then sleeps to bring the average rate into compliance.

Due to the internal buffering of data, the --progress option may not be an accurate reflection on how fast the data is being sent. This is because some files can show up as being rapidly sent
when the data is quickly buffered, while other can show up as very slow when the flushing of the output buffer occurs. This may be fixed in a future version.

The man page is very explicit on how we can limit the bandwidth of our rsync process.

What I'll do?

I will backup a file that is 5,3MB in size, and will limit the bandwith to 400Kbps. These means it will take up to 10 or 12 seconds to upload my file.

iamgroot@laptop:~$ rsync --progress --bwlimit=400 -vhre ssh /home/iamgroot/backup-to-upload/backup_iamgroot.tar.gz iamgroot@222.222.222.222:backups/
sending incremental file list
backup_iamgroot.tar.gz
          5.30M 100%  407.18kB/s    0:00:12 (xfr#1, to-chk=0/1)

sent 5.30M bytes  received 35 bytes  321.43K bytes/sec
total size is 5.30M  speedup is 1.00

here is an animated gif of rsync while transfering:

Everything was completely uploaded, let me show you:

iamgroot@remote-instance:~$ ls -l --b=KB  backups/backup_iamgroot.tar.gz | cut -d " " -f5
5303kB

Simple right?.

Well that's it for today's example, hope you enjoyed reading.

Thanks for reading, keep it safe!!

Follow, ❤ or 🦄