Day 03

10 examples in 11 days

Exclude and Include

Now it's time to explain the exclude and include option. This options are great if you want to get control of what needs to be sync.

With the --exclude option, you can exclude a file, a directory and even a list of files/directories listed inside a file. Let's check out the the rsync syntax with the --exclude option.

I will create a file called exclude-me.txt inside ~/Users folder

iamgroot@laptop:~$ touch ~/Users/exclude-me.txt && echo "Hello World" >> ~/Users/exclude-me.txt

Now Let's check the size and files inside ~/Users and ~/Sync/Users folders

iamgroot@laptop:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
12 /home/iamgroot/Users/exclude-me.txt
151 /home/iamgroot/Users/user01.json
57 /home/iamgroot/Users/user02.json
69 /home/iamgroot/Users/user03.json
43 /home/iamgroot/Users/user.config
151 /home/iamgroot/Sync/Users/user01.json
57 /home/iamgroot/Sync/Users/user02.json
69 /home/iamgroot/Sync/Users/user03.json
88 /home/iamgroot/Sync/Users/user.config
4096 /home/iamgroot/Sync/Users/Users

To exclude a file, we need to wrap the file name in quotes after the --exclude option. This way, rsync leaves the file in source and doesn't include it inside destination.

iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-me.txt' ~/Users ~/Sync/
sending incremental file list

sent 146 bytes  received 17 bytes  326.00 bytes/sec
total size is 320  speedup is 1.96

Let's check if the file exclude-me.txt was copied from source to destination

iamgroot@laptop:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
12 /home/iamgroot/Users/exclude-me.txt
151 /home/iamgroot/Users/user01.json
57 /home/iamgroot/Users/user02.json
69 /home/iamgroot/Users/user03.json
43 /home/iamgroot/Users/user.config
151 /home/iamgroot/Sync/Users/user01.json
57 /home/iamgroot/Sync/Users/user02.json
69 /home/iamgroot/Sync/Users/user03.json
88 /home/iamgroot/Sync/Users/user.config
4096 /home/iamgroot/Sync/Users/Users

It wasn't included. Like I said, this option comes very handy. It also works with directories. Let's create a folder and include multiple files in it

iamgroot@laptop:~$ mkdir ~/Users/exclude-dir && touch ~/Users/exclude-dir/e{1..5}.txt && ls -a ~/Users/exclude-dir
.  ..  e1.txt  e2.txt  e3.txt  e4.txt  e5.txt

Now we have a folder called exclude-dir with 5 files calles e1.txt, e2.txt, etc

Let's sync them excluding this folder

iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-dir' ~/Users ~/Sync
sending incremental file list
Users/exclude-me.txt
Users/user.config
Users/user01.json
Users/user02.json
Users/user03.json

sent 731 bytes  received 112 bytes  1.69K bytes/sec
total size is 332  speedup is 0.39
jorge@code-lab:~$ stat -c "%s %n" ~/Users/* && stat -c "%s %n" ~/Sync/Users/*
4096 /home/jorge/Users/exclude-dir
12 /home/jorge/Users/exclude-me.txt
151 /home/jorge/Users/user01.json
57 /home/jorge/Users/user02.json
69 /home/jorge/Users/user03.json
43 /home/jorge/Users/user.config
12 /home/jorge/Sync/Users/exclude-me.txt
151 /home/jorge/Sync/Users/user01.json
57 /home/jorge/Sync/Users/user02.json
69 /home/jorge/Sync/Users/user03.json
43 /home/jorge/Sync/Users/user.config

We excluded the exclude-dir folder but it copied the exclude-me.txt file that we previously did not wanted to include at destination. How can we tell rsync to exclude this file as well?, easy, just tell rsync you want to eclude it as well

I will remove the file from destination and repeat the rsync command and check what was sync into destination

iamgroot@laptop:~$ rm ~/Sync/Users/exclude.me-txt
iamgroot@laptop:~$ rsync -vhr --exclude 'exclude-dir' --exclude 'exclude-me.txt' ~/Users ~/Sync
sending incremental file list

sent 150 bytes  received 17 bytes  334.00 bytes/sec
total size is 320  speedup is 1.92
iamgroot@laptop:~$ stat -c "%n" ~/Sync/Users/*
/home/iamgroot/Sync/Users/user01.json
/home/iamgroot/Sync/Users/user02.json
/home/iamgroot/Sync/Users/user03.json
/home/iamgroot/Sync/Users/user.config

Great, we achieve what we wanted.

If you really think about it, using mutiple --exclude options is not very clean and specially if you are working with multiple files/folders. There is a simple way to use --exclude in the same rsync line and its using curly braces

iamgroot@laptop:~$ rsync -vhr --exclude={'exclude-dir','exclude-me.txt'} ~/Users ~/Sync
sending incremental file list

sent 150 bytes  received 17 bytes  334.00 bytes/sec
total size is 320  speedup is 1.92

Note: You have to watch out and not to include whitespaces between the files/folders isnide the curly braces, if you include a whitespace it will sync all files/folders

GOOD rsync -vhr --exclude={'exclude-dir','exclude-me.txt'} ~/Users ~/Sync

NOT GOOD rsync -vhr --exclude={'exclude-dir', 'exclude-me.txt'} ~/Users ~/Sync

Again, if we have multiple, by multiple a mean hundreds or even more, using this technique is not so handy.

The good news is that rsync have a way to do this easily hehe (well done rsync!!!).

Today, we are very used to use git and just like git has .gitignore with rsync is pretty much the same thing. We can create a file and list all the files we want to exclude. To achieve this, we have the --exclude-from option.

I will create a file called rsync-ignore.txt and inside of it I will list all files that we want to exclude, notice that I will include the rsync-ignore.txt file as well

iamgroot@laptop:~$ touch ~/Users/rsync-ignore.txt && echo -e "exclude-dir\nexclude-me.txt\nrsync-ignore.txt" >> ~/Users/rsync-ignore.txt && cat ~/Users/rsync-ignore.txt
exclude-dir
exclude-me.txt
rsync-ignore.txt

iamgroot@laptop:~$ rsync -vha --exclude-from='/home/iamgroot/Users/rsync-ignore.txt' ~/Users ~/Sync
sending incremental file list
Users/

sent 165 bytes  received 20 bytes  370.00 bytes/sec
total size is 320  speedup is 1.73

notice I included the full path of the --exclude-from, using (~/) will not work

iamgroot@laptop:~$ ls -h ~/Sync/Users
user01.json  user02.json  user03.json  user.config

As you can see, everything worked just as expected. Now I will point out some other options to be used with the --exclude option.

What about the Include option?

The include option behaves pretty much the same as --exclude, is just the opposite. It's useful when you mix them along the same line. Let's see an example

iamgroot@laptop:~$ rsync -avhm --include '*.json' --exclude '*.txt' ~/Users ~/Sync
building file list ... done
Users/
Users/user.config
Users/user01.json
Users/user02.json
Users/user03.json

sent 695 bytes  received 95 bytes  1.58K bytes/sec
total size is 320  speedup is 0.41

What this did was:

• included all .json files
• excluded all the .txt files
• using the -m option it didn't copy the exclude-dir folder since it was empty because inside there were only .txt files and got empty

Below are some more options to be used.

Exclude Files using the wildcard (*)

This option let's you specify a pattern you want to exclude. To achieve this, you need to use the wilcard (*), it can also be used to exclude specific filetypes (it's all about how you use the wildcard. Let's see an example:

iamgroot@laptop:~$ rsync -avh --exclude '*.json' ~/Users ~/Sync

The above example will exclude all .json files. The * can be used at the beggining, at the end, in the middle, etc. i.e. rsync -avh --exclude '*ignore*' ~/Users ~/Sync will ignore the *rsync-ignore.txt* file

Exclude Files based in File Size

This option let's you specify between the maximum or minimum size of the files you want to exclude.

iamgroot@laptop:~$ rsync -avh --min-size=1K --max-size=50K ~/Users ~/Sync

The above example will exclude any file bigger than 1K and smaller than 50K

Here is what the man page says about it

--max-size=SIZE

This tells rsync to avoid transferring any file that is larger than the specified SIZE. The SIZE value can be suffixed with a string to indicate a size multiplier, and may be a fractional value
(e.g. "--max-size=1.5m").

This option is a transfer rule, not an exclude, so it doesn’t affect the data that goes into the file-lists, and thus it doesn’t affect deletions. It just limits the files that the receiver requests to be transferred.

The suffixes are as follows: "K" (or "KiB") is a kibibyte (1024), "M" (or "MiB") is a mebibyte (1024*1024), and "G" (or "GiB") is a gibibyte (1024*1024*1024). If you want the multiplier to be 1000 instead of 1024, use "KB", "MB", or "GB". (Note: lower-case is also accepted for all values.) Finally, if the suffix ends in either "+1" or "-1", the value will be offset by one byte in the indicated direction.

Examples: --max-size=1.5mb-1 is 1499999 bytes, and --max-size=2g+1 is 2147483649 bytes.

Note that rsync versions prior to 3.1.0 did not allow --max-size=0.

--min-size=SIZE

This tells rsync to avoid transferring any file that is smaller than the specified SIZE, which can help in not transferring small, junk files. See the --max-size option for a description of SIZE and other information.

Note that rsync versions prior to 3.1.0 did not allow --min-size=0.

Ok, that'll be for today's example, thanks for reading!!!

Follow, ❤ or 🦄