YugabyteDB, like every database, allows to specify the directory where the database files are stored. This parameter, or gflag, in YugabyteDB is fs_data_dirs.
Of course the directory can also be a mount point for a filesystem, so that the Operating System and the binaries are completely independent from the database files.
Tablet server and Master
The parameter fs_data_dirs is a setting for both the tablet server, as well as the master. Both store rocksdb database files.
It is not a problem to have both the master and the tablet server running on the same machine/node, and it also is not a problem to have them both be pointing to the same directory with fs_data_dirs. They will create their individual, separate, master and tserver directories, where they will store and manipulate their files.
fs_data_dirs contents
Inside the directory set by the fs_data_dirs parameter, the tablet server will create when freshly started:
- pg_data (only if
enable_ysqlistrue) - yb-data/tserver
The master server creates:
- yb-data/master
Inside the yb-data/tserver and yb-data/master directories, a directory structure is created which contains:
- wals: the YugabyteDB write ahead log.
- tablet-meta: tablet metadata.
- data: the rocksdb database directories.
- consensus-meta: RAFT/consensus metadata.
- logs: textual log files of the server and optionally of the PostgreSQL logfile.
The information that is stored for each tablet in the tablet-metadata directory links wal and rocksdb together, for which it is using the operating system paths.
ADDING a directory to fs_data_dirs
If you want to add another directory or path for storing data, you can do that by specifying a second directory separated by a comma without a space, such as: --fs_data_dirs=/mnt/d0,mnt/d1.
If you add a directory and restart the tablet server, it will create the directory yb-data/tserver in the path specified in fs_data_dirs, and create the directories wals, tablet-meta, data and consensus-meta in the created directory.
The first thing to notice is that the logs directory is not in the list of directories in yb-data/tserver. The reason is the logs will be stored in the first directory that the server finds in fs_data_dirs.
How YugabyteDB works with multiple data directories
When multiple data directories are set in fs_data_dirs, there is no dynamic or automatic moving or load balancing that will happen.
If a directory is added to fs_data_dirs later on, it will be initialised, and the allocations in the first directory will stay where they are.
Also, when an object is created, and therefore its tablet(s) which require wal, tablet-meta, rocksdb and consensus-meta allocations, these will still be created in the first directory for the first tablet; only when a second tablet is specified, it will be allocated in the second directory, a third in the first directory (if two are specified), a fourth in the second directory again, and so on.
This means that if you are running with a filesystem mounted at the path specified in fs_data_dirs, and the filesystem is getting filled up, and it's impossible to extend the filesystem (if you can extend the partition that contains the filesystem, XFS allows online resizing), adding a second data directory is quite likely not the solution.
REMOVING or CHANGING a directory in fs_data_dirs
Currently, we do NOT support removing or changing a directory specified in fs_data_dirs. The reason is that in the metadata, the full path is specified, and thus moving it will break the specified paths.
If you must change a path, or do something else that forces you to change the paths, the best way of doing that is adding a tablet server node to the cluster with the correct configuration for fs_data_dirs, and blacklisting the problem tablet server. This will make the tablets to move away from the problem node, and the tablet replicas be spread over the cluster again.
Conclusion
YugabyteDB stores metadata that contains the paths, so you cannot move YugabyteDB datafiles to another directory and add that to fs_data_dirs to circumvent a filesystem from filling up.
Adding a directory to fs_data_dirs is possible, but this only will get tablets placed if multiple are specified during creation, and therefore is not a way to balance filesystem space usage.
If you run into problems with the path specified in fs_data_dirs, add a node or nodes to the cluster which are sufficiently configured to store the data, and blacklist the problem node or nodes to make them move the tablets.
The idea of running YugabyteDB is to run simple reasonably small nodes with local (SSD) drives, and scale up the cluster, not the nodes. Also removing or changing a node should be a normal non-intrusive action, which is why dealing with fs_data_dirs in this way is sufficient.
Top comments (0)