YugabyteDB advanced analysis: log inspection

This blog is about an analysis and troubleshooting option that the tool yb_stats provides for YugabyteDB clusters: looking at the logging output.

Two log inspection options

yb_stats offers two ways to inspect the logs of a YugabyteDB cluster: one is to print the entire (last one megabyte) of the logging from all servers, and the other one is to show only logging lines as they get created, alike the tail -f command.

print-log

The first switch is the print-log option: yb_stats --print-log. This switch quite simply takes the logs of all the servers that have a /log endpoint that match the hosts and ports set, orders it by time (!), and then prints the logs.

tail-log

The second switch is the tail-log option: yb_stats --tail-log. This switch also takes the logs of all the servers that have a /log endpoint that match the hosts and ports set, but after it fetched the logs, the tool remains active, and only prints shows log lines which are produced after the first fetch. This is similar to how tail -f works on files, but now for all the servers in scope, over the network.

Enhanced output

YugabyteDB processes can produce a lot of log information. In order to help log inspection, a number enhancements are made:

1. Timestamps are in a easy to understand format, and the timezone (UTC) is clearly indicated:
   

   2023-01-30 11:18:35.746704 UTC
   

2. Severity is indicated, and coloured:
   

   • An informal message is shown with a green 'I'.
   • A warning message is shown with a yellow 'W'.
   • An error message is shown with a red 'E'.
   • A fatal message is shown with a purple 'F'. This allows to spot different severities more easy.

Filtering

Outside of these enhancements, the log output can still get overwhelming, if only because we combine the log output of multiple servers at a single place.

For that reason, a number of filtering options are available:

1. --log-severity: By default, the I (informal) messages are not shown. However, it's perfectly possible to add the informal messages, if that is necessary by setting the argument of --log-severity to 'IWEF'. It's also possible to filter for only error and fatal errors by setting it to 'EF'.
2. --hostname-match: By default, the log lines of all hosts that provide glog format errors at /logs at the specified hosts and ports are shown. This might be too much: for example: the YCQL port (12000) will show the /logs output identical to the tablet server http port (9000). It's easy to set --hostname-match to '(7000|9000)' for only the master and tablet server endpoints, for example.
3. --stat-name-match: If you are looking for specific code locations (such as 'log.cc'), or for specific names in the log lines (such as tablet UUID), you can use --stat-name-match to filter on these.

If you using YugabyteDB and want to give yb_stats a spin: the installation instructions can be found in the yb_stats book, in the install chapter.