This is the third part of our series about Sysdig. The first tutorial walked you through the process of capturing and analyzing data with the Sysdig command-line utility. The second tutorial showed how to use Csysdig, the ncurses based GUI. Now, we'll focus on explaining how you can troubleshoot your containers with Sysdig Inspect.
Prerequisites
Before starting this tutorial, you should follow the first part of the series. We assume that Docker Compose and Sysdig are installed on your system. Also, you should have a running WordPress installation. Enter the following command to verify that everything is properly set up:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f2a5bf941eba wordpress:latest "docker-entrypoint.s…" About a minute ago Up About a minute 0.0.0.0:8000->80/tcp wordpress-sysdig_wordpress_1
95c44dd63ba9 mysql:5.7 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp, 33060/tcp wordpress-sysdig_db_1
8405c2c87f59 sysdig/sysdig "/docker-entrypoint.…" About a minute ago Up About a minute sysdig
Make sure the following three Docker containers are running on your system:
- WordPress
- MySQL
- Sysdig
Install Sysdig Inspect
There are two ways in which you can install Sysdig inspect:
- Download the pre-built binary for your operating system from the project's GitHub repository.
- Run it as a Docker container.
For this tutorial, you will install Sysdig Inspect inside of a Docker container.
In this section, you'll install Sysdig Inspect and make it so that the trace files captured with sysdig
can be accessed from the Sysdig Inspect container.
1.In a new terminal window (on the host), run the following command to create a directory called captures
:
mkdir captures
2.You can install Sysdig Inspect in a container with the following docker run
command:
docker run -d -v /home/vagrant/captures/:/captures -p8080:3000 sysdig/sysdig-inspect:latest
Unable to find image 'sysdig/sysdig-inspect:latest' locally
latest: Pulling from sysdig/sysdig-inspect
f661dfa5eb7d: Pull complete
572ec58ac70f: Pull complete
c1d2c84e7140: Pull complete
1ebe1ad7056b: Pull complete
e5e1416d36d0: Pull complete
9b379815df57: Pull complete
9034f4cfeb96: Pull complete
91638da20eef: Pull complete
a4105e889626: Pull complete
b7359900512b: Pull complete
2daad7eb7645: Pull complete
d0fb8981639d: Pull complete
ed8bc4c05912: Pull complete
29d1d8034bab: Pull complete
fead5f9a4603: Pull complete
Digest: sha256:0c6270a09cfb28f2c248761f4783d0bc53d3dfd9b83d27f03b98039d5366cf45
Status: Downloaded newer image for sysdig/sysdig-inspect:latest
ee88efbc56205164a269102d266509ce5a677d024071d29774b12cd3414a7a69
A few things to note about the above command:
- The
-v
flag specifies the files on the host that Sysdig Inspect can access. In this example, the files placed inside of the/home/vagrant/captures/
directory will be available in thecaptures
folder in the container running Sysdig Inspect. - The
-d
flag starts the container in detached mode - The
-p
flag publishes the3000
port to the8000
port on the host.
3.At this point you can inspect the status of the Sysdig Inspect container with:
docker ps
Note that the status is currently set to starting
. Wait a bit until it changes to healthy
:
4.Now you can point your browser to http://localhost:8080:
Capture Data
As you can see in the screenshot above, you can specify the path to a capture file taken with Sysdig. In this section, you'll create a new trace file, simulate an issue by deleting a file, and then load the data into Csysdig Inspect for inspection.
1.In a new terminal window (on the host), display the list of your active containers:
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7da3fee501db sysdig/sysdig-inspect:latest "/docker-entrypoint.…" 2 minutes ago Up 8 minutes (healthy) 0.0.0.0:8080->3000/tcp agitated_leakey
f2a5bf941eba wordpress:latest "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 0.0.0.0:8000->80/tcp wordpress-sysdig_wordpress_1
95c44dd63ba9 mysql:5.7 "docker-entrypoint.s…" 8 minutes ago Up 8 minutes 3306/tcp, 33060/tcp wordpress-sysdig_db_1
8405c2c87f59 sysdig/sysdig "/docker-entrypoint.…" 9 minutes ago Up 9 minutes sysdig
2.Start a shell inside of the wordpress
container by entering the following docker exec
command and passing the identifier of the container:
docker exec -ti f2a5bf941eba /bin/bash
☞ In the above example, the identifier of our container is
f2a5bf941eba
but yours will be different.
3.Similarly, start a shell inside of the csysdig
container with:
docker exec -ti 8405c2c87f59 /bin/bash
4.Initiate a new capture by entering the following command inside of the sysdig
container:
sysdig -w monitoring-wordpress-error.scap
5.Open a new terminal window (on the host), and use ApacheBench to make 10000 requests with a maximum of 100 requests running concurrently:
ab -n 10000 -c 100 http://localhost:8000/
6.While ApacheBench is running, move back to the wordpress
container and delete the index.php
file:
rm -rf index.php
7.When the benchmark is finished, move to the sysdig
container and stop capturing data by entering "CTRL+C".
8.If everything went well, a file named monitoring-wordpress-error.scap should have been created:
ls -la monitoring-wordpress-error.scap
-rw-r--r--. 1 root root 146795720 Jan 16 20:56 monitoring-wordpress-error.scap
9.Switch back to the host, and use the docker cp
command to copy the capture file to the captures
directory:
docker cp 8405c2c87f59:/monitoring-wordpress-error.scap captures/
10.Verify whether the file has been successfully copied to the host with:
ls -la captures/
total 143356
drwxrwxr-x. 2 vagrant vagrant 45 Jan 16 21:02 .
drwx------. 8 vagrant vagrant 208 Jan 16 18:27 ..
-rw-r--r--. 1 vagrant vagrant 146795720 Jan 16 20:56 monitoring-wordpress-error.scap
11.Point your browser to http://localhost:8080 and type the path to the trace file (that is /captures/monitoring-wordpress-error.scap
) in the "capture file path" text area:
12.Press the Enter
key and then wait a bit while Sysdig Inspect loads the trace file:
13.When everything is ready, you will see the overview page:
How to Navigate the Interface
In this section, we'll explain the basic concepts behind Sysdig's Inspect interface. Once the trace file is loaded, you'll be presented with a screen similar to the one below (the annotations are ours):
There's a lot of information packed in here. Let's break this down:
(A) At the top of the screen, Sysdig Inspect shows the name of the currently open trace file.
(B) The drill-down section shows the list of active filters. Select the "Containers" view and then navigate to the "Processes view":
☞ You can customize what data is displayed by manually editing the "Sysdig Filter" textbox.
(C) Similar to how Csysdig works, Sysdig Inspect lets you filter the data using several built-in views.
(D) The main window contains several tiles. Each tile shows the values of a particular metric and its evolution in the period of time in which the capture was taken. Tiles are grouped together to make navigation easier.
(E) The timeline allows you to use two sliders to narrow down the period during which the data is analyzed. Continuing our previous example in which we selected the "Container" view and then "Navigated" to the process view, we used the sliders to narrow down the period of time for which the data is displayed:
We've covered the basics of using the interface! In the next section, we'll show how you can use Sysdig Inspect to troubleshoot your containers.
Put It to the Test
Remember that, in the "Capture Data" section, we've purposely introduced an error. Now, we'll make use of that to showcase how you can use Sysdig Inspect to analyze data and troubleshoot your containers.
1.First, navigate to the overview page:
2.From the "Views" section, select "Errors". You'll see a high number of ENOENT
(which stands for Error NO ENTry) errors:
3.Highlight the ENOENT
line by clicking on it. This will display three round buttons:
- Drill-Down
- I/O Streams
- Syscalls
4.Select "Drill-down' and Sysdig Inspect will show the list of processes that raised this error:
5.Pick the one from top of the list. Then, select "Syscalls". Since we've deleted the index.php
file a bit later in the capture, you must scroll down a few lines until you see lots of error messages similar to the ones below:
☞ Note that the Sysdig Filter checkbox has been automatically updated.
Now, this sheds some light, but it doesn't tell you what has happened.
6.Let's clear all the filters and navigate to the default view by selecting "Overview":
7.See the "Security" section. It shows that two commands were executed, and one file was deleted.
8.To get more details, select the Deleted Files
tile:
Note that the Drill-down
button is now displayed. Also, the moment when the file was deleted is shown at the bottom of the page.
9.Select the Drill-down
button, and Sysdig Inspect will show that the index.php
file was deleted:
10.If you select the Executed commands -> Drill-down
tile, then you'll see
As you can see, Sysdig Inspect displays a host of useful information you can rely on to troubleshoot your containers.
In this example, it wasn't overly difficult to figure out what caused the issue, but there may be cases in which you'll have to dig a bit deeper. The knowledge you acquired is a complete tool belt that'll help you get through in any situation that requires troubleshooting your container-based applications.
Thanks for reading!
Top comments (0)