While interning, I once had an application that logged every request and response to the Linux file system. That app easily fills up our disks due to traffic. I got confused when I tried to delete log files because disk usage didn’t decrease even after the log files were deleted. So I restarted the app and finally, the disk usage decreased.
But it is ineffective because we need to kill the application which is making a downtime at that time. Recently, I learned that it was caused by something called a file descriptor
.
TL;DR
$ df -h <big file dir> # check the current usage
$ rm -f <big file> # remove the big file
$ df -h <big file dir> # it still the same
$ lsof +L1 | grep <big file> # get the running app PID
$ cd /proc/<PID>/fd/ # go to the running app fd
$ ls -l | grep <big file> # get the symlink
$ > <symlink> # empty the symlink
$ df -h <big file dir> # it backs to normal
Let’s reproduce that incident.
Creating a big file
We need to create a big file, to fill our disk up. I think 3GB is enough. Well, you don’t need to make it big just to reproduce, but more than 1GB could show the difference easily in the df -h
.
$ yes lorem ipsum dolor sit amet \
| head -c 3GB >> big-file.txt
Yep, now you have a 3GB file contains lorem ipsum called big-file.txt
Create a simple app
Then we need to create an app that simply read big-file.txt
. But I want to show you even if the big-file.txt
is deleted, our app still writing to it. So I make the app read and append the big-file.txt
using Golang, you can do it with whatever you prefer. Let’s call it write-to-file.go
package main
import (
"fmt"
"os"
)
func main() {
f, err := os.OpenFile("big-file.txt",
os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
panic(err)
}
defer f.Close()
for i := 0; ; i++ {
fmt.Scanln()
fmt.Fprintf(f, "%d\n", i)
fmt.Println(i, "written")
}
}
It will wait for our input and loop forever until we terminate the app.
Check current disk usage
Well, because I put it in the home dir
, I check the size of the /home
using df
.
$ df -h /home
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 913G 34G 832G 4% /home
Tail the appended text in the file
tail -f
will follow the last 10 lines in the file and it would show this
$ tail -f big-file.txt
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lor
Run the app
Let’s run our write-to-file.go
$ go run write-to-file.go
For every Enter
you input, it will append the big-file.txt
like this
$ tail -f big-file.txt
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lor0
1
2
3
$ go run write-to-file.go
0 written
1 written
2 written
3 written
Remove the file
Now let’s stop the tail -f
and remove that big-file.txt
.
$ rm big-file.txt
Does your disk usage decreased? Unfortunately, mine is not. It might be karma working there isn’t it?
$ df -h /home
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 913G 34G 832G 4% /home
Now, how to really remove
the big-file.txt
?
For doing that we need to check the current running process that still open that big-file.txt
.
$ lsof +L1 | grep big-file.txt
write-to- 102333 clavinjune 3w REG 8,17 3000000008 0 7341204 /home/clavinjune/Public/file-descriptor/big-file.txt (deleted)
Now we got the app’s name, its PID, and we know that the Linux know the big-file.txt
is already deleted
.
Let’s open the file descriptor
of the app at /proc/<PID>/fd
.
$ cd /proc/102333/fd
$ ls -l | grep big-file.txt
l-wx------ 1 clavinjune clavinjune 64 Jan 23 01:38 3 -> /home/clavinjune/Public/file-descriptor/big-file.txt (deleted)
It seems like, we have a symlink called 3
that linked to our big-file.txt
.
Now if you tail -f
to the symlink and input some Enter
to the app, it will show something like this
$ tail -f 3
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lorem ipsum dolor sit amet
lor0
1
2
3
4
5
6
7
8
9
10
Interesting isn’t it? Even if the big-file.txt
is deleted, the app still writing into the symlink. That symlink held our disk space, so we need to make that symlink is empty. By simply echoing nothing to the symlink in this case echoing nothing to 3
.
$ > 3
Now check the disk usage it will return to normal like we really delete the big-file.txt
and the program is still running and still appending to the file.
$ df -h /home
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 913G 32G 835G 4% /home
Let’s input some Enter
to the app again then the symlink will filled up again by the app.
$ tail -f 3
11
12
13
14
15
16
Top comments (0)