如何清理 lsof 中 deleted 状态的文件

在日常运维中，经常遇到磁盘空间满，但是找不到相应文件的情况。
通常这种情况都是文件被删除，但是还被进程占用，造成du与df结果不一致。
处理办法通常是停止占用文件的进程。

但是如果进程不能被停止呢？

另一个处理办法就是通过清空文件释放空间。
处理办法如下：

1. 通过lsof | grep deleted 找到未能删除掉的文件，确定占用的进程号；
2. 通过 ls -l /proc/PID/fd/* | grep 文件名，找到相应文件句柄；
3. 清除文件内容 echo > /proc/PID/fd/FD_NUM

这个操作被不会将文件删除，而是通过将文档内容清空的方法释放空间，文件还是存在的。

实验如下：

1.创造一个大文件

使用dd创建1个5000MB的文件，看df的输出，可用空间从13G降到了7.5G。

[root@test1 /]# df -TH
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  2.0G     0  2.0G   0% /dev
tmpfs                   tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs                   tmpfs     2.0G   30M  2.0G   2% /run
tmpfs                   tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        39G   27G   13G  68% /
/dev/sda1               xfs       1.1G  394M  671M  37% /boot
tmpfs                   tmpfs     396M     0  396M   0% /run/user/0
[root@test1 /]# dd if=/dev/zero of=/delete.tmp bs=1000MB count=5
5+0 records in
5+0 records out
5000000000 bytes (5.0 GB) copied, 5.35441 s, 934 MB/s
[root@test1 /]# df -TH
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  2.0G     0  2.0G   0% /dev
tmpfs                   tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs                   tmpfs     2.0G   30M  2.0G   2% /run
tmpfs                   tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        39G   32G  7.5G  81% /
/dev/sda1               xfs       1.1G  394M  671M  37% /boot
tmpfs                   tmpfs     396M     0  396M   0% /run/user/0
[root@test1 /]# du -sh /delete.tmp 
4.7G    /delete.tmp

2.使用tail 打开文件

用tail 打开文件，保证删除文件时，文件仍被占用

[root@test1 /]# tail -f /delete.tmp 

3.删除文件

使用rm 删除文件，在以下df输出中会发现，可用空间还是7.5G，没有变化，但是文件已经消失了。

[root@test1 /]# rm -f /delete.tmp 
[root@test1 /]# df -TH
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  2.0G     0  2.0G   0% /dev
tmpfs                   tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs                   tmpfs     2.0G   30M  2.0G   2% /run
tmpfs                   tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        39G   32G  7.5G  81% /
/dev/sda1               xfs       1.1G  394M  671M  37% /boot
tmpfs                   tmpfs     396M     0  396M   0% /run/user/0
[root@test1 /]# du -sh /delete.tmp 
du: cannot access ‘/delete.tmp’: No such file or directory

4.查找删除的文件

lsof 显示了deleted状态的文件名和大小（5000000000）。

[root@test1 ~]# lsof | grep deleted 
tail         419                  root    3r      REG              253,0 5000000000      55981 /delete.tmp (deleted)

5.查找文件句柄

[root@test1 ~]# ll /proc/419/fd | grep delete.tmp
lr-x------ 1 root root 64 May 23 16:05 3 -> /delete.tmp (deleted)

6.清空文件

[root@test1 ~]# echo > /proc/419/fd/3
[root@test1 ~]# df -TH
Filesystem              Type      Size  Used Avail Use% Mounted on
devtmpfs                devtmpfs  2.0G     0  2.0G   0% /dev
tmpfs                   tmpfs     2.0G     0  2.0G   0% /dev/shm
tmpfs                   tmpfs     2.0G   30M  2.0G   2% /run
tmpfs                   tmpfs     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mapper/centos-root xfs        39G   27G   13G  68% /
/dev/sda1               xfs       1.1G  394M  671M  37% /boot
tmpfs                   tmpfs     396M     0  396M   0% /run/user/0

那么/proc/PID/fd 是啥呢？

man proc

/proc/[pid]/fd/

          This is a subdirectory containing one entry for each file
          which the process has open, named by its file descriptor,
          and which is a symbolic link to the actual file.  Thus, 0
          is standard input, 1 standard output, 2 standard error,
          and so on.

          For file descriptors for pipes and sockets, the entries
          will be symbolic links whose content is the file type with
          the inode.  A readlink(2) call on this file returns a
          string in the format:

              type:[inode]

          For example, socket:[2248868] will be a socket and its
          inode is 2248868.  For sockets, that inode can be used to
          find more information in one of the files under
          /proc/net/.

          For file descriptors that have no corresponding inode
          (e.g., file descriptors produced by bpf(2),
          epoll_create(2), eventfd(2), inotify_init(2),
          perf_event_open(2), signalfd(2), timerfd_create(2), and
          userfaultfd(2)), the entry will be a symbolic link with
          contents of the form

              anon_inode:<file-type>

          In many cases (but not all), the file-type is surrounded
          by square brackets.

          For example, an epoll file descriptor will have a symbolic
          link whose content is the string anon_inode:[eventpoll].

          In a multithreaded process, the contents of this directory
          are not available if the main thread has already
          terminated (typically by calling pthread_exit(3)).

          Programs that take a filename as a command-line argument,
          but don't take input from standard input if no argument is
          supplied, and programs that write to a file named as a
          command-line argument, but don't send their output to
          standard output if no argument is supplied, can
          nevertheless be made to use standard input or standard
          output by using /proc/[pid]/fd files as command-line
          arguments.  For example, assuming that -i is the flag
          designating an input file and -o is the flag designating
          an output file:

              $ foobar -i /proc/self/fd/0 -o /proc/self/fd/1 ...

          and you have a working filter.

          /proc/self/fd/N is approximately the same as /dev/fd/N in
          some UNIX and UNIX-like systems.  Most Linux MAKEDEV
          scripts symbolically link /dev/fd to /proc/self/fd, in
          fact.

          Most systems provide symbolic links /dev/stdin,
          /dev/stdout, and /dev/stderr, which respectively link to
          the files 0, 1, and 2 in /proc/self/fd.  Thus the example
          command above could be written as:

              $ foobar -i /dev/stdin -o /dev/stdout ...

          Permission to dereference or read (readlink(2)) the
          symbolic links in this directory is governed by a ptrace
          access mode PTRACE_MODE_READ_FSCREDS check; see ptrace(2).

          Note that for file descriptors referring to inodes (pipes
          and sockets, see above), those inodes still have
          permission bits and ownership information distinct from
          those of the /proc/[pid]/fd entry, and that the owner may
          differ from the user and group IDs of the process.  An
          unprivileged process may lack permissions to open them, as
          in this example:

              $ echo test | sudo -u nobody cat
              test
              $ echo test | sudo -u nobody cat /proc/self/fd/0
              cat: /proc/self/fd/0: Permission denied

          File descriptor 0 refers to the pipe created by the shell
          and owned by that shell's user, which is not nobody, so
          cat does not have permission to create a new file
          descriptor to read from that inode, even though it can
          still read from its existing file descriptor 0.

代码007普通

打赏 收藏 海报 链接