Causes and consequences of inode occupation

0. Problem description

  • phenomenon
    After receiving the pin alarm, online Solr server / var partition inode is full.

  • Influence
    /The var partition can no longer create new files.

  • What is inode
    The inod is the index node of the block in the file system. Each file in the os will occupy one inode, and the size of the inode is limited.

1. investigation

# 1. Use df-i to view the usage of inode in each partition
[root@xxx ~]# df -i
Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/sda2       2097152   26111  2071041    2% /
tmpfs          33059719       4 33059715    1% /dev/shm
/dev/sda1         32768      46    32722    1% /boot
/dev/sda7      77840384   49535 77790849    1% /data
/dev/sda3       2097152  199748  1897404   10% /usr
/dev/sda5       2097152 2061500    35652   99% /var
# Conclusion: confirm that the inode of var partition is almost full

# 2. Use the find /var directory to view the files generated in the last hour
[root@xxx ~]# find /var/  -type f -mtime +60
....
/var/spool/postfix/maildrop/8B39F1B854F
/var/spool/postfix/maildrop/EA79986E6
/var/spool/postfix/maildrop/7576417CEB2
/var/spool/postfix/maildrop/EFEEA128578
/var/spool/postfix/maildrop/DF9C5216A
/var/spool/postfix/maildrop/895FB124392
....

# 3. Temporary solutions
[root@xxx maildrop]# cd /var/spool/postfix/maildrop
[root@xxx maildrop]# rm -rf * 
-bash: /bin/rm: Argument list too long # Cannot delete
# 3.1 as there are too many files to delete, find out all the files in the current directory through the following script, and then delete them
[root@xxx maildrop]# find . -type f -name '*' | xargs rm -rf '*'

[root@xxx maildrop]# df -i 
Filesystem       Inodes  IUsed    IFree IUse% Mounted on
/dev/sda2       2097152  26111  2071041    2% /
tmpfs          33059719      4 33059715    1% /dev/shm
/dev/sda1         32768     46    32722    1% /boot
/dev/sda7      77840384  50439 77789945    1% /data
/dev/sda3       2097152 199748  1897404   10% /usr
/dev/sda5       2097152   3952  2093200    1% /var # Return to normal

2. Cause analysis and Solutions

Through the above operations, it is found that the reason is that the directory of / var/spool/postfix/maildrop generates a large number of files.

  • Further analysis
    The reason for the file generation under the maildrop folder is:

When Linux executes cron, it will send the output and warning information in the cron execution script to the cron owner in the form of mail. However, because sendmail and postfix in the customer environment are not running normally, the mail sending is not successful. All the small files are piled up under the maildrop directory, and there is no mechanism to automatically clean up and transform, so it is as long as Over the course of a year, this directory has accumulated a large number of files. Check the information of man cron, and you can know that it will be sent to the cron owner

  • Radical cure
    cron task execution result is not shielded. Solution:
# 1. Problem setting method
*/10 * * * * /tmp/test.sh

# 2. Use > / dev / null 2 > & 1 to mask the output and warning during execution, but the log of cron task execution will still be found in / var/log/cron
*/10 * * * * /tmp/test.sh >/dev/null 2>&1

99. quotations

Tags: Programming Linux solr

Posted on Tue, 03 Dec 2019 05:16:08 -0500 by craigh