0. Problem description
After receiving the pin alarm, online Solr server / var partition inode is full.
/The var partition can no longer create new files.
What is inode
The inod is the index node of the block in the file system. Each file in the os will occupy one inode, and the size of the inode is limited.
# 1. Use df-i to view the usage of inode in each partition [root@xxx ~]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 2097152 26111 2071041 2% / tmpfs 33059719 4 33059715 1% /dev/shm /dev/sda1 32768 46 32722 1% /boot /dev/sda7 77840384 49535 77790849 1% /data /dev/sda3 2097152 199748 1897404 10% /usr /dev/sda5 2097152 2061500 35652 99% /var # Conclusion: confirm that the inode of var partition is almost full # 2. Use the find /var directory to view the files generated in the last hour [root@xxx ~]# find /var/ -type f -mtime +60 .... /var/spool/postfix/maildrop/8B39F1B854F /var/spool/postfix/maildrop/EA79986E6 /var/spool/postfix/maildrop/7576417CEB2 /var/spool/postfix/maildrop/EFEEA128578 /var/spool/postfix/maildrop/DF9C5216A /var/spool/postfix/maildrop/895FB124392 .... # 3. Temporary solutions [root@xxx maildrop]# cd /var/spool/postfix/maildrop [root@xxx maildrop]# rm -rf * -bash: /bin/rm: Argument list too long # Cannot delete # 3.1 as there are too many files to delete, find out all the files in the current directory through the following script, and then delete them [root@xxx maildrop]# find . -type f -name '*' | xargs rm -rf '*' [root@xxx maildrop]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda2 2097152 26111 2071041 2% / tmpfs 33059719 4 33059715 1% /dev/shm /dev/sda1 32768 46 32722 1% /boot /dev/sda7 77840384 50439 77789945 1% /data /dev/sda3 2097152 199748 1897404 10% /usr /dev/sda5 2097152 3952 2093200 1% /var # Return to normal
2. Cause analysis and Solutions
Through the above operations, it is found that the reason is that the directory of / var/spool/postfix/maildrop generates a large number of files.
- Further analysis
The reason for the file generation under the maildrop folder is:
When Linux executes cron, it will send the output and warning information in the cron execution script to the cron owner in the form of mail. However, because sendmail and postfix in the customer environment are not running normally, the mail sending is not successful. All the small files are piled up under the maildrop directory, and there is no mechanism to automatically clean up and transform, so it is as long as Over the course of a year, this directory has accumulated a large number of files. Check the information of man cron, and you can know that it will be sent to the cron owner
- Radical cure
cron task execution result is not shielded. Solution:
# 1. Problem setting method */10 * * * * /tmp/test.sh # 2. Use > / dev / null 2 > & 1 to mask the output and warning during execution, but the log of cron task execution will still be found in / var/log/cron */10 * * * * /tmp/test.sh >/dev/null 2>&1