Posts

Linux: No space left on device – running out of Inodes – PHP sessions

/in /by

A production Ubuntu server could not create anymore files claiming that no space was left. Strangely enough it was not about storage space but inode availability. The first response might be to increase disk size but it will not help unless you format the drive to assign a new number of inodes. Below is the sample output that highlights the problem.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       17G  7.7G  9.0G  46% /
devtmpfs       1001M  4.0K 1001M   1% /dev
none            201M  168K  201M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none           1001M   76K 1001M   1% /run/shm
/dev/xvdg        18G   11G  5.9G  66% /srv/mobi_BACKUP
# df -i
Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/xvda1     1066240 1061813    4427  100% /
devtmpfs        256189     333  255856    1% /dev
none            256229     250  255979    1% /run
none            256229       2  256227    1% /run/lock
none            256229      29  256200    1% /run/shm
/dev/xvdg      1179648   68692 1110956    6% /srv/mobi_BACKUP

The Problem

It should be very hard to exhaust the inodes unless huge amount of 0-sized or very small files are created. If you look in the image above, more than 1 million inodes have been used.

The Solution

It’s necessary to locate all those small files in the partition to get to the source of the problem. We want to find folders with unusual large number of files. A bash command can be of help starting at the root / (borrowed from Ivan).

for i in /*; do echo $i; find $i | wc -l; done

The command will list the directories with the number of files inside, from that point on you can keep on looking up some sub-folders until you find the source. If a sub-folder has too many files, the script might just hang for a long time, so you might have found the troublemaker.

for i in /var/lib/*; do echo $i; find $i | wc -l; done

In this particular case it turned out that PHP sessions were at the source of the issue.

Reason: PHP Sessions

This server is configured to keep the sessions using files thus the Garbage Collection does not take place. Therefore, the sessions have to be cleaned manually. From the previous Stack Overflow link, a cron task can be used for the cleanup:

# /etc/cron.d/php5: crontab fragment for php5
#  This purges session files older than X, where X is defined in seconds
#  as the largest value of session.gc_maxlifetime from all your php.ini
#  files, or 24 minutes if not defined.  See /usr/lib/php5/maxlifetime

# Look for and purge old sessions every 30 minutes
09,39 *     * * *     root   [ -d /var/lib/php5 ] && find /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -print0 | xargs -r -0 rm

In order to work, the previous code has to be located inside the file /etc/cron.d/php5

The time can be adjusted according to the actual PHP configuration.

In this particular case there were about 1 million PHP session files, running the cron script would just take too much time. Therefore, a manual delete did the work.