Linux server support

Disk FullDisk full

It’s been a while since I had a Linux server run out of inodes.  A long while.  In fact its only EVER happened to me when I was trying to build a CDDB. It’s happened a few times for our customers.  Always as a result of not correctly managing a server.

Today it happened for a customer, and this time I thought I’d blog about it.  I will stress that this customer likes to manage their own server knowing they can defer to us should they need to.

Disk full? But it isn’t!

So the customer called to say they’d tried to patch the server, but it had run out of disk space.  This is an Ubuntu server and it annoyingly fills up it’s boot partition when new kernels are available, so I logged in with fixing that issue in mind:

root@mail:~/Maildir# df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/mail0-root   18G  5.9G   11G  36% /
udev                    1.9G  4.0K  1.9G   1% /dev
tmpfs                   377M  324K  377M   1% /run
none                    5.0M  4.0K  5.0M   1% /run/lock
none                    1.9G     0  1.9G   0% /run/shm
/dev/sda1               228M  179M   38M  83% /boot
/dev/mapper/home-home   123G   81G   37G  69% /home

Boot is fine, space left…  Root is fine space left… so not that.  Must be inode:

root@mail:~/Maildir# df -i
Filesystem              Inodes  IUsed   IFree IUse% Mounted on
/dev/mapper/mail0-root 1163264 1139888   23376   98% /
/dev/mapper/home-home  8192000 627119 7564881    8% /home

Yep inodes, and the root partition too… damn.

This is a mail server and the users are located in the /home partition so its not a “user” getting lots of small emails.  But it is likely to be a user getting lots of small emails!  So I ran the following command in the /root folder (that’s a user on the system that isn’t stored in the /home partition:

for i in `find . -type d `; do echo `ls -a $i | wc -l` $i; done | sort -n

Which showed that there was over 700K files inside the /root/Maildir folder.  Checking the emails on the server showed that this was lack of server administration on the part of the customer (reading, dealing with and deleting their emails).  A few conversations later and deleting all the emails in the root folder yielded:

root@mail:~/Maildir# df -i
Filesystem              Inodes  IUsed   IFree IUse% Mounted on 
/dev/mapper/mail0-root 1163264 397429  765835   35% /

Space on inodes and files side!  Back over to the customer.

What do we learn from this?

For the customer they need to understand that sysadmin means understanding the whole system, and checking all parts of the server and not just the usual watering and feeding that they currently perform.

For us it’s to suggest to our customers that they might like to lose the sysadmin part of their day and hand that over to us!