Disk full
It’s been a while since I had a Linux server run out of inodes. A long while. In fact its only EVER happened to me when I was trying to build a CDDB. It’s happened a few times for our customers. Always as a result of not correctly managing a server.
Today it happened for a customer, and this time I thought I’d blog about it. I will stress that this customer likes to manage their own server knowing they can defer to us should they need to.
Disk full? But it isn’t!
So the customer called to say they’d tried to patch the server, but it had run out of disk space. This is an Ubuntu server and it annoyingly fills up it’s boot partition when new kernels are available, so I logged in with fixing that issue in mind:
root@mail:~/Maildir# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/mail0-root 18G 5.9G 11G 36% /
udev 1.9G 4.0K 1.9G 1% /dev
tmpfs 377M 324K 377M 1% /run
none 5.0M 4.0K 5.0M 1% /run/lock
none 1.9G 0 1.9G 0% /run/shm
/dev/sda1 228M 179M 38M 83% /boot
/dev/mapper/home-home 123G 81G 37G 69% /home
Boot is fine, space left… Root is fine space left… so not that. Must be inode:
root@mail:~/Maildir# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/mail0-root 1163264 1139888 23376 98% /
/dev/mapper/home-home 8192000 627119 7564881 8% /home
Yep inodes, and the root partition too… damn.
This is a mail server and the users are located in the /home
partition so its not a “user” getting lots of small emails. But it is likely to be a user getting lots of small emails! So I ran the following command in the /root folder (that’s a user on the system that isn’t stored in the /home
partition:
for i in `find . -type d `; do echo `ls -a $i | wc -l` $i; done | sort -n
Which showed that there was over 700K files inside the /root/Maildir
folder. Checking the emails on the server showed that this was lack of server administration on the part of the customer (reading, dealing with and deleting their emails). A few conversations later and deleting all the emails in the root folder yielded:
root@mail:~/Maildir# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/mail0-root 1163264 397429 765835 35% /
Space on inodes and files side! Back over to the customer.
What do we learn from this?
For the customer they need to understand that sysadmin means understanding the whole system, and checking all parts of the server and not just the usual watering and feeding that they currently perform.
For us it’s to suggest to our customers that they might like to lose the sysadmin part of their day and hand that over to us!