You may occasionally come across an issue where runningÂ
dfÂ will produce output that disagreeâ€™s with the output of theÂ
duÂ command. If you arenâ€™t familiar withÂ these two commands do see my post aboutÂ filesystem and directory size. The reason for the difference in reported size is thatÂ
dfÂ does not differentiate between files that are open in memory but have been deleted, or altered on the disk, whereasÂ
duÂ will only see the files that are on the disk. You should recognize that these tools serve different functions and that you will need to rely on both of them to get a truly accurate portrayal of disk usage on your system.
Lets say you runÂ Â
df -hÂ to get an idea of how much space you have on each of the filesystems on your server or PC only to see that /var is 98% full, 9.8G out of 10G just to keep it simple. Like a good admin you runÂ Â
du -h --max-depth=1 /varÂ to find out which directories are the largest and may have files that need to be zipped up, moved, or deleted. The problem becomes apparent whenÂ Â
duÂ returns that just 3G are in use on that filesystem. What do you do now?
Check for deleted files in memory.
Have you heard the old saying around the Unix world that â€œEverything is a fileâ€. Well itâ€™s true, everything in Unix, and by association Linux, is a file. This includes deleted files that now live as chunks of memory that are in use by a process.
You can view all open files on a system withÂ theÂ Â
lsofÂ command, including deleted files that live in memory and are in use by a process (for example an old configuration file). For instance:
sudo lsof | grep root
will show you a full output of all the files currently in use by the root user. (Probably a lot of files). RunningÂ Â
sudo lsof | lessÂ will show you all of the open files on your system. It will look something like this. (Iâ€™m only grabbing the first 3 lines for brevity).
COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd 1 root cwd DIR 202,1 4096 2 / systemd 1 root rtd DIR 202,1 4096 2 / systemd 1 root txt REG 202,1 1577232 396000 /lib/systemd/systemd
Here you can see the command, the process id (PID), which user has the file open, the file descriptor (FD), the size in bytes, and the location. In our scenario we want to find out if there are any large files open that may have been deleted. We can find those files like this:
sudo lsof | grep -i deleted
Keep an eye on the 8th column which if you recall is the SIZE column. Once you identify your large files check which user has the file open (4th column), usually this will be a service account like www-data, apache, mysql.Â Or pay attention to the command column to identify the process or service that is using the old file. After you identify the offending process all you need to do is restart the service usingÂ Â
systemctl, service, or kill -HUP
Donâ€™t panic, take a breath, and assess what you are seeing, think aboutÂ how your tools work and what they are showing you. Above all donâ€™t just start deleting things to free up space! The reason thatÂ
dfÂ andÂ Â
duÂ are having a disagreement here is thatÂ Â
dfÂ seeâ€™s these deleted files along with their replacements and calculates the total disk usage,Â Â
duÂ on the other hand only seeâ€™s the new file. Now that you know how to find the zombie files you shouldnâ€™t have too much trouble bringing these two system tools back into agreement.
Luke has an RHCSA for Red Hat Enterpirse Linux 7 and currently works as a Linux Systems Adminstrator in Ohio.