You know that sinking feeling when your server starts throwing disk space errors at 2 AM? I’ve been there more times than I’d like to admit. After ten years of managing Linux systems, I’ve learned that checking disk usage isn’t just about running a command—it’s about understanding what you’re looking at and knowing where to dig when things get weird.
Let me walk you through everything I wish someone had told me when I was starting out.
Why Disk Usage Monitoring Matters (And Why You’re Probably Doing It Wrong)
Here’s the thing most tutorials won’t tell you: disk space problems are rarely about the disk being actually full. They’re about inodes running out, processes holding onto deleted files, or mount points hiding gigabytes of data underneath them. I once spent three hours tracking down a “disk full” error only to discover a log rotation script had failed six months earlier, and a single log file had grown to 47GB.
The first step to solving disk space issues is understanding the difference between the two workhorses of disk analysis: df and du.
Understanding df vs du: The Foundation of Disk Analysis
Think of df (disk free) as your satellite view and du (disk usage) as your ground-level investigation. They serve completely different purposes, and confusing them is like using a map when you need a metal detector.
The df Command: Your High-Level Overview
df reports on filesystem-level disk space. It asks the filesystem itself, “Hey, how much space do you have?” This is fast, efficient, and gives you the big picture.
df -h
The -h flag gives you human-readable output (megabytes and gigabytes instead of blocks). Here’s what you’ll see:
Filesystem Size Used Avail Use% Mounted on /dev/sda1 50G 42G 5.5G 89% / /dev/sdb1 200G 156G 34G 82% /var tmpfs 7.8G 1.2M 7.8G 1% /run
That /var partition at 82%? That’s where I’d start investigating. Red Hat’s engineering team has written extensively about why df and du sometimes show different numbers—it usually comes down to deleted files still being held open by processes.
The du Command: Your Detail-Oriented Investigator
While df gives you the aerial view, du actually walks through directories and adds up file sizes. It’s slower, but it tells you exactly where your disk space went.
du -sh /var/*
The -s flag summarizes (instead of showing every subdirectory), and -h keeps things readable:
4.2G /var/cache 38G /var/log 2.1G /var/lib 890M /var/tmp
Boom. There’s your culprit: 38GB in logs. Now we’re getting somewhere.
Finding the Space Hogs: Practical Techniques
Once you’ve identified a problematic partition with df, you need to drill down. Here’s my standard troubleshooting workflow.
Quick Sorting to Find Large Directories
Pipe du through sort to see your biggest offenders first:
du -h /var | sort -rh | head -20
This shows the top 20 largest directories in descending order. The -r reverses the sort (biggest first), and head -20 limits output to 20 lines. Simple, effective, and I use this almost daily.
Finding Individual Large Files
Sometimes it’s not a directory—it’s one massive file hiding somewhere. The find command is perfect for this:
find /var -type f -size +1G -exec ls -lh {} \; 2>/dev/nullThis hunts down any file over 1GB in /var. The 2>/dev/null suppresses permission errors—you’re looking for big files, not dealing with access denied messages.
The Hidden Killer: Deleted Files Held by Processes
Here’s a scenario that’s bitten me more than once: df says the disk is 95% full, but du can’t find enough files to account for it. What gives?
A process has an open file handle to a deleted file. The file is gone from the filesystem, but the kernel can’t reclaim the space until the process closes it or dies.
lsof | grep deleted | grep /var
This shows processes holding deleted files. Once you find the culprit, you can kill the process (if it’s safe) or restart the service. The space will immediately become available again.
Server Fault has an excellent thread documenting this exact issue with detailed troubleshooting steps.
Using ncdu: The Tool I Wish I’d Found Sooner
After years of piping du into sort and squinting at terminal output, I discovered ncdu—and it changed everything.
ncdu (NCurses Disk Usage) is an interactive disk usage analyzer. Think of it as du with a user interface that doesn’t make you want to cry.
Installing ncdu
Most distributions include it in their repos:
# Debian/Ubuntu sudo apt install ncdu # RHEL/CentOS (requires EPEL) sudo yum install ncdu # Arch sudo pacman -S ncdu
Using ncdu Effectively
Just point it at a directory and let it scan:
ncdu /var
It’ll build an interactive tree you can navigate with arrow keys. Press Enter to dive into a directory, press d to delete files (carefully!), and press q to quit.
The real magic? You can sort by size, see percentages at a glance, and drill down without typing new commands every 30 seconds. TecMint’s ncdu guide covers all the navigation shortcuts if you want to become a power user.
For production systems, I still prefer the command-line tools because they’re scriptable and available everywhere. But for interactive troubleshooting? ncdu wins every time.
Automating Disk Usage Monitoring
Manual checks are fine when you’re actively troubleshooting, but you don’t want to discover disk space issues at 2 AM. You want to know about them before they become critical.
Setting Up a Simple Monitoring Script
Here’s a basic script I use on systems without full monitoring:
#!/bin/bash
# disk_monitor.sh
THRESHOLD=85
EMAIL="your@email.com"
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
usage=$(echo $output | awk '{ print $1}' | cut -d'%' -f1)
partition=$(echo $output | awk '{ print $2 }')
if [ $usage -ge $THRESHOLD ]; then
echo "Disk usage on $partition is at ${usage}%" | mail -s "Disk Space Alert" $EMAIL
fi
done
Save that, make it executable (chmod +x disk_monitor.sh), and set up a cron job to run it daily.
Checking Inode Usage
Sometimes you’ll run out of inodes before you run out of space—especially on systems that create millions of tiny files. Check with:
df -i
If the inode usage (IUse%) is above 90%, you’ve got too many files. This often happens with mail servers, log directories, or poorly configured temporary file cleanup.
Advanced Troubleshooting: When Nothing Makes Sense
Every so often you’ll hit a situation where the numbers just don’t add up. Here are a few gotchas I’ve encountered.
Mount Points Hiding Data
If you created files in /var before mounting a partition there, those files still exist—they’re just hidden underneath the mount point. Unmount the partition temporarily to check:
sudo umount /var ls -lh /var sudo mount /var
If you see files there, that’s your hidden data.
Reserved Filesystem Space
Most ext4 filesystems reserve 5% of space for root. On a 1TB drive, that’s 50GB that normal users can’t access. Check with:
sudo tune2fs -l /dev/sda1 | grep "Reserved block count"
If you’re tight on space and don’t need that safety buffer, you can reduce it:
sudo tune2fs -m 1 /dev/sda1
This sets the reserved space to 1% instead of 5%. I do this on data partitions but leave system partitions alone—you want that safety margin when your root filesystem starts filling up.
Using iotop to Find Active Writers
If disk space is disappearing in real-time, iotop shows which processes are actively writing to disk right now:
sudo iotop -o
The -o flag shows only processes doing I/O. Combine this with watch df -h in another terminal, and you can catch the culprit red-handed.
Putting It All Together: A Real-World Example
Last month, a client’s database server hit 98% disk usage. Here’s how I tracked it down in about five minutes:
- Ran
df -hto confirm which partition was full:/varat 98% - Used
du -sh /var/*to identify the problem:/var/lib/postgresqlwas enormous - Checked PostgreSQL logs—WAL archiving had failed, and write-ahead logs were accumulating
- Fixed the archiving configuration and manually cleared old WAL files
- Disk usage dropped to 42% immediately
The key was understanding the tools and knowing where to look. If I’d just started deleting random log files, I would’ve missed the actual problem.
Best Practices for Disk Usage Management
After dealing with hundreds of disk space emergencies, here’s what actually works:
- Monitor proactively: Set up alerts at 80% usage, not 95%. You want time to plan, not panic.
- Automate log rotation: Use
logrotateand test your configs. Most disk space disasters are unrotated logs. - Understand your growth patterns: Database partitions grow differently than web server partitions. Know your system.
- Keep ncdu installed: It’s a 2MB package that’ll save you hours of frustration.
- Document your filesystem layout: Know what’s mounted where and why. Future you will be grateful.
Final Thoughts
Disk space management isn’t glamorous, but it’s fundamental. Master df, du, and ncdu, understand the difference between them, and you’ll troubleshoot 90% of disk issues faster than most people can Google the error message.
The other 10%? That’s where you learn the really interesting stuff—like the time I discovered a rogue backup script had been creating daily snapshots for three years without deleting old ones. But that’s a story for another article.
Now go check your disk usage. I’ll bet you find something interesting.
