$ df -i -h /; df -h / Filesystem Inodes IUsed IFree IUse% Mounted on tmpfs 7.9M 160K 7.7M 2% / Filesystem Size Used Avail Use% Mounted on tmpfs 2.0G 1.9G 189M 91% / $ du -h -x / -d 0 26M /
How can it have 1.9G used if there's only 26M there? The
-x option for
du tells it not to visit other partitions so that should be everything on
A quick search suggested that I probably had some large deleted files open and to check
lsof, but there wasn't anything there either. There weren't many other suggestions and I was stuck here for a while.
While looking at
mount's output I noticed something suspicious:
tmpfs on /var/lib/containers/storage/overlay type tmpfs (rw,relatime,size=2097152k,mode=755)
That has the same mount options as the
/ tmpfs. It seems that somehow podman put
/var/lib/containers/storage/overlay on the
/ tmpfs, even though I bind mounted
/var/lib/containers on persistent storage.
This NixOS system uses a tmpfs / in the style of nix-community/impermanence to reduce statefulness.
Quick debugging tip: bind mount your / somewhere else
Here's how to really see what's on your root fs. Bind mount it somewhere else so other mounts can't get in the way of finding files on it.
# mount -o bind / /mnt/actuallyroottho # du -h -x /mnt/actuallyroottho -d 1 ... 1.9G /mnt/actuallyroottho/var ...