Did you really ran out of disk space?

I’ve been using Fedora Linux for over a decade, and upgrading the OS is a routine process for me. I started an upgrade about 36 hours ago on my personal laptop, expecting it to be smooth—just like the past few years. However, this time, things went sideways. I encountered an error stating that there was no space left on the device. That was odd because I always double-check disk space before performing any upgrades on my laptop or servers. So, how big was this upgrade?

Unfortunately, my laptop was completely unbootable. To troubleshoot, I burned a live distro onto a USB drive, booted from it, and mounted my disk. Running df -h confirmed that the available space was indeed 0:

$ df -h
Filesystem   Size   Used   Avail   Use%    Mounted on
/dev/dm-0    237G  190G   0   100%   /

The available space was 0, but mathematically, 237G – 190G should leave about 47G free. So, where did my disk space disappear?

The Root Cause: Btrfs Metadata Exhaustion

A few years ago, Fedora recommended using Btrfs as the default filesystem. I accepted the recommendation, read a bit about it, and moved on. What I didn’t realize was that Btrfs stores filesystem metadata separately from data, and df -h only reports the data partition’s space usage. I had actually run out of metadata space.

To check Btrfs usage, I ran the following command, replacing / with the correct mount path from my live USB session:

# btrfs fi us /

Here’s the relevant output:

Metadata,single: Size:9.0GiB, Used:8.5GiB (94.44%)

Btrfs reserves about 0.5GB for metadata changes, and when this space is exhausted, the system effectively runs out of space. The solution? I needed to balance Btrfs chunks to reclaim space.

Reclaiming Space with Btrfs Balance

Running the following command helped me reclaim space by consolidating underutilized chunks:

btrfs balance start -dusage=5 &lt;btrfs partition path>

This command finds all chunks that are less than 5% full and rewrites them into new chunks, freeing up space. I incrementally increased the dusage value up to 50%, which freed about 15-20 chunks.

However, if metadata is completely full, the balance operation won’t work because it needs some free space to operate. To fix this, I created temporary space by adding a loopback device:

$ dd if=/dev/zero of=/var/tmp/btrfs bs=1G count=5
$ losetup -v -f /var/tmp/btrfs
$ losetup -a  # Identify the loop device (e.g., /dev/loop3)
$ btrfs device add /dev/loop3 &lt;btrfs partition path>

Once the loopback device was added, I retried the balance command, and this time, it worked perfectly.

Fixing the Broken Fedora Upgrade

With space freed up, I removed all USB drives and attempted to boot my laptop. As expected, it didn’t boot properly; instead, it showed a screen with a mouse pointer replaced by an “X.” Luckily, I was able to switch to a TTY session using Ctrl+Alt+F3.

I attempted to synchronize my system with the new release:

dnf distro-sync --releasever=41

However, this threw an error:

Problem: The operation would result in removing the following protected packages: sudo

Since my system was already broken, I decided to override the protected package list:

dnf distro-sync --releasever=41 --setopt=protected_packages=

After a long wait, the synchronization completed successfully, and my laptop finally booted into Fedora 41.

Lessons Learned

RTFM – Btrfs docs saved the day.
Always take a Btrfs snapshot before upgrading.
Monitor both data and metadata space when using Btrfs.
Distro-sync with protected packages overridden can help recover a broken Fedora upgrade. Use it only as a last resort though.

This was a frustrating yet valuable learning experience. Hopefully, this post helps anyone who runs into a similar issue!

Aditya Patawari