The kernel team is working on final integration for Linux kernel 6.1. This version was just recently released, and will arrive soon in Fedora Linux. As a result, the Fedora Linux kernel and QA teams have organized a test week from Tuesday, Jan 03, 2023 to Sunday, Jan 07, 2023. Refer to the wiki page for links to the test images you’ll need to participate. Continue reading for details.

How does a test week work?

A test week is an event where anyone can help make sure changes in Fedora Linux work well in an upcoming release. Fedora community members often participate, and the public is welcome at these events. If you’ve never contributed before, this is a perfect way to get started.

To contribute, you only need to be able to do the following things:

- Download test materials, which include some large files
- Read and follow directions step by step

The wiki page for the kernel test day has a lot of good information on what and how to test. After you’ve done some testing, you can log your results in the test day web application. If you’re available on or around the days of the event, please do some testing and report your results. We have a document which provides all the necessary steps.

Happy testing, and we hope to see you on test day.

This article will explore what Btrfs snapshots are, how they work, and how you can benefit from taking snapshots in every-day situations. This is part of a series that takes a closer look at Btrfs, the default filesystem for Fedora Workstation and Fedora Silverblue since Fedora Linux 33.

In case you missed it, here’s the previous article from this series: https://fedoramagazine.org/working-with-btrfs-subvolumes/

Introduction

Imagine you work on a file over extended periods of time, repeatedly adding changes and undoing them. Then, at some point you realize: Parts of the changes you undid two hours ago would be very helpful now. And yesterday you had already changed this particular bit, too, before you trashed that design. But of course, because you regularly save your files, old changes are lost. Many people have probably experienced a situation like this before. Wouldn’t it be great if you could recover old file versions without having to manually copy them at regular intervals?

This is just one typical situation where Btrfs snapshots can help you out. When used correctly, snapshots also give you a great backup solution for your PC.

Below you will find a lot of examples related to snapshots. If you want to follow along, you must have access to a Btrfs filesystem and root access. You can check the file system of a directory using the following command:

$ findmnt -no FSTYPE /home
btrfs

Here the findmnt command shows the type of filesystem for your /home/ directory. If it says btrfs, you’re all set. Let’s create a new directory in which to perform some experiments:

$ mkdir ~/btrfs-snapshot-test
$ cd ~/btrfs-snapshot-test

In the text below, you will find lots of command responses in boxes such as shown above. Please keep in mind while reading/comparing command output that the box contents may be wrapped at the end of the line. This may make it difficult to recognize long lines that are broken across multiple lines for readability. When in doubt, try to resize your browser window and see how the text behaves!

Snapshots in Btrfs

Let’s start with an elementary question: What is a Btrfs snapshot? If you look in the Docs [1] and Wiki [2], you won’t immediately find an answer to this question. In fact, it is nowhere to be found in the “Features” section. If you search a little, you will find snapshots mentioned extensively along with Btrfs subvolumes [3]. So now what?

Remember that snapshots were both mentioned in the previous articles of this series? There it said:

What is the advantage of CoW? In simple terms: a history of the modified and edited files can be kept. Btrfs will keep the references to the old file versions (inodes) somewhere they can be easily accessed. This reference is a snapshot: An image of the filesystem state at some point in time.

Working with Btrfs: General Concepts

and also:

Another advantage of separating / and /home is that you can take snapshots separately. A subvolume is a boundary for snapshots, and snapshots will never contain the contents of other subvolumes below the subvolume that the snapshot is taken of.

Working with Btrfs: Subvolumes

It seems snapshots have something to do with Btrfs subvolumes. You may have heard about snapshots in other contexts before, for example with LVM, the Logical Volume Manager. While technically they serve the same purpose, they are different in terms of how they reach their goal.

Every Btrfs snapshot is a subvolume. However, not every subvolume is a snapshot. The difference is in what the subvolume contains. A snapshot is a subvolume with added content: it holds references to current and/or past versions of files (inodes). Let’s see where snapshots come from!

Creating Btrfs snapshots

To use snapshots, you need a Btrfs subvolume to take snapshots of. Let’s create one inside our test folder (~/btrfs-snapshot-test):

$ cd ~/btrfs-snapshot-test
$ sudo btrfs subvolume create demo
Create subvolume './demo'
$ sudo chown -R $(id -u):$(id -g) demo/
$ cd demo

Since by default Btrfs subvolumes are owned by root, you must call chown to modify the files in the subvolume to be owned by a regular user. Now add a few files inside it:

$ touch foo bar baz
$ echo "Lorem ipsum dolor sit amet, " > foo

Your directory now looks something like this:

$ ls -l
total 4
-rw-r--r--. 1 hartan hartan 0 Dec 20 08:11 bar
-rw-r--r--. 1 hartan hartan 0 Dec 20 08:11 baz
-rw-r--r--. 1 hartan hartan 29 Dec 20 08:11 foo

Let’s create the very first snapshot from that:

$ cd ..
$ sudo btrfs subvolume snapshot demo demo-1
Create a snapshot of 'demo' in './demo-1'

And that’s it. Let’s see what was achieved:

$ ls -l
total 0
drwxr-xr-x. 1 hartan hartan 18 Dec 20 08:11 demo
drwxr-xr-x. 1 hartan hartan 18 Dec 20 08:11 demo-1
$ tree
.
├── demo
│   ├── bar
│   ├── baz
│   └── foo
└── demo-1 ├── bar ├── baz └── foo 2 directories, 6 files

It seems it made a copy! To verify, let’s read the contents of foo from the snapshot:

$ cat demo/foo
Lorem ipsum dolor sit amet,
$ cat demo-1/foo
Lorem ipsum dolor sit amet,

The real effect becomes apparent when we modify the original file:

$ echo "consectetur adipiscing elit, " >> demo/foo
$ cat demo/foo
Lorem ipsum dolor sit amet, consectetur adipiscing elit,
$ cat demo-1/foo
Lorem ipsum dolor sit amet,

This shows that the snapshot still holds the “old” version of the data: The content of foo hasn’t changed. So far, you could have achieved the exact same thing with a simple file copy. You can now go ahead and continue working on the old file, too:

$ echo "sed do eiusmod tempor incididunt" >> demo-1/foo
$ cat demo-1/foo
Lorem ipsum dolor sit amet, sed do eiusmod tempor incididunt

Under the hood, however, our snapshot is in fact a new Btrfs subvolume. You can verify this with the following command:

$ sudo btrfs subvolume list -o .
ID 259 gen 265 top level 256 path home/hartan/btrfs-snapshot-test/demo
ID 260 gen 264 top level 256 path home/hartan/btrfs-snapshot-test/demo-1

Btrfs snapshots vs. file copies

So what’s the point of all this? Up until now snapshots seem to be a complicated way to copy files around. In fact, there is more to snapshots than meets the eye. Let’s create a bigger file:

$ dd if=/dev/urandom of=demo/bigfile bs=1M count=512
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 1.3454 s, 399 MB/s

There is now a new file demo/bigfile that is 512 MiB in size. Let’s make another snapshot so you don’t lose it when you modify the data:

$ sudo btrfs subvolume snapshot demo demo-2
Create a snapshot of 'demo' in './demo-2'

Now let’s simulate some changes by appending a small string to the file:

$ echo "small changes" >> demo/bigfile

Here’s the resulting file structure:

$ tree
.
├── demo
│   ├── bar
│   ├── baz
│   ├── bigfile
│   └── foo
├── demo-1
│   ├── bar
│   ├── baz
│   └── foo
└── demo-2 ├── bar ├── baz ├── bigfile └── foo 3 directories, 11 files

But the real magic happens somewhere else. Had you copied demo/bigfile, you would now have two files of about 512 MiB in size with mostly the same content. However, since they are distinct copies, they would occupy about 1 GiB of storage total. Keep in mind that the difference between both files is hardly more than 10 Bytes – that’s almost nothing compared to the original file size.

Btrfs snapshots work different than file copies: They keep references to current and past inodes instead. When you appended the change to the file, under the hood Btrfs allocated some more space to store the changes in and added a reference to this new data to the original inode. The previous contents remain untouched. If it helps your mental model, you can think of this as “storing” merely the difference between the original file and the modified version.

Let’s have a look at the effect of this:

$ sudo compsize .
Processed 11 files, 5 regular extents (9 refs), 3 inline.
Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 512M 512M 1.0G none 100% 512M 512M 1.0G

The interesting figure here is seen in line “TOTAL”:

• “Referenced” is the total size of all the files in the current directory, summed up
• “Disk Usage” is the amount of storage space allocated on your disk to store the files

While you have a total of 1 GiB files, it takes merely 512 MiB to store them.

Btrfs snapshots and backups

So far, in this article, you have seen how to create Btrfs snapshots and what makes them so special. One may be tempted to think: If I take a series of Btrfs snapshots locally on my PC, I have a solid backup strategy. This is not the case. If the underlying data, which is shared by Btrfs subvolumes, is accidentally damaged (by something outside of Btrfs’ influence, e.g. cosmic rays), all the subvolumes pointing to this data contain the same error.

To turn the snapshots into real backups you should store them on a different Btrfs filesystem, such as on an external drive. For the purposes of this article let’s create a new Btrfs filesystem contained inside a file and mount it to simulate an external drive. If you have an external drive formatted with Btrfs lying around, feel free to substitute all the paths mentioned in the following commands to try it out! Let’s create a new Btrfs filesystem:

Note: The commands below will create a new file of 8 GB size on your filesystem. If you want to follow the steps below, please ensure you have at least 8 GB of disk space available. Do not allocate less than 8 GB to the file, as Btrfs may otherwise encounter issues during mounting.

$ truncate -s 8G btrfs_filesystem.img
$ sudo mkfs.btrfs -L "backup-drive" btrfs_filesystem.img
btrfs-progs v5.18
See http://btrfs.wiki.kernel.org for more information. [ ... ] Devices: ID SIZE PATH 1 8.00GiB btrfs_filesystem.img

These commands created a new file of 8 GB in size named btrfs_filesystem.img and formatted a Btrfs filesystem inside it. Now you can mount it as if it were an external drive:

$ mkdir backup-drive
$ sudo mount btrfs_filesystem.img backup-drive
$ sudo chown -R $(id -u):$(id -g) backup-drive
$ ls -lh
total 4.7M
drwxr-xr-x. 1 hartan hartan 0 Dec 20 08:35 backup-drive
-rw-r--r--. 1 hartan hartan 8.0G Dec 20 08:37 btrfs_filesystem.img
drwxr-xr-x. 1 hartan hartan 32 Dec 20 08:14 demo
drwxr-xr-x. 1 hartan hartan 18 Dec 20 08:11 demo-1
drwxr-xr-x. 1 hartan hartan 32 Dec 20 08:14 demo-2

Great, now there is an independent Btrfs filesystem mounted under backup-drive! Let’s try to take another snapshot and place it there:

$ sudo btrfs subvolume snapshot demo backup-drive/demo-3
Create a snapshot of 'demo' in 'backup-drive/demo-3'
ERROR: cannot snapshot 'demo': Invalid cross-device link

What happened? Well, you tried to take a snapshot of demo and store it in a different Btrfs filesystem (a different device from Btrfs’ point of view). Remember that a Btrfs subvolume only holds references to files and their contents (inodes)? This is exactly the problem: The files and contents exist in our home filesystem, but not in the newly-created backup-drive. You have to find a way to transfer the subvolume along with its contents to the new filesystem.

Storing snapshots on a different Btrfs filesystem

The Btrfs utilities include two special commands for this purpose. Let’s see how they work first:

$ sudo btrfs send demo | sudo btrfs receive backup-drive/
ERROR: subvolume /home/hartan/btrfs-snapshot-test/demo is not read-only
ERROR: empty stream is not considered valid

Another error! This time it tells you that the subvolume we’re trying to transfer is not read-only. This is true: You can write new contents to all of the snapshots/subvolumes created so far. You can create read-only snapshots like this:

$ sudo btrfs subvolume snapshot -r demo demo-3-ro
Create a readonly snapshot of 'demo' in './demo-3-ro'

Unlike previously, here the -r option is added to the snapshot subcommand. This creates a read-only snapshot, which is easily verified:

$ touch demo-3-ro/another-file
touch: cannot touch 'demo-3-ro/another-file': Read-only file system

Now you can retry transferring the subvolumes:

$ sudo btrfs send demo-3-ro | sudo btrfs receive backup-drive/
At subvol demo-3-ro
At subvol demo-3-ro
$ tree ├── backup-drive
│   └── demo-3-ro
│   ├── bar
│   ├── baz
│   ├── bigfile
│   └── foo
├── btrfs_filesystem.img
├── demo
[ ... ]
└── demo-3-ro ├── bar ├── baz ├── bigfile └── foo 6 directories, 20 files

It worked! You have successfully transferred a read-only snapshot of our original subvolume demo to an external Btrfs filesystem.

Storing snapshots on non-Btrfs filesystems

Above you have seen how you can store Btrfs subvolumes/snapshots on another Btrfs filesystem. But what can you do if you do not have another Btrfs filesystem and cannot create one, for example because the external drives need a filesystem that allows compatibility with Windows or MacOS hosts? In such cases you can store subvolumes in files:

$ sudo btrfs send -f demo-3-ro-subvolume.btrfs demo-3-ro
At subvol demo-3-ro
$ ls -lh demo-3-ro-subvolume.btrfs -rw-------. 1 root root 513M Dec 21 10:39 demo-3-ro-subvolume.btrfs

The file demo-3-ro-subvolume.btrfs now contains everything that is needed to recreate the demo-3-ro subvolume at a later point in time.

Incrementally sending subvolumes

If you perform this action repeatedly for different subvolumes, you will notice at some point that the different subvolumes do not share their file contents any more. This is because when sending a subvolume such as above, all the data needed to recreate this standalone subvolume is transferred to the target. You can, however, instruct Btrfs to only send the difference between two subvolumes to the target! This so-called incremental send will ensure that shared references remain shared between the subvolumes. To demonstrate this, add a few more changes to our original subvolume:

$ echo "a few more changes" >> demo/bigfile

Next create another read-only snapshot:

$ sudo btrfs subvolume snapshot -r demo demo-4-ro
Create a readonly snapshot of 'demo' in './demo-4-ro'

And now send it:

$ sudo btrfs send -p demo-3-ro demo-4-ro | sudo btrfs receive backup-drive
At subvol demo-4-ro
At snapshot demo-4-ro

In the command above, the -p option specifies a parent subvolume, against which the differences are calculated. It is important to keep in mind that both the source and target Btrfs filesystem must contain the same, unmodified parent subvolume! Ensure that the new subvolume is really there:

$ ls backup-drive/
demo-3-ro demo-4-ro
$ ls -lR backup-drive/demo-4-ro/
backup-drive/demo-4-ro/:
total 524296
-rw-r--r--. 1 hartan hartan 0 Dec 20 08:11 bar
-rw-r--r--. 1 hartan hartan 0 Dec 20 08:11 baz
-rw-r--r--. 1 hartan hartan 536870945 Dec 21 10:49 bigfile
-rw-r--r--. 1 hartan hartan 59 Dec 20 08:13 foo

But how do you know whether the incremental send only transferred the difference between both subvolumes? Let’s transfer the data stream to a file and see how big it is:

$ sudo btrfs send -f demo-4-ro-diff.btrfs -p demo-3-ro demo-4-ro
At subvol demo-4-ro
$ ls -l demo-4-ro-diff.btrfs -rw-------. 1 root root 315 Dec 21 10:55 demo-4-ro-diff.btrfs

According to ls, the file is merely 315 bytes in size! This means that the incremental send only transferred the changes between the two subvolumes, along with additional Btrfs-specific metadata.

Restoring subvolumes from snapshots

Before continuing, let’s do some cleaning up of the things you don’t need at the moment:

$ sudo rm -rf demo-4-ro-diff.btrfs demo-3-ro-subvolume.btrfs
$ sudo btrfs subvolume delete demo-1 demo-2 demo-3-ro demo-4-ro
$ ls -l
total 531516
drwxr-xr-x. 1 hartan hartan 36 Dec 21 10:50 backup-drive
-rw-r--r--. 1 hartan hartan 8589934592 Dec 21 10:51 btrfs_filesystem.img
drwxr-xr-x. 1 hartan hartan 32 Dec 20 08:14 demo

So far you have managed to create read/write and read-only snapshots of Btrfs subvolumes and send them to an external location. In order to turn this into a backup strategy, however, there has to be a way to send the subvolumes back to the original filesystem and make them writable again. For this purpose, let’s move the demo subvolume somewhere else and try to recreate it from the most recent snapshot. First: Rename the “broken” subvolume. It will be deleted once the restore was successful:

$ mv demo demo-broken

Second: Transfer the most recent snapshot back to this filesystem:

$ sudo btrfs send backup-drive/demo-4-ro | sudo btrfs receive .
At subvol backup-drive/demo-4-ro
At subvol demo-4-ro
[hartan@fedora btrfs-snapshot-test]$ ls
backup-drive btrfs_filesystem.img demo-4-ro demo-broken

Third: Create a read-write subvolume from the snapshot:

$ sudo btrfs subvolume snapshot demo-4-ro demo
Create a snapshot of 'demo-4-ro' in './demo'
$ ls
backup-drive btrfs_filesystem.img demo demo-4-ro demo-broken

The last step is important: You cannot just rename demo-4-ro to demo, because it would still be a read-only subvolume! Finally you can check whether everything you need is there:

$ tree demo
demo
├── bar
├── baz
├── bigfile
└── foo 0 directories, 4 files
$ tail -c -19 demo/bigfile a few more changes

The last command above tells you that the last 19 characters in bigfile are in fact the change last performed. At this point, you may want to copy recent changes from demo-broken to the new demo subvolume. Since you didn’t perform any other changes, you can now delete the obsolete subvolumes:

$ sudo btrfs subvolume delete demo-4-ro demo-broken
Delete subvolume (no-commit): '/home/hartan/btrfs-snapshot-test/demo-4-ro'
Delete subvolume (no-commit): '/home/hartan/btrfs-snapshot-test/demo-broken'

And that’s it! You have successfully restored the demo subvolume from a snapshot that was previously stored on a different Btrfs filesystem (external media).

Subvolumes as boundary for snapshots

In the second article of this series I mentioned that subvolumes are boundaries for snapshots, but what exactly does that mean? In simple terms, a snapshot of a subvolume will only contain the content of this particular subvolume, and none of the nested subvolumes below. Let’s have a look at this:

$ sudo btrfs subvolume create demo/nested
Create subvolume 'demo/nested'
$ sudo chown -R $(id -u):$(id -g) demo/nested
$ touch demo/nested/another_file

Let’s take a snapshot as before:

$ sudo btrfs subvolume snapshot demo demo-nested
Create a snapshot of 'demo' in './demo-nested'

And check out the contents:

$ tree demo-nested
demo-nested
├── bar
├── baz
├── bigfile
├── foo
└── nested 1 directory, 4 files $ tree demo
demo
├── bar
├── baz
├── bigfile
├── foo
└── nested └── another_file 1 directory, 5 files

Notice that another_file is missing, even though the folder nested is present. This happens because nested is a subvolume: The snapshot of demo contains the folder (mountpoint) for the nested subvolume, but its contents aren’t present. Currently there is no way to perform snapshots recursively to include nested subvolumes. However, we can take advantage of this to exclude folders from snapshots! This is typically useful for data that you can reproduce easily, or that will rarely change. Examples include virtual machine or container images, movies, game files and more.

Before we wrap up the article, let’s remove everything we created while testing:

$ sudo btrfs subvolume delete demo/nested demo demo-nested
Delete subvolume (no-commit): '/home/hartan/btrfs-snapshot-test/demo/nested'
Delete subvolume (no-commit): '/home/hartan/btrfs-snapshot-test/demo'
Delete subvolume (no-commit): '/home/hartan/btrfs-snapshot-test/demo-nested'
$ sudo umount backup-drive
$ cd ..
$ rm -rf btrfs-snapshot-test/

Final thoughts on Btrfs-based backups

If you decide you want to use Btrfs to perform regular backups of your data, you may want to use a tool that automates this task for you. The Btrfs wiki has a list of backup tools specialized on Btrfs [4]. On this page you will also find another summary of the steps to perform Btrfs backups by hand. Personally, I have had a lot of good experiences with btrbk [5] and I am using it to perform my own backups. In addition to backups, btrbk can also keep a list of Btrfs snapshots locally on your PC. I use this to safeguard against accidental data deletion.

If you want to know more about performing backups using Btrfs, leave a comment below and I’ll consider writing a follow-up article that deals exclusively with this topic.

Conclusion

This article investigated Btrfs snapshots, which are Btrfs subvolumes under the hood. You learned how to create read/write and read-only snapshots, and how this mechanism can help safeguard against data loss.

The next articles in this series will deal with:

• Compression – Transparently saving storage space
• Qgroups – Limiting your filesystem size
• RAID – Replace your mdadm configuration

If there are other topics related to Btrfs that you want to know more about, have a look at the Btrfs Wiki [2] and Docs [1]. Don’t forget to check out the first two articles of this series, if you haven’t already! If you feel that there is something missing from this article series, let me know in the comments below. See you in the next article!

Sources

[1]: https://btrfs.readthedocs.io/en/latest/Introduction.html
[2]: https://btrfs.wiki.kernel.org/index.php/Main_Page
[3]: https://btrfs.readthedocs.io/en/latest/Subvolumes.html
[4]: https://btrfs.wiki.kernel.org/index.php/Incremental_Backup#Available_Backup_Tools
[5]: https://github.com/digint/btrbk

Introduction

GitHub enables distributed and collaborative code development. To ensure software works correctly, many projects use continuous integration to build and test each new contribution before including it. The continuous integration service on GitHub is GitHub actions.

Background

GitHub offers testing on Ubuntu, macOS and Windows operating systems. However, there is a wide variety of other operating systems and you may want to ensure that an open source project developed on GitHub runs well on another operating system, in particular Fedora Linux.

Podman is a command line tool that can run a different Linux operating system in a container. This provides a convenient way to test software on other operating systems. The article Getting Started with Podman in Fedora Linux introduces how to run Podman on Fedora.

This article demonstrates how to run Fedora Linux in a container using Podman. The host operating system can be any distro that has Podman installed, even macOS or Windows. In the following demo, the host operating system is Ubuntu. This will allow us to test that projects developed on GitHub will work successfully on Fedora, even if Fedora is not available as a base operating system for GitHub actions.

Example GitHub Actions Configuration

As an example, we add continuous integration for Fedora Linux to RedAmber, a project enabling the use of dataframes for machine learning and other data science applications in Ruby. This project relies on Apache Arrow release 10 or greater, so we need to use Fedora Linux Rawhide (F38) since Fedora Linux 37 currently has Apache Arrow release 9 in the Fedora repositories.

GitHub has great documentation on using GitHub Actions. In summary, we need to create a yaml file in the .github/workflows directory of the project, and then enable GitHub Actions if it is not already enabled. A sample yaml file which you can easily modify is below:

name: CI
on: push: branches: - main pull_request: jobs: test: name: fedora runs-on: ubuntu-latest steps: - name: Setup Podman run: | sudo apt update sudo apt-get -y install podman podman pull fedora:38 - name: Get source uses: actions/checkout@v3 with: path: 'red_amber' - name: Create container and run tests run: | { echo 'FROM fedora:38' echo '# set TZ to ensure the test using timestamp' echo 'ENV TZ=Asia/Tokyo' echo 'RUN dnf -y update' echo 'RUN dnf -y install gcc-c++ git libarrow-devel libarrow-glib-devel ruby-devel' echo 'RUN dnf clean all' echo 'COPY red_amber red_amber' echo 'WORKDIR /red_amber' echo 'RUN bundle install' echo 'RUN bundle exec rake test' } > podmanfile podman build --tag fedora38test -f ./podmanfile

Adding the above yaml file enables testing on Fedora Linux running as a guest on Ubuntu. Similar workflows should work for other projects developed on GitHub, thereby ensuring a wide variety of software will run well on Fedora Linux.

Acknowledgements

Benson Muite is grateful to Hirokazu SUZUKI for creating RedAmber, improving the workflow, and using it to test RedAmber on Fedora Linux.

[This message comes directly from the desk of Matthew Miller, the Fedora Project Leader. — Ed.]

This “love letter to the community” started in 2020 as a way to shine a little light in a very dark time, and to encourage everyone — including me — by reminding us all of the great work done by great people in Fedora. But it’s become one of my favorite things to do all year. We’re no longer just trying to get through a dark time. We’re looking forward to an exciting era in Fedora’s future.

The work we did this year sets a great foundation for building our future. I don’t just mean the Fedora Linux 36 and 37 releases, although we should definitely be proud of those. But there’s a continued sense of excitement around the community. We’re growing and bringing new energy.

This year, Nest With Fedora grew even more in a time where everyone is feeling virtual event fatigue. And we introduced Hatch — regional events where you could meet with other local-ish contributors. Reading the recaps, I wish I could have gone to all of them. But it was great to spend time with some of you in Rochester. I’ve really, really missed our in-person interactions. Virtual events help keep our global community connected, and help bring in new people who might not be able to join us otherwise, but they can’t substitute for face-to-face meetups. More on that in a moment.

It’s not just a few days of events that has me excited, though. When I look around the project, I see a lot happening. The Fedora CoreOS and Cloud teams promoted their deliverables to Edition status. We wrapped up a huge revamp of our community outreach that began in 2020. The Docs team is more active than it has been in years (and they’ve added a search bar to the site!). We have a complete renovation of our websites in the works. The Marketing team is exploring new ways to promote Fedora, including a presence in the Fediverse. We’re finally almost ready to merge Ask Fedora and Fedora Discussion, bringing more of our conversations together.

That’s a lot of work for one year. The best part is how organic this work is. This wasn’t some demand from on high (that’s not how Fedora works), but it was people in the community saying “I see work to be done. I’m going to do it!” Fedora is us.

We will celebrate so much more in 2023. We’re still working on the details, but we expect to have a greater in-person experience next year, including funding for hackfests and the return of Flock to Fedora! And of course, it’s the 20th anniversary of Fedora. The world — and the technology that drives it — has changed so much since then. But our values haven’t. The Fedora community remains an inclusive, welcoming, and open-minded community. I’m proud to be a part of it. Happy new year, everyone!

This article is part of a series of articles that takes a closer look at Btrfs, the default filesystem for Fedora Workstation and Fedora Silverblue since Fedora Linux 33.

In case you missed it, here’s the previous article from the series: https://fedoramagazine.org/working-with-btrfs-general-concepts/

Introduction

Subvolumes allow for the partitioning of a Btrfs filesystem into separate sub-filesystems. This means that you can mount subvolumes from a Btrfs filesystem as if they were independent filesystems. In addition, you can, for example, define the maximum space a subvolume may take up via qgroups (We’ll talk about this in another article in this series), or use subvolumes to specifically include or exclude files from snapshots (We’ll talk about this, too, in another article in this series). Every default Fedora Workstation and Fedora Silverblue installation since Fedora Linux 33 makes use of subvolumes. In this article we will explore how it works.

Below you will find a lot of examples related to subvolumes. If you want to follow along, you must have access to some Btrfs filesystem and root access. You can verify whether your /home/ directory is Btrfs via the following command:

$ findmnt -no FSTYPE /home
btrfs

This command will output the name of the filesystem of your /home/ directory. If it says btrfs, you’re all set. Let’s create a new directory to perform some experiments in:

$ mkdir ~/btrfs-subvolume-test
$ cd ~/btrfs-subvolume-test

In the text below, you will find lots of command outputs in boxes such as shown above. Please keep in mind while reading/comparing command outputs that the box contents are wrapped at the end of the line. This makes it difficult to recognize long lines that are broken across multiple lines for readability. When in doubt, try to resize your browser window and see how the text behaves!

Creating and playing with subvolumes

We can create a Btrfs subvolume with the following command:

$ sudo btrfs subvolume create first
Create subvolume './first'

When we inspect the current directory we will see that it now has a new folder named first. Note the first character d in the output below:

$ ls -l
total 0
drwxr-xr-x. 1 root root 0 Oct 15 18:09 first

We can handle this like any regular folder: We can rename it, move it, create new files and folders inside, etc. Note that the folder belongs to root, so we must be root to do these things.

If it acts like a folder and looks like a folder, how do we know whether it’s a Btrfs subvolume? We can use the btrfs tools to list all subvolumes:

$ sudo btrfs subvolume list .
ID 256 gen 30 top level 5 path home
ID 257 gen 30 top level 5 path root
ID 258 gen 25 top level 257 path root/var/lib/machines
ID 259 gen 29 top level 256 path hartan/btrfs-subvolume-test/first

If you’re on a recent and unmodified Fedora Linux installation you will likely see the same output as above. We will inspect home and root as well as the meaning of all the numbers later. For now, we see that there is a subvolume at the path we specified. We can limit the output to the subvolumes below our current location:

$ sudo btrfs subvolume list -o .
ID 259 gen 29 top level 256 path home/hartan/btrfs-subvolume-test/first

Let’s rename the subvolume:

$ sudo mv first second
$ sudo btrfs subvolume list -o .
ID 259 gen 29 top level 256 path home/hartan/btrfs-subvolume-test/second

We can also nest subvolumes:

$ sudo btrfs subvolume create second/third
Create subvolume 'second/third'
$ sudo btrfs subvolume list .
ID 256 gen 34 top level 5 path home
ID 257 gen 37 top level 5 path root
ID 258 gen 25 top level 257 path root/var/lib/machines
ID 259 gen 37 top level 256 path hartan/btrfs-subvolume-test/second
ID 260 gen 37 top level 259 path hartan/btrfs-subvolume-test/second/third

And we can also remove subvolumes, either like we remove folders:

$ sudo rm -r second/third

or via special Btrfs commands:

$ sudo btrfs subvolume delete second
Delete subvolume (no-commit): '/home/hartan/btrfs-subvolume-test/second'

Handling Btrfs subvolumes like separate filesystems

The introduction mentioned that Btrfs subvolumes act like separate filesystems. This means that we can mount subvolumes and pass some mount options to them. First we will create a small folder structure to get a better understanding of what happens:

$ mkdir -p a a/1 a/1/b
$ sudo btrfs subvolume create a/2
Create subvolume 'a/2'
$ sudo touch a/1/c a/1/b/d a/2/e

Here’s what the structure looks like:

$ tree
.
└── a ├── 1 │   ├── b │   │   └── d │   └── c └── 2 └── e 4 directories, 3 files

Verify that there is now a new Btrfs subvolume:

$ sudo btrfs subvolume list -o .
ID 261 gen 41 top level 256 path home/hartan/btrfs-subvolume-test/a/2

To mount the subvolume we must know the path of the block device where the Btrfs filesystem subvolume resides. The following command tells us:

$ findmnt -vno SOURCE /home/
/dev/vda3

Now we can mount the subvolume. Make sure you replace the arguments with the values for your PC:

$ sudo mount -o subvol=home/hartan/btrfs-subvolume-test/a/2 /dev/vda3 a/1/b

Observe that we use the -o flag to give additional options to the mount program. In this case we tell it to mount the subvolume with name home/hartan/btrfs-subvolume-test/a/2 from the btrfs filesystem on device /dev/vda3. This is a Btrfs-specific option and isn’t available in other filesystems.

We see that the directory structure has changed:

$ tree
.
└── a ├── 1 │   ├── b │   │   └── e │   └── c └── 2 └── e 4 directories, 3 files

Note that the file e exists twice now and d is gone. We are now able to access the same Btrfs subvolume by two different paths. All changes we perform in either of the paths are immediately reflected in all other locations:

$ sudo touch a/1/b/x
$ ls -lA a/2
total 0
-rw-r--r--. 1 root root 0 Oct 15 18:14 e
-rw-r--r--. 1 root root 0 Oct 15 18:16 x

Let’s play some more with the mount options. For example we can mount the subvolume as read-only under a/1/b like this (Insert arguments for your PC!):

$ sudo umount a/1/b
$ sudo mount -o subvol=home/hartan/btrfs-subvolume-test/a/2,ro /dev/vda3 a/1/b

We use the same command as above, except that we add ro at the end. Now we can no longer create files via this mount:

$ sudo touch a/1/b/y
touch: cannot touch 'a/1/b/y': Read-only file system

but accessing the subvolume directly still works like before:

$ sudo touch a/2/y
$ tree
.
└── a ├── 1 │   ├── b │   │   ├── e │   │   ├── x │   │   └── y │   └── c └── 2 ├── e ├── x └── y 4 directories, 7 files

Don’t forget to clean up before we move on:

$ sudo rm -rf a
rm: cannot remove 'a/1/b/e': Read-only file system
rm: cannot remove 'a/1/b/x': Read-only file system
rm: cannot remove 'a/1/b/y': Read-only file system

Oh no, what happened? Well, since we mounted the subvolume read-only above, we cannot delete it. A deletion from a filesystems’ perspective is a write operation: To delete a/1/b/e, we remove the directory entry for e from the directory contents of its parent directory, a/1/b in this case. In other words, we must write to a/1/b to tell it that e doesn’t exist any longer. So first we unmount the subvolume again, and then we remove the folder:

$ sudo umount a/1/b
$ sudo rm -rf a
$ tree
. 0 directories, 0 files

Subvolume IDs

Remember the first output of the subvolume list subcommand? That contained a lot of numbers, so let’s see what that is all about. I copied the output here to take another look:

ID 256 gen 30 top level 5 path home
ID 257 gen 30 top level 5 path root
ID 258 gen 25 top level 257 path root/var/lib/machines
ID 259 gen 29 top level 256 path hartan/btrfs-subvolume-test/first

We see there are three columns of numbers, each prefixed with a few letters to describe what they do. The first column of numbers is a subvolumes ID. Subvolume IDs are unique in a Btrfs filesystem and as such uniquely identify subvolumes. This means that the subvolume named home can also be referred to by its ID 256. In the mount command above we wrote:

$ sudo mount -o subvol=hartan/...

Another perfectly legal option is to use subvolume IDs:

$ sudo mount -o subvolid=...

Subvolume IDs start at 256 and increase by 1 for every created subvolume. There is however one exception to this: The filesystem root always has the subvolume name / and the subvolume ID 5. That is right, even the root of a Btrfs filesystem is technically a subvolume. This is just implicitly known, hence it doesn’t show up in the output of btrfs subvolume list. If you mount a Btrfs filesystem without the subvol or subvolid argument, the root subvolume with subvolid=5 is assumed as default. Below we’ll see an example of when one may want to explicitly mount the filesystem root.

The second column of numbers is the generation counter and incremented on every Btrfs transaction. This is mostly an internal counter and won’t be discussed further here.

Finally, the third column of numbers is the subvolume ID of the subvolumes parent. In the output above we see that both subvolume home and root have 5 as their parent subvolume ID. Remember that ID 5 has a special meaning: It is the filesystem root. So we know that home and root are children to the root subvolume. hartan/btrfs-subvolume-test/first on the other hand is a child of the subvolume with ID 256, which in our case is home.

In the next section we have a look at where the subvolumes root and home come from.

Inspecting default subvolumes in Fedora Linux

When you create a new Btrfs filesystem from scratch, there will be no subvolumes in it (Except of course for the root subvolume). So where do the home and root subvolumes in Fedora Linux come from?

These are created by the installer at install time. Traditional installations would often include a separate filesystem partition for the / and /home directories. During boot, these are then appropriately mounted to assemble one full filesystem. But there is an issue with this approach: Unless you use technologies such as lvm, it is very hard to change a partitions size at some point in the future. As a consequence you may end up in a situation where either your / or /home runs out of space, while the respective other partition has lots of unused, free space left.

Since Btrfs subvolumes are all part of the same filesystem, they will share the space that the underlying filesystem offers. Remember when we created the subvolumes above? We never told Btrfs how big they are: A subvolume can take up all the space the filesystem has, by default nothing keeps it from doing so. However, we could dynamically impose size limits via Btrfs qgroups, which can also be modified during runtime (And we’ll see how in a later article in this series).

Another advantage of separating / and /home is that we can take snapshots separately. A subvolume is a boundary for snapshots, and snapshots will never contain the contents of other subvolumes below the subvolume that the snapshot is taken of. More details on snapshots follow in the next article in this series.

Enough of the theory! Let’s see what this is all about. First ensure that your root filesystem is in fact of type Btrfs:

$ findmnt -no FSTYPE /
btrfs

And then get the partition it resides on:

$ findmnt -vno SOURCE /
/dev/vda3

Remember we can mount the filesystem root by its special subvolume ID 5 (Adapt the filesystem partition!):

$ mkdir fedora-rootsubvol
$ sudo mount -o subvolid=5 /dev/vda3 ./fedora-rootsubvol
$ ls fedora-rootsubvol/
home root

And there are the subvolumes of our Fedora Linux installation! But how does Fedora Linux know that the subvolume root belongs to /, and home belongs to /home?

The file /etc/fstab contains so-called static information about the filesystem. In simple terms, during booting your system reads this file, line by line, and mounts all the filesystems listed there. On my system, the file looks like this:

$ cat /etc/fstab
# [ ... ]
# /etc/fstab
# Created by anaconda on Sat Oct 15 12:01:57 2022
# [ ... ]
#
UUID=5e4e42bb-4f2f-4f0e-895f-d1a46ea47807 / btrfs subvol=root,compress=zstd:1 0 0
UUID=e3a798a8-b8f2-40ca-9da7-5e292a6412aa /boot ext4 defaults 1 2
UUID=5e4e42bb-4f2f-4f0e-895f-d1a46ea47807 /home btrfs subvol=home,compress=zstd:1 0 0

(Note that the “UUID” lines above have been wrapped into two lines)

The UUID at the beginning of each line is simply a means to identify disks and filesystem partitions in your system (roughly equivalent to /dev/vda3 as I used above). The second column is the path in the filesystem tree where this filesystem should be mounted. The third column is the filesystem type. We see that the entries for / and /home are of type btrfs, just what we expect! Finally, in the fourth column we see the magic: These are the mount options, and there it says to mount / with the option subvol=root. That is exactly the subvolume we saw in the output of btrfs subvolume list / all the time!

With this information, we can reconstruct the call to mount that creates this filesystem entry:

$ sudo mount -o subvol=root,compress=zstd:1 UUID=5e4e42bb-4f2f-4f0e-895f-d1a46ea47807 /

(again, the line above has been wrapped into two)

And that is how Fedora Linux uses Btrfs subvolumes! If you’re curious as to why Fedora Linux decided to use Btrfs as the default filesystem, refer to the change proposal linked below [1].

More on Btrfs subvolumes

The Btrfs wiki has additional information on subvolumes and most importantly on the mount options that can be applied to Btrfs subvolumes. Some options, like compress can only be applied on a filesystem-wide level and thus affect all subvolumes of a Btrfs filesystem. You can find the entry linked below [2].

If you find it confusing to tell which directories are plain directories and which are subvolumes, you can feel free to adopt a special naming convention for your subvolumes. For example, you could prefix your subvolume names with an “@” to make them easily distinguishable.

Now that you know that subvolumes behave like filesystems, one may ask how best to place a subvolume in a certain location. Say you want a Btrfs subvolume under ~/games, where your home directory (~) is itself a subvolume, how can you achieve that? Given the example above, you may use a command like sudo btrfs subvolume create ~/games. This way, you create so-called nested subvolumes: Inside your subvolume ~, there is now a subvolume games. That is a perfectly fine way to approach this situation.

Another valid solution is to do what Fedora does by default: Create all subvolumes under the root subvolume (i.e. such that their parent subvolume ID is 5), and mount them into the appropriate locations. The Btrfs wiki has an overview of these approaches along with a short discussion about their respective implications on filesystem management [5].

Conclusion

In this article we discovered Btrfs subvolumes, which act like separate Btrfs filesystems inside a Btrfs filesystem. We learned how to create, mount and delete subvolumes. Finally, we explored how Fedora Linux makes use of subvolumes – without us noticing at all.

The next articles in this series will deal with:

• Snapshots – Going back in time
• Compression – Transparently saving storage space
• Qgroups – Limiting your filesystem size
• RAID – Replace your mdadm configuration

If there are other topics related to Btrfs that you want to know more about, have a look at the Btrfs Wiki [3] and Docs [4]. Don’t forget to check out the first article of this series, if you haven’t already! If you feel that there is something missing from this article series, let us know in the comments below. See you in the next article!

Sources

[1]: https://fedoraproject.org/wiki/Changes/BtrfsByDefault#Benefit_to_Fedora
[2]: https://btrfs.readthedocs.io/en/latest/Subvolumes.html
[3]: https://btrfs.wiki.kernel.org/index.php/Main_Page
[4]: https://btrfs.readthedocs.io/en/latest/Introduction.html
[5]: https://btrfs.wiki.kernel.org/index.php/SysadminGuide#Layout