This article will explore transparent filesystem compression in Btrfs and how it can help with saving storage space. This is part of a series that takes a closer look at Btrfs, the default filesystem for Fedora Workstation, and Fedora Silverblue since Fedora Linux 33.<\/p>\n
In case you missed it, here\u2019s the previous article from this series: https:\/\/fedoramagazine.org\/working-with-btrfs-snapshots<\/a><\/p>\n <\/span> <\/p>\n Most of us have probably experienced running out of storage space already. Maybe you want to download a large file from the internet, or you need to quickly copy over some pictures from your phone, and the operation suddenly fails. While storage space is steadily becoming cheaper, an increasing number of devices are either manufactured with a fixed amount of storage or are difficult to extend by end-users.<\/p>\n But what can you do when storage space is scarce? Maybe you will resort to cloud storage, or you find some means of external storage to carry around with you.<\/p>\n In this article I’ll investigate another solution to this problem: transparent filesystem compression, a feature built into Btrfs. Ideally, this will solve your storage problems while requiring hardly any modification to your system at all! Let’s see how.<\/p>\n First, let’s investigate what transparent<\/em> compression means. You can compress files with compression algorithms such as gzip, xz, or bzip2. This is usually an explicit operation: You take a compression utility and let it operate on your file. While this provides space savings, depending on the file content, it has a major drawback: When you want to access the file to read or modify it, you have to decompress it first.<\/p>\n This is not only a tedious process, but also temporarily defeats the space savings you had achieved previously. Moreover, you end up (de)compressing parts of the file that you didn’t intend to touch in the first place. Clearly there is something better than that!<\/p>\n Transparent compression on the other hand takes place at the filesystem level. Here, compressed files still look like regular uncompressed files to the user. However, they are stored with compression applied on disk. This works because the filesystem selectively decompresses only the parts of a file that you access and makes sure to compress them again as it writes changes to disk.<\/p>\n The compression here is transparent in that it isn’t noticeable to the user, except possibly for a small increase in CPU load during file access. Hence, you can apply this to existing systems without performing hardware modifications or resorting to cloud storage.<\/p>\n Btrfs offers multiple compression algorithms to choose from. For technical reasons it cannot use arbitrary compression programs. It currently supports:<\/p>\n The good news is that, due to how transparent compression works, you don’t have to install these programs for Btrfs to use them. In the following paragraphs, you will see how to run a simple benchmark to compare the individual compression algorithms. In order to perform the benchmark, however, you must install the necessary executables. There’s no need to keep them installed afterwards, so you’ll use a podman container to make sure you don’t leave any traces in your system.<\/p>\n Because typing the same commands over and over is a tedious task, I have prepared a ready-to-run bash script that is hosted on Gitlab (https:\/\/gitlab.com\/hartang\/btrfs-compression-test<\/a>). This will run a single compression and decompression with each of the above-mentioned algorithms at varying compression levels.<\/p>\n First, download the script:<\/p>\n Next, spin up a Fedora Linux container that mounts your current working directory so you can exchange files with the host and run the script in there:<\/p>\n Finally run the script with:<\/p>\n The output on my machine looks like this:<\/p>\n It is important to note a few things before making decisions based on the numbers from the script:<\/p>\n Given the numbers in my output, I decided to use the zstd compression algorithm with compression level 3 on my systems. Depending on your needs, you may want to choose higher compression levels (for example, if your storage devices are comparatively slow). To get an estimate of the achievable read\/write speeds, you can divide the source archives size (about 260 MB) by the (de)compression times.<\/p>\n The compression test works on the GNU libc 2.36 source code by default. If you want to see the results for a custom file, you can give the script a file path as the first argument. Keep in mind that the file must be accessible from inside the container.<\/p>\n Feel free to read the script code and modify it to your liking if you want to test a few other things or perform a more detailed benchmark!<\/p>\n Transparent filesystem compression in Btrfs is configurable in a number of ways:<\/p>\n I’ll only take a look at the first two of these.<\/p>\n There is a Btrfs mount option that enables file compression:<\/p>\n For example, to mount a filesystem and compress it with the zstd<\/em> algorithm on level 3, you would write:<\/p>\n Setting the compression level is optional. It is important to note that the compress<\/em> mount option applies to the whole Btrfs filesystem and all of its subvolumes. Additionally, it is the only currently supported way of specifying the compression level to use.<\/p>\n In order to apply compression to the root filesystem, it must be specified in \/etc\/fstab<\/em>. The Fedora Linux Installer, for example, enables zstd<\/em> compression on level 1 by default, which looks like this in \/etc\/fstab<\/em>:<\/p>\n Another way of specifying compressions is via Btrfs filesystem properties. To read the compression setting for any file, folder or subvolume, use the following command:<\/p>\n Likewise, you can configure compression like this:<\/p>\n For example, to enable zlib<\/em> compression for all files under \/etc<\/em>:<\/p>\n You can get a list of supported values with man btrfs-property<\/em>. Keep in mind that this interface doesn’t allow specifying the compression level. In addition, if a compression property is set, it overrides other compression configured at mount time.<\/p>\n At this point, if you apply compression to your existing filesystem and check the space usage with df<\/em> or similar commands, you will notice that nothing has changed. That is because Btrfs, by itself, doesn’t “recompress” all your existing files. Compression will only take place when writing new data to disk. There are a few ways to perform an explicit recompression:<\/p>\n The last option is probably the most convenient, but it comes with a caveat on Btrfs filesystems that already contain snapshots: it will break shared extent between snapshots. In other words, all the shared content between two snapshots, or a snapshot and its’ parent subvolume, will be present multiple times after a defrag operation.<\/p>\n Hence, if you already have a lot of snapshots on your filesystem, you shouldn’t run a defragmentation on the whole filesystem. This isn’t necessary either, since with Btrfs you can defragment specific directories or even single files, if you wish to do so.<\/p>\n You can use the following command to perform a defragmentation:<\/p>\n For example, you can defragment your home directory like this:<\/p>\n In case of doubt it’s a good idea to start with defragmenting individual large files and continuing with increasingly large directories while monitoring free space on the file system.<\/p>\n At some point, you may wonder just how much<\/em> space you have saved thanks to file system compression. But how do you tell? First, to tell if a Btrfs filesystem is mounted with compression applied, you can use the following command:<\/p>\n If you get a result, the filesystem at the given mount point is using compression! Next, the command compsize<\/em> can tell you how much space your files need:<\/p>\n On my home directory, the result looks like this:<\/p>\n The individual lines tell you the “Type” of compression applied to files. The “TOTAL” is the sum of all the lines below it. The columns, on the other hand, tell you how much space our files need:<\/p>\n “Referenced” can differ from the numbers in “Uncompressed” if, for example, one has deduplicated files previously, or if there are snapshots that share extents. In the example above, you can see that 91 GB worth of uncompressed files occupy only 74 GB of storage on my disk! Depending on the type of files stored in a directory and the compression level applied, these numbers can vary significantly.<\/p>\n Btrfs uses a heuristic algorithm to detect compressed files. This is done because compressed files usually do not compress well, so there is no point in wasting CPU cycles in attempting further compression. To this end, Btrfs measures the compression ratio when compressing data before writing it to disk. If the first portions of a file compress poorly, the file is marked as incompressible and no further compression takes place.<\/p>\n If, for some reason, you want Btrfs to compress all data it writes, you can mount a Btrfs filesystem with the compress-force<\/em> option, like this:<\/p>\n When configured like this, Btrfs will compress all data it writes to disk with the zstd<\/em> algorithm at compression level 3.<\/p>\n An important thing to note is that a Btrfs filesystem with a lot of data and compression enabled may take a few seconds longer to mount than without compression applied. This has technical reasons and is normal behavior which doesn’t influence filesystem operation.<\/p>\n This article detailed transparent filesystem compression in Btrfs. It is a built-in, comparatively cheap, way to get some extra storage space out of existing hardware without needing modifications.<\/p>\n The next articles in this series will deal with:<\/p>\n If there are other topics related to Btrfs that you want to know more about, have a look at the Btrfs Wiki [1]<\/a> and Docs [2]<\/a>. Don\u2019t forget to check out the first three articles of this series, if you haven\u2019t already! If you feel that there is something missing from this article series, let me know in the comments below. See you in the next article!<\/p>\n [1]: https:\/\/btrfs.wiki.kernel.org\/index.php\/Main_Page<\/a> This article will explore transparent filesystem compression in Btrfs and how it can help with saving storage space. This is part of a series that takes a closer look at Btrfs, the default filesystem for Fedora Workstation, and Fedora Silverblue since Fedora Linux 33. In case you missed it, here\u2019s the previous article from this […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[48],"tags":[1157,45,61,46,47],"class_list":["post-131914","post","type-post","status-publish","format-standard","hentry","category-fedora-os","tag-btrfs","tag-fedora","tag-fedora-project-community","tag-magazine","tag-news"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/131914","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=131914"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/131914\/revisions"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=131914"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=131914"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=131914"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Introduction<\/h2>\n
Transparent compression explained<\/h2>\n
Comparing compression algorithms<\/h2>\n
\n
$ curl -LO https:\/\/gitlab.com\/hartang\/btrfs-compression-test\/-\/raw\/main\/btrfs_compression_test.sh<\/pre>\n
$ podman run --rm -it --security-opt label=disable -v \"$PWD:$PWD\" \\ -w \"$PWD\" registry.fedoraproject.org\/fedora:37<\/pre>\n
$ chmod +x .\/btrfs_compression_test.sh\n$ .\/btrfs_compression_test.sh<\/pre>\n
[INFO] Using file 'glibc-2.36.tar' as compression target\n[INFO] Target file 'glibc-2.36.tar' not found, downloading now...\n################################################################### 100.0%\n[ OK ] Download successful!\n[INFO] Copying 'glibc-2.36.tar' to '\/tmp\/tmp.vNBWYg1Vol\/' for benchmark...\n[INFO] Installing required utilities\n[INFO] Testing compression for 'zlib' Level | Time (compress) | Compression Ratio | Time (decompress)\n-------+-----------------+-------------------+------------------- 1 | 0.322 s | 18.324 % | 0.659 s 2 | 0.342 s | 17.738 % | 0.635 s 3 | 0.473 s | 17.181 % | 0.647 s 4 | 0.505 s | 16.101 % | 0.607 s 5 | 0.640 s | 15.270 % | 0.590 s 6 | 0.958 s | 14.858 % | 0.577 s 7 | 1.198 s | 14.716 % | 0.561 s 8 | 2.577 s | 14.619 % | 0.571 s 9 | 3.114 s | 14.605 % | 0.570 s [INFO] Testing compression for 'zstd' Level | Time (compress) | Compression Ratio | Time (decompress)\n-------+-----------------+-------------------+------------------- 1 | 0.492 s | 14.831 % | 0.313 s 2 | 0.607 s | 14.008 % | 0.341 s 3 | 0.709 s | 13.195 % | 0.318 s 4 | 0.683 s | 13.108 % | 0.306 s 5 | 1.300 s | 11.825 % | 0.292 s 6 | 1.824 s | 11.298 % | 0.286 s 7 | 2.215 s | 11.052 % | 0.284 s 8 | 2.834 s | 10.619 % | 0.294 s 9 | 3.079 s | 10.408 % | 0.272 s 10 | 4.355 s | 10.254 % | 0.282 s 11 | 6.161 s | 10.167 % | 0.283 s 12 | 6.670 s | 10.165 % | 0.304 s 13 | 12.471 s | 10.183 % | 0.279 s 14 | 15.619 s | 10.075 % | 0.267 s 15 | 21.387 s | 9.989 % | 0.270 s [INFO] Testing compression for 'lzo' Level | Time (compress) | Compression Ratio | Time (decompress)\n-------+-----------------+-------------------+------------------- 1 | 0.447 s | 25.677 % | 0.438 s 2 | 0.448 s | 25.582 % | 0.438 s 3 | 0.444 s | 25.582 % | 0.441 s 4 | 0.444 s | 25.582 % | 0.444 s 5 | 0.445 s | 25.582 % | 0.453 s 6 | 0.438 s | 25.582 % | 0.444 s 7 | 8.990 s | 18.666 % | 0.410 s 8 | 34.233 s | 18.463 % | 0.405 s 9 | 41.328 s | 18.450 % | 0.426 s [INFO] Cleaning up...\n[ OK ] Benchmark complete!<\/pre>\n
\n
Configuring compression in Btrfs<\/h2>\n
\n
Enabling compression at mount-time<\/h3>\n
$ sudo mount -o compress=<ALGORITHM>:<LEVEL> ...<\/pre>\n
$ sudo mount -o compress=zstd:3 ...<\/pre>\n
$ cat \/etc\/fstab\n[ ... ]\nUUID=47b03671-39f1-43a7-b0a7-db733bfb47ff \/ btrfs subvol=root,compress=zstd:1,[ ... ] 0 0<\/pre>\n
Enabling compression per-file<\/h3>\n
$ btrfs property get <PATH> compression<\/pre>\n
$ sudo btrfs property set <PATH> compression <VALUE><\/pre>\n
$ sudo btrfs property set \/etc compression zlib<\/pre>\n
Compressing existing files<\/h2>\n
\n
$ sudo btrfs filesystem defragment -r \/path\/to\/defragment<\/pre>\n
$ sudo btrfs filesystem defragment -r \"$HOME\"<\/pre>\n
Measuring filesystem compression<\/h2>\n
$ findmnt -vno OPTIONS \/path\/to\/mountpoint | grep compress<\/pre>\n
$ sudo compsize -x \/path\/to\/examine<\/pre>\n
$ sudo compsize -x \"$HOME\"\nProcessed 942853 files, 550658 regular extents (799985 refs), 462779 inline.\nType Perc Disk Usage Uncompressed Referenced\nTOTAL 81% 74G 91G 111G\nnone 100% 67G 67G 77G\nzstd 28% 6.6G 23G 33G<\/pre>\n
\n
Additional notes about file compression<\/h2>\n
$ sudo mount -o compress-force=zstd:3 ...<\/pre>\n
Conclusion<\/h2>\n
\n
Sources<\/h2>\n
[2]: https:\/\/btrfs.readthedocs.io\/en\/latest\/Introduction.html<\/a><\/p><\/p>\n","protected":false},"excerpt":{"rendered":"