Tag: Fedora Project community

Fedora Linux Workstation features a GNOME desktop environment which is an easy to use, intuitive, and efficient desktop environment. But this is not the only option if you would like to use Fedora Linux. There are other spins that provide alternative desktop environments like KDE, XFCE, Cinnamon, etc. This article describes how you can try different desktop environments if you are using an OSTree based Fedora Linux spins.

Main version of Fedora Workstation

If you installed a non-OSTree Fedora Workstation or one of the Spins and would like to try different desktop environment, you have some possibilities:

• install different desktop environment using dnf
• dual boot multiple spins of Fedora

If you choose the first option, you have to install another desktop environment by using the dnf install command. This technique enables you to select which desktop environment you would like to use on the login screen after system boots. Using this method will pull in a lot of dependencies. This is especially true when you have a GTK based desktop environment (like GNOME) and install a QT based (like KDE), or vice-versa. It can be difficult to completely uninstall one of the installed desktop environment if you are not satisfied with them.

Another issue with this is that system apps may be doubled in the application menu on each environment. For example, if you have installed GNOME and install KDE, you have Nautilus and Dolphin for browsing files, GNOME Terminal and Konsole for terminal emulation, etc. You have to remember which app to use on which environment, because apps from KDE behave worse on GNOME and vice-versa.

If you choose the second option, you have to make some free unpartitioned space on your hard drive to install another Fedora spin alongside one you are currently using. In this way, the systems are separated from each other and the system apps will not be doubled. You can decide to share the /home partition between them. This technique enables you select the system to use in the bootloader menu before system boots. But if you use this method, you have to maintain these systems separately (for example installing updates) and it consumes a lot of free space on the hard drive.

OSTree based version of Fedora Workstation

Some variants of Fedora Linux are OSTree based. OSTree provides immutability and transactional upgrades with the possibility of rollback in case something goes wrong. You can read more about it in this great article. Right now, we have three OSTree based Fedora Workstation variants:

• Silverblue – provides GNOME desktop environment
• Kinoite – provides KDE Plasma desktop environment
• Sericea – provides Sway window manager (not recommended for beginners)

If you are running one of these variants of Fedora Linux, you can easily switch your system to another OSTree compatible one to try different desktop environment. This process is similar to doing a system upgrade. OSTree guarantees that the operation is transactional (finishes successfully or nothing is changed) and you are able to rollback if you are not satisfied with the change. The operation does not consume much space on the hard drive, and system apps are not doubled.

How to use OSTree rebase to switch to a new variant

To start, I recommend executing the following command to pin the current deployment. This makes certain it will not be deleted automatically in the future and provides the ability to roll back to it.

$ sudo ostree admin pin 0

If you have a pending update, the command may fail with the message:

error: Cannot pin staged deployment

In this case, reboot your system to apply pending updates, and try again.

After pinning the deployment, execute:

$ ostree remote refs fedora

It outputs a list of all available branches that you can rebase into. Every branch has an architecture, version, and the name of the variant. Select carefully. In the following examples I assume you would like to rebase into the current stable version of Fedora for x86_64 (version 38).

• for Fedora Silverblue, use fedora:fedora/38/x86_64/silverblue
• for Fedora Kinoite, use fedora:fedora/38/x86_64/kinoite
• for Fedora Sericea, use fedora:fedora/38/x86_64/sericea

Choose the branch you wish to rebase into and execute the following command (change the branch name provided in the example if necessary):

$ rpm-ostree rebase fedora:fedora/38/x86_64/kinoite

When this command succeeds, restart the system to begin using the new desktop environment. If it fails, the system should continue to work unmodified thanks to transactional updates provided by OSTree.

Undo the rebase into an OSTree variant

If you are not satisfied with the new environment the following command will return you to your original variant:

rpm-ostree rollback

Restart your system once again to switch back to the previous variant of Fedora.

Red Hat is excited to announce significant changes to ELN kernel RPM NVR in the kernel-ark project.  This change will be to limited to the kernel-ark ELN RPMs and does not impact Fedora.  If you don’t use Fedora ELN builds you can likely stop reading as this change won’t affect you. 

What is the kernel-ark project?

The kernel-ark project is an upstream kernel-based repository from which the Fedora kernel RPMs are built (contributions welcomed!).  This project is also used by the Centos Stream and Red Hat Enterprise Linux (RHEL) maintainers to implement, test, and verify code that is destined to be used in Centos Stream and RHEL. In other words, the kernel-ark repository contains code that is enabled to build several different kernels which may contain unique code for different use cases.  The kernel RPMs used for CentOS Stream and RHEL are commonly referred to as the ‘ELN’ (Enterprise Linux Next) RPMs.

Why are there ELN RPMs?  Why can’t Centos Stream and Red Hat use Fedora RPMs?

While Fedora Linux is the source of a lot of code that lands in CentOS Stream and later RHEL, the kernel configuration used in each operating system is unique.  Fedora Linux is configured to achieve its specific goals and targets.  CentOS Stream and RHEL do the same but for a slightly different set of goals and targets.

The differences are significant enough that the Fedora Linux, Centos Stream, and RHEL kernel maintainers recognized the need for separate RPMs targeted for Fedora and those targeted for Centos Stream and RHEL.  Examples of these differences are BTRFS is enabled in Fedora Linux but not in ELN and there are some specific devices that are disabled in ELN but are enabled in Fedora Linux.

Red Hat uses kernel-ark’s ELN RPMs to continuously test upstream changes with a Red Hat specific configuration.  This enables Red Hat to monitor performance and resolve issues before they make it into a Red Hat Enterprise Linux (RHEL) release.  In accordance with Red Hat’s long established ‘upstream first’ policy, all fixes and suggestions for improvements are sent back to the upstream kernel community.  This benefits the entire Linux community with improvements due to issues resolved in ELN.

This structure also allows Red Hat to test and make changes without affecting the Fedora Linux kernel except in a positive or desired way, such as through bug fixes.  Fedora Linux, CentOS Stream, and RHEL have the opportunity to accept or reject changes easily.

What ELN NVR changes are being made?

Before explaining the changes to the ELN kernel rpm NVR, it is important to understand what an NVR is.  All RPMs have Name, Version, and Release (NVR) directives that describe the RPM package.  In the case of the Fedora Linux kernel RPMs the Name is ‘kernel’ and the Version is the kernel’s uname (aka UTSNAME or the kernel version).  The last field, the Release, contains additional upstream tagging information (release candidate tags), a monotonically increasing build number, and a distribution tag.  The NVR is separate from the kernel’s uname and the uname is not generated from it.  Instead, we have traditionally generated the NVR from the uname.

For example, for a recent Fedora Linux kernel build,

$ rpm -qi kernel-6.3.0-0.rc5.42.fc39.src.rpm

Name: kernel
Version: 6.3.0
Release: 0.rc5.42.fc39

In the next few weeks, the Centos Stream and RHEL maintainers will introduce ELN RPMs that have new kernel Name, Version, and Release (NVR) directives that are unique to the ELN builds.  This change has no impact on the kernel uname.  The net result of the change is that the version number will have more meaning to CentOS Stream and RHEL builds, instead of being solely based on the kernel uname.  For example, an ELN kernel may have NVR kernel-redhat-1.0.39.eln which packages a kernel with a kernel uname of 6.3.0-39.eln.

We have already decided that the new ‘Name’ directive for the ELN kernel NVR will be changed from ‘kernel’ to ‘kernel-redhat’. More information on the Version and Release directive changes will be released in the following months as they are finalized by the Centos Stream and RHEL kernel maintainers.  You can follow these discussions on the Fedora Kernel Mailing List.

Why is the ELN NVR being changed?

The new ELN NVR will allow for better coordination of feature introduction, bug fixes, and CVE resolutions in future versions of Centos Stream and RHEL.  More information on these improvements to the Centos Stream and RHEL ecosystems will be released in the upcoming months. 

How is Fedora Linux impacted?

Fedora Linux is not impacted by these changes.  

Since the inception of the kernel-ark project, the Fedora Linux, Centos Stream, and RHEL maintainers have been extraordinarily careful to ensure that Fedora Linux kernel builds are not impacted by ELN kernel builds (and vice-versa) in the kernel-ark project.  The commitment to prevent cross-OS issues is strictly enforced by the maintainers.  Due to the maintainers continued diligence, there is no impact to Fedora Linux.

How can I obtain Fedora Linux or ELN kernels?

Fedora Linux and ELN kernels can be downloaded from Fedora Project’s koji instance.

bcache is a simple and good way to have large disks (typically rotary and slow) exhibit performance quite similar to an SSD disk, using a small SSD disk or a small part of an SDD.

In general, bcache is a system for having devices composed of slow and large disks, with fast and small disks attached as a cache.

This article will discuss performance and some optimization tips as well as configuration of bcache.

The following terms are used in bcache to describe how it works and the parts of bcache:

backing device	slow and large disk (disk intended to actually hold the data)
cache device	fast and small disk (cache)
dirty cache	data present only in the cache device
writeback	writing to the cache device and later (much later) to the backing device
writeback rate	cache write speed in the backing device

A disk data cache has always existed, it is the free RAM in the operating system. When data is read from the disk it is copied to RAM. If the data is already in RAM, it is read from RAM rather than being read from disk again. When data is written to the disk, it is written to RAM and after a few moments written to the disk as well. The time data spends only in RAM is very short since RAM is volatile.

bcache is similar, only it has various modes of cache operation. The mode that is faster in writing data is writeback. It works the same as for RAM, only instead of RAM there is a SATA or NVME SSD device. The data may reside only in the cache for much longer, even forever, so it is a bit riskier (if you break the SSD, the data that resided only in the cache is lost, with a good chance that the whole filesystem becomes inaccessible).

Performance Comparison

It is very difficult to gather reliable data from any tests, either with real cases or with special programs. They always give extremely variable, different, unstable values. The various caches present and the type of filesystem (btrfs, journaled, etc.), make the values very variable. It is advisable to ignore small differences (say 5-10%).

The following performance data refers to the test below (random and multiple reads/writes), trying to always maintain the same conditions and repeating three times in immediate sequence.

$ sysbench fileio --file-total-size=2G --file-test-mode=rndrw --time=30 --max-requests=0 run

The tables below show the performance of the separate devices:

Performance of the backing device (RAID 1 with 1TB rotary disks)

Throughput:
read, MiB/s: 0.22	read, MiB/s: 0.23	read, MiB/s: 0.19
written, MiB/s: 0.15	written, MiB/s: 0.16	written, MiB/s 0.13
Latency (ms):
max: 174.92	max: 879.59	max: 1335.30
95th percentile: 87.56	95th percentile: 87.56	95th percentile: 89.16

Performance of the cache device (SSD SATA 100GB)

Throughput:
read, MiB/s: 7.28	read, MiB/s: 7.21	read, MiB/s: 7.51
written, MiB/s: 4.86	written, MiB/s: 4.81	written, MiB/s 5.01
Latency (ms):
max: 126.55	max: 102.39	max: 107.95
95th percentile: 1.47	95th percentile: 1.47	95th percentile: 1.47

The theoretical expectation that a bcache device will be as fast as the cache device is (physically) impossible to achieve. On average, bcache is significantly slower and only sometimes approaches the same performance as the cache device. Improved performance almost always requires various compromises.

Consider an example assuming there is a 1TB bcache device and a 100GB cache. When writing a 1TB file, the cache device is filled, then partially emptied to the backing device, and refilled again, until the file is fully written.

Because of this (and also because part of the cache also serves data when reading) there is a limit on the length of the file’s sequential data that are written to the cache. Once the limit is exceeded, the file data is written (or read) directly to the backing device, bypassing the cache.

bcache also limits the response delay of the disks, but disproportionately so, especially for SSD SATA, degrading the performance of the cache.

The dirty cache should be emptied to decrease the risk of data loss and to have cache available when it is needed. This should only be done when the devices exhibit little or no activity, otherwise the performance available for normal use collapses.

Unfortunately, the default settings are too low, and the writeback rate adjustment is crude. To improve the writeback rate adjustment it is necessary to write a program (I wrote a script for this).

The following commands provide the necessary optimizations (required at each startup) to get better performance from the bcache device.

# echo 0 > /sys/block/bcache0/bcache/cache/congested_write_threshold_us
# echo 0 > /sys/block/bcache0/bcache/cache/congested_read_threshold_us
# echo 600000000 > /sys/block/bcache0/bcache/sequential_cutoff
# echo 40 > /sys/block/bcache0/bcache/writeback_percent

The following tables compare the performance with the default values and the optimization results.

Performance with default values

Throughput:
read, MiB/s: 3.37	read, MiB/s: 2.67	read, MiB/s: 2.61
written, MiB/s: 2.24	written, MiB/s: 1.78	written, MiB/s 1.74
Latency (ms):
max: 128.51	max: 102.61	max: 142.04
95th percentile: 9.22	95th percentile: 10.84	95th percentile: 11.04

Performance with optimizations

Throughput:
read, MiB/s: 5.96	read, MiB/s: 3.89	read, MiB/s: 3.81
written, MiB/s: 3.98	written, MiB/s: 2.59	written, MiB/s 2.54
Latency (ms):
max: 131.95	max: 133.23	max: 117.76
95th percentile: 2.61	95th percentile: 2.66	95th percentile: 2.66

Performance with the writeback rate adjustment script

Throughput:		
read, MiB/s: 6.25	read, MiB/s: 4.29	read, MiB/s: 5.12
written, MiB/s: 4.17	written, MiB/s: 2.86	written, MiB/s 3.41
Latency (ms):		
max: 130.92	max: 115.96	max: 122.69
95th percentile: 2.61	95th percentile: 2.66	95th percentile: 2.61

In single operations (without anything else happening in the system) on large files, adjusting the writeback rate becomes irrelevant.

Prepare the backing, cache and bcache device

To create a bcache device you need to install the bcache-tools. The command for this is:

# dnf install bcache-tools

bcache devices are visible as /dev/bcacheN (for example /dev/bcache0 ). Once created, they are managed like any other disk.

More details are available at https://docs.kernel.org/admin-guide/bcache.html

CAUTION: Any operation performed can immediately destroy the data on the partitions and disks on which you are operating. Backup is advised.

In the following example /dev/md0 is the backing device and /dev/sda7 is the cache device.

WARNING: bcache device cannot be resized.
NOTE: bcache refuses to use partitions or disks with a filesystem already present.

To delete an existing filesystem you can use:
# wipefs -a /dev/md0 # wipefs -a /dev/sda7 

Create the backing device (and therefore the bcache device)

# bcache make -B /dev/md0
if necessary (device status is inactive)
# bcache register /dev/md0

Creating the cache device (and hooking the cache to the backing device)

# bcache make -C /dev/sda7
if necessary (device status is inactive)
# bcache register /dev/sda7

# bcache attach /dev/sda7 /dev/md0

# bcache set-cachemode /dev/md0 writeback

Check the status

# bcache show

The output from this command includes information similar to the following:
(if the status of a device is inactive, it means that it must be registered)

Name	Type	State	Bname	AttachToDev
/dev/md0	1 (data)	clean(running)	bcache0	/dev/sda7
/dev/sda7	3 (cache)	active	N/A	N/A

Optimize

# echo 0 > /sys/block/bcache0/bcache/cache/congested_write_threshold_us
# echo 0 > /sys/block/bcache0/bcache/cache/congested_read_threshold_us
# echo 600000000 > /sys/block/bcache0/bcache/sequential_cutoff
# echo 40 > /sys/block/bcache0/bcache/writeback_percent

In closing

Hopefully this article will provide some insight on the benefits of bcache if it suits your needs.

As always, nothing fits all cases and all people’s preferences. However, understanding (even roughly) how things work, and especially how they don’t work, as well as how to adapt them, makes the difference in having satisfactory results or not

---

Addendum

The following charts show the performance with a SSD NVME cache device rather than SSD SATA as shown above.

Performance of the cache device (SSD NVME 100GB)

Throughput:		
read, MiB/s: 16.31	read, MiB/s: 16.17	read, MiB/s: 15.77
written, MiB/s: 10.87	written, MiB/s: 10.78	written, MiB/s 10.51
Latency (ms):		
max: 17.50	max: 15.30	max: 46.61
95th percentile: 1.10	95th percentile: 1.10	95th percentile: 1.10

Performance with optimizations

Throughput:
read, MiB/s: 7.96	read, MiB/s: 6.87	read, MiB/s: 7.73
written, MiB/s: 5.31	written, MiB/s: 4.58	written, MiB/s 5.15
Latency (ms):
max: 50.79	max: 84.40	max: 108.71
95th percentile: 2.00	95th percentile: 2.03	95th percentile: 2.00

Performance with the writeback rate adjustment script

Throughput:		
read, MiB/s: 8.43	read, MiB/s: 7.52	read, MiB/s: 7.34
written, MiB/s: 5.62	written, MiB/s: 5.02	written, MiB/s 4.89
Latency (ms):		
max: 72.71	max: 78.60	max: 50.61
95th percentile: 2.00	95th percentile: 2.03	95th percentile: 2.11