Tag: fedora

The Linux startup process goes through several stages before reaching the final graphical or multi-user target. The initramfs stage occurs just before the root file system is mounted. Dracut is a tool that is used to manage the initramfs. The dracut emergency shell is an interactive mode that can be initiated while the initramfs is loaded.

This article will show how to use the dracut command to modify the initramfs. Some basic troubleshooting commands that can be run from the dracut emergency shell will also be demonstrated.

The InitRAMFS

Initramfs stands for Initial Random-Access Memory File System. On modern Linux systems, it is typically stored in a file under the /boot directory. The kernel version for which it was built will be included in the file name. A new initramfs is generated every time a new kernel is installed.

By default, Fedora keeps the previous two versions of the kernel and its associated initramfs. This default can be changed by modifying the value of the installonly_limit setting the /etc/dnf/dnf.conf file.

You can use the lsinitrd command to list the contents of your initramfs archive:

The above screenshot shows that my initramfs archive contains the nouveau GPU driver. The modinfo command tells me that the nouveau driver supports several models of NVIDIA video cards. The lspci command shows that there is an NVIDIA GeForce video card in my computer’s PCI slot. There are also several basic Unix commands included in the archive such as cat and cp.

By default, the initramfs archive only includes the drivers that are needed for your specific computer. This allows the archive to be smaller and decreases the time that it takes for your computer to boot.

The Dracut Command

The dracut command can be used to modify the contents of your initramfs. For example, if you are going to move your hard drive to a new computer, you might want to temporarily include all drivers in the initramfs to be sure that the operating system can load on the new computer. To do so, you would run the following command:

The force parameter tells dracut that it is OK to overwrite the existing initramfs archive. The no-hostonly parameter overrides the default behavior of including only drivers that are germane to the currently-running computer and causes dracut to instead include all drivers in the initramfs.

By default dracut operates on the initramfs for the currently-running kernel. You can use the uname command to display which version of the Linux kernel you are currently running:

Once you have your hard drive installed and running in your new computer, you can re-run the dracut command to regenerate the initramfs with only the drivers that are needed for the new computer:

There are also parameters to add arbitrary drivers, dracut modules, and files to the initramfs archive. You can also create configuration files for dracut and save them under the /etc/dracut.conf.d directory so that your customizations will be automatically applied to all new initramfs archives that are generated when new kernels are installed. As always, check the man page for the details that are specific to the version of dracut you have installed on your computer:

The Dracut Emergency Shell

Sometimes something goes wrong during the initramfs stage of your computer’s boot process. When this happens, you will see “Entering emergency mode” printed to the screen followed by a shell prompt. This gives you a chance to try and fix things up manually and continue the boot process.

As a somewhat contrived example, let’s suppose that I accidentally deleted an important kernel parameter in my boot loader configuration:

The next time I reboot my computer, it will seem to hang for several minutes while it is trying to find the root partition and eventually give up and drop to an emergency shell.

From the emergency shell, I can enter journalctl and then use the Space key to page down though the startup logs. Near the end of the log I see a warning that reads “/dev/mapper/fedora-root does not exist”. I can then use the ls command to find out what does exist:

Hmm, the fedora-root LVM volume appears to be missing. Let’s see what I can find with the lvm command:

Ah ha! There’s my root partition. It’s just inactive. All I need to do is activate it and exit the emergency shell to continue the boot process:

The above example only demonstrates the basic concept. You can check the troubleshooting section of the dracut guide for a few more examples.

It is possible to access the dracut emergency shell manually by adding the rd.break parameter to your kernel command line. This can be useful if you need to access your files before any system services have been started.

Check the dracut.kernel man page for details about what kernel options your version of dracut supports:

When it comes to backing up a Fedora Silverblue system, some of the traditional tools may not function as expected. BorgBackup (Borg) is an alternative available that can provide backup capability for your Silverblue based systems. This how-to explains the steps for using BorgBackup 1.1.8 as a layered package to back up Fedora Silverblue 29 system.

On a normal Fedora Workstation system, dnf is used to install a package. However, on Fedora Silverblue, rpm-ostree install is used to install new software. This is termed layering on the Silverblue system, since the core ostree is an immutable image and the rpm package is layered onto the core system during the install process resulting in a new local image with the layered package.

“BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.”

From the Borg website

Additionally, the main way to interact with Borg is via the command line. Reading the Quick Start guide it becomes apparent that Borg is well suited to scripting. In fact, it is pretty much necessary to use some form of shell script when performing repeated thorough backup’s of a system. A basic script is provided in the Borg Quick Start guide , as a point to get started.

Installing Borg

In a terminal, type the following command to install BorgBackup as a layered package:

This installs BorgBackup to the Fedora Silverblue system. To use it, reboot into the new ostree with:

Now Borg is installed, and ready to use.

Some notes about Silverblue and it’s file system, layered packages and flatpaks

The file system

Silverblue is an immutable operating system based on ostree, with support for layering rpm’s through the use of rpm-ostree. At the user level, this means the path that appears as /home in a flatpak, will actually be /var/home to the system. For programs like Borg, and other backup tools this is important to remember since they often require the actual path, so in this example that would be /var/home instead of just /home.

Before starting a backup it’s a good idea to understand where potential data could be stored, and then if that data should be backed up. Silverblue’s file system layout is very specific with respect to what is writable and what is not. On Silverblue /etc and /var are the only places that are not immutable, therefore writable. On a single user system, typically the user home directory would be a likely choice for data backup. Normally excluding Downloads, but including Documents and more. Also, /etc is a logical choice for some configuration options you don’t want to go through again. Take notes of what to exclude from your home directory and from /etc. Some files and subdirectories of /etc you need root or sudo privileges to access.

Flatpaks

Flatpak applications store data in your home directory under $HOME/.var/app/flatpakapp, regardless of whether they were installed as user or system. If installed at a user level, there is also data found in $HOME/.local/share/flatpak/app/, or if installed at a system level it will be found in /var/lib/flatpak/app For the purposes of this article, it was enough to list the flatpak’s installed and redirect the output to a file for backing up. Reasoning that if there is a need to reinstall them (flatpaks) the list file could be used to do it from. For a more robust approach, examining the flatpak file system layouts can be done here.

Layering and rpm-ostree

There is no easy way for a user to retrieve the layered package information aside from the

command. Which shows the current and previous ostree commit’s layered packages, and if any commits are pinned they would be listed too. Below is the output on my system, note the LayeredPackages label at the end of each commit listing.

The command

is useful to retrieve a history of commits for the system. Type it in your terminal to see the output.

Preparing the backup repo

In order to use Borg to back up a system, you need to first initialize a Borg repo. Before initializing, the decision must be made to use encryption (or not) and if so, what mode.

With Borg the data can be protected using 256-bit AES encryption. The integrity and authenticity of the data, which is encrypted on the clientside, is verified using HMAC-SHA256. The encryption modes are listed below.

Encryption modes

The encryption mode decided on was keyfile-blake2, which requires a passphrase to be entered as well as the keyfile being needed.

Borg can use the following compression types which you can specify at backup creation time.

• lz4 (super fast, low compression)
• zstd (wide range from high speed and low compression to high compression and lower speed)
• zlib (medium speed and compression)
• lzma (low speed, high compression)

For compression lzma was chosen at setting 6, the highest sensible compression level. The initial backup took 4 minutes 59.98 seconds to complete, while subsequent ones have taken less than 20 seconds as a rule.

Borg init

To be able to perform backups with Borg, first, create a directory for your Borg repo:

and then change to it.

Next, initialize the Borg repo with the borg init command:

Borg will prompt for your passphrase, which is case sensitive, and at creation must be entered twice. A suitable passphrase of alpha-numeric characters and symbols, and of a reasonable length should be created. It can be changed later on if needed without affecting the keyfile, or your encrypted data. The keyfile can be exported and should be for backup purposes, along with the passphrase, and stored somewhere secure.

Creating a backup

Next, create a test backup of the Documents directory, remember on Silverblue the actual path to the user Documents directory is /var/home/username/Documents. In practice on Silverblue, it is suitable to use ~/ or $HOME to indicate your home directory. The distinction between the actual path and environment variables being the real path does not change whereas the environment variable can be changed. From within the Borg repo, type the following command

and that will create a backup of the Documents directory named borgtest. To break down the command a bit; create requires a repo location, in this case . since we are in the top level of the repo. That makes the path .::borgtest for the backup name. Finally /var/home/username/Documents is the location of the data we are backing up.

The following command

returns a listing of your backups, after a few days it look similar to this:

To delete the test backup, type the following in the terminal

at this time Borg will prompt for the encryption passphrase in order to delete the backup.

Pulling it together into a shell script

As mentioned Borg is an eminently script friendly tool. The Borg documentation links provided are great places to find out more about BorgBackup, and there is more. The example script provided by Borg was modified to suit this article. Below is a version with the basic parts that others could use as a starting point if desired. It tries to capture the three information pieces of the system and apps mentioned earlier. The output of flatpak list, rpm-ostree status, and ostree log as human readable files given the same names each time so overwritten each time. The repo setup had to be changed since the original example is for a remote server login with ssh, and this was intended to be used locally. The other changes mostly involved correcting directory paths, tailoring the excluded content to suit this systems home directory, and choosing the compression.

 
#!/bin/sh



# This gets the ostree commit data, this file is overwritten each time

sudo ostree log fedora-workstation:fedora/29/x86_64/silverblue > ostree.log



rpm-ostree status > rpm-ostree-status.lst



# Flatpaks get listed too

flatpak list > flatpak.lst



# Setting this, so the repo does not need to be given on the commandline:

export BORG_REPO=/var/home/usernamehere/borg_testdir



# Setting this, so you won't be asked for your repository passphrase:(Caution advised!)

export BORG_PASSPHRASE='usercomplexpassphrasehere'



# some helpers and error handling:

info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }

trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERM



info "Starting backup"



# Backup the most important directories into an archive named after

# the machine this script is currently running on:

borg create                         \

    --verbose                       \

    --filter AME                    \

    --list                          \

    --stats                         \

    --show-rc                       \

    --compression auto,lzma,6       \

    --exclude-caches                \

    --exclude '/var/home/*/borg_testdir'\

    --exclude '/var/home/*/Downloads/'\

    --exclude '/var/home/*/.var/'   \

    --exclude '/var/home/*/Desktop/'\

    --exclude '/var/home/*/bin/'    \

                                    \

    ::'{hostname}-{now}'            \

    /etc                            \

    /var/home/ssnow                 \



    backup_exit=$?



    info "Pruning repository"



    # Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly

    # archives of THIS machine. The '{hostname}-' prefix is very important to

    # limit prune's operation to this machine's archives and not apply to

    # other machines' archives also:



    borg prune                          \

        --list                          \

        --prefix '{hostname}-'          \

        --show-rc                       \

        --keep-daily    7               \

        --keep-weekly   4               \

        --keep-monthly  6               \



    prune_exit=$?



    # use highest exit code as global exit code

    global_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))



    if [ ${global_exit} -eq 0 ]; then

        info "Backup and Prune finished successfully"

    elif [ ${global_exit} -eq 1 ]; then

        info "Backup and/or Prune finished with warnings"

    else

        info "Backup and/or Prune finished with errors"

    fi



    exit ${global_exit}
 

This listing is missing some more excludes that were specific to the test system setup and backup intentions, and is very basic with room for customization and improvement. For this test to write an article it wasn’t a problem having the passphrase inside of a shell script file. Under normal use it is better to enter the passphrase each time when performing the backup.