Tag: Fedora Project community

We recently interviewed Joe Doss on how he uses Fedora. This is part of a series on the Fedora Magazine. The series profiles Fedora users and how they use Fedora to get things done. Contact us on the feedback form to express your interest in becoming a interviewee.

Who is Joe Doss?

Joe Doss lives in Chicago, Illinois USA and his favorite food is pizza. He is the Director of Engineering Operations and Kenna Security, Inc. Doss describes his employer this way: “Kenna uses data science to help enterprises combine their infrastructure and application vulnerability data with exploit intelligence to measure risk, predict attacks and prioritize remediation.”

His first Linux distribution was Red Hat Linux 5. A friend of his showed him a computer that wasn’t running Windows. Doss thought it was just a program to install on Windows when his friend gave him a Red Hat Linux 5 install disk. “I proceeded to install this Linux ‘program’ on my Father’s PC,” he says. Luckily for Doss, his father supported his interest in computers. “I ended up totally wiping out the Windows 95 install as a result and this was how I got my first computer.”

At Kenna, Doss’ group makes use of Fedora and Ansible: “We run Fedora Cloud in multiple VPC deployments in AWS and Google Compute with over 200 virtual machines. We use Ansible to automate everything we do with Fedora.”

Doss brews beer at home and contributes to open source in my free time. He also has a cat named Tibby. “I rescued Tibby off the street the Hyde Park neighborhood of Chicago when she was 7 months old. She is not very smart, but she makes up for that with cuteness.” His favorite place to visit is his childhood home of Michigan, but Doss says, “anywhere with a warm beach, a cool drink, and the ocean is pretty nice too.”

The Fedora community

Doss became involved with Fedora and the Fedora community through his job at Kenna Security. When he first joined the company they were using Ubuntu and Chef in production. There was a desire to make the infrastructure more reproducible and reliable, and he says, “I was able to greenfield our deployments with Fedora Cloud and Ansible.” This project got him involved in the Fedora Cloud release.

When asked about his first impression of the Fedora community, Doss said, “Overwhelming to be honest. There is so much going on and it is hard to figure out who are the stakeholders of each part of Fedora.” Once he figured out who he needed to talk to he found the community very welcoming and super supportive.

One of the ideas he had to improve the community was to unite the various projects and team under on bug tracking tool and community resource. “Pagure, Bugzilla, Github, Fedora Forums, Discourse Forums, Mailing lists… it is all over the place and hard to navigate at first.” Despite the initial complexity of becoming familiar with the Fedora Project, Doss feels it is amazingly rewarding to be involved. “It feels awesome it to be apart of a Linux distro that impacts so many people in very positive ways. You can make a difference.”

Doss called out Dusty Mabe at Red Hat for helping him become involved, saying Dusty “has been an amazing mentor and resource for enabling me to contribute back to Fedora.”

Doss has an interesting way of explaining to non-technical friends what he does. “Imagine changing the tires on a very large bus while it is going down the highway at 70 MPH and sometimes you need to get involved with the tire manufacturer to help make this process work well.” This metaphor helps people understand what replacing 200-plus VMs across more than five production VPCs in AWS and Google Compute with every Fedora release.

Doss drew my attention to one specific incident with Fedora 29 and Vagrant. “Recently we encountered an issue where Vagrant wouldn’t set the hostname on a Fresh Fedora 29 Beta VM. This was due to Fedora 29 Cloud no longer shipping the network service stub in favor of NetworkManager. This led to me working with a colleague at Kenna Security to send a patch upstream to the Vagrant project to help their developers produce a fix for Fedora 29. Vagrant usage with Fedora is a very large part of our development cycle at Kenna, and having this broken before the Fedora 29 release would have impacted us a lot.” As Doss said, “Sometimes you need to help make the tires before they go on the bus.”

Doss is the COPR Fedora, RHEL, and CentOS package maintainer for WireGuard VPN. “The CentOS repo just went over 60 thousand downloads last month which is pretty awesome.”

What Hardware?

Doss uses Fedora 29 cloud in the over five VPC deployments in AWS and Google computer. At home he has a SuperMicro SYS-5019A-FTN4 1U Server that runs Fedora 29 Server with Openshift OKD installed on it. His laptops are all Lenovo. “For Laptops I use a ThinkPad T460s for work and a ThinkPad 25 at home. Both have Fedora 29 installed. ThinkPads are the best with Fedora.”

What Software?

Doss used GNOME 3 as his preferred desktop on Fedora Workstation. “I use Sublime Text 3 for my text editor on the desktop or vim on servers.” For development and testing he uses Vagrant. “Ansible is what I use for any kind of automation with Fedora. I maintain an Ansible playbook for setting up my workstation.”

Ansible

I asked Doss if he had advice for people trying to learn Ansible.

“Start small. Automate the stuff that makes your life easier, but don’t over complicate it. Ansible Galaxy is a great resource to get things done quickly, but if you truly want to learn how to use Ansible, writing your own roles and playbooks the path I would take.

“I have helped a lot of my coworkers that have joined my Operations team at Kenna get up to speed on using Ansible by buying them a copy of Ansible for Devops by Jeff Geerling. This book will give anyone new to Ansible the foundation they need to start using it everyday. #ansible on Freenode is a great resource as well along with the official Ansible docs.”

Doss also said, “Knowing what to automate is most likely the most difficult thing to master without over complicating things. Debugging complex playbooks and roles is a close second.”

Home lab

He recommended setting up a home lab. “At Kenna and at home I use Vagrant with the Vagrant-libvirt plugin for developing Ansible roles and playbooks. You can iterate quickly to build your roles and playbooks on your laptop with your favorite editor and run vagrant provision to run your playbook. Quick feedback loop and the ability to burn down your Vagrant VM and start over quickly is an amazing workflow. Below is a sample Vagrant file that I keep handy to spin up a Fedora VM to test my playbooks.”

-- mode: ruby --
 vi: set ft=ruby :
 Vagrant.configure(2) do |config|
 config.vm.provision "shell", inline: "dnf install nfs-utils rpcbind @development-tools @ansible-node redhat-rpm-config gcc-c++ -y"
 config.ssh.forward_agent = true
 config.vm.define "f29", autostart: false do |f29|
 f29.vm.box = "fedora/29-cloud-base"
 f29.vm.hostname = "f29.example.com"
 f29.vm.provider "libvirt" do |vm|
 vm.memory = 2048
 vm.cpus = 2
 vm.driver = "kvm"
 vm.nic_model_type = "e1000"
 end
config.vm.synced_folder '.', '/vagrant', disabled: true 

config.vm.provision "ansible" do |ansible| 
 ansible.groups = { 
 } 
 ansible.playbook = "playbooks/main.yml"
 ansible.inventory_path = "inventory/development"
 ansible.extra_vars = {
 ansible_python_interpreter: "/usr/bin/python3" 
 }
# ansible.verbose = 'vvv' end
end
end

The Linux startup process goes through several stages before reaching the final graphical or multi-user target. The initramfs stage occurs just before the root file system is mounted. Dracut is a tool that is used to manage the initramfs. The dracut emergency shell is an interactive mode that can be initiated while the initramfs is loaded.

This article will show how to use the dracut command to modify the initramfs. Some basic troubleshooting commands that can be run from the dracut emergency shell will also be demonstrated.

The InitRAMFS

Initramfs stands for Initial Random-Access Memory File System. On modern Linux systems, it is typically stored in a file under the /boot directory. The kernel version for which it was built will be included in the file name. A new initramfs is generated every time a new kernel is installed.

By default, Fedora keeps the previous two versions of the kernel and its associated initramfs. This default can be changed by modifying the value of the installonly_limit setting the /etc/dnf/dnf.conf file.

You can use the lsinitrd command to list the contents of your initramfs archive:

The above screenshot shows that my initramfs archive contains the nouveau GPU driver. The modinfo command tells me that the nouveau driver supports several models of NVIDIA video cards. The lspci command shows that there is an NVIDIA GeForce video card in my computer’s PCI slot. There are also several basic Unix commands included in the archive such as cat and cp.

By default, the initramfs archive only includes the drivers that are needed for your specific computer. This allows the archive to be smaller and decreases the time that it takes for your computer to boot.

The Dracut Command

The dracut command can be used to modify the contents of your initramfs. For example, if you are going to move your hard drive to a new computer, you might want to temporarily include all drivers in the initramfs to be sure that the operating system can load on the new computer. To do so, you would run the following command:

The force parameter tells dracut that it is OK to overwrite the existing initramfs archive. The no-hostonly parameter overrides the default behavior of including only drivers that are germane to the currently-running computer and causes dracut to instead include all drivers in the initramfs.

By default dracut operates on the initramfs for the currently-running kernel. You can use the uname command to display which version of the Linux kernel you are currently running:

Once you have your hard drive installed and running in your new computer, you can re-run the dracut command to regenerate the initramfs with only the drivers that are needed for the new computer:

There are also parameters to add arbitrary drivers, dracut modules, and files to the initramfs archive. You can also create configuration files for dracut and save them under the /etc/dracut.conf.d directory so that your customizations will be automatically applied to all new initramfs archives that are generated when new kernels are installed. As always, check the man page for the details that are specific to the version of dracut you have installed on your computer:

The Dracut Emergency Shell

Sometimes something goes wrong during the initramfs stage of your computer’s boot process. When this happens, you will see “Entering emergency mode” printed to the screen followed by a shell prompt. This gives you a chance to try and fix things up manually and continue the boot process.

As a somewhat contrived example, let’s suppose that I accidentally deleted an important kernel parameter in my boot loader configuration:

The next time I reboot my computer, it will seem to hang for several minutes while it is trying to find the root partition and eventually give up and drop to an emergency shell.

From the emergency shell, I can enter journalctl and then use the Space key to page down though the startup logs. Near the end of the log I see a warning that reads “/dev/mapper/fedora-root does not exist”. I can then use the ls command to find out what does exist:

Hmm, the fedora-root LVM volume appears to be missing. Let’s see what I can find with the lvm command:

Ah ha! There’s my root partition. It’s just inactive. All I need to do is activate it and exit the emergency shell to continue the boot process:

The above example only demonstrates the basic concept. You can check the troubleshooting section of the dracut guide for a few more examples.

It is possible to access the dracut emergency shell manually by adding the rd.break parameter to your kernel command line. This can be useful if you need to access your files before any system services have been started.

Check the dracut.kernel man page for details about what kernel options your version of dracut supports:

When it comes to backing up a Fedora Silverblue system, some of the traditional tools may not function as expected. BorgBackup (Borg) is an alternative available that can provide backup capability for your Silverblue based systems. This how-to explains the steps for using BorgBackup 1.1.8 as a layered package to back up Fedora Silverblue 29 system.

On a normal Fedora Workstation system, dnf is used to install a package. However, on Fedora Silverblue, rpm-ostree install is used to install new software. This is termed layering on the Silverblue system, since the core ostree is an immutable image and the rpm package is layered onto the core system during the install process resulting in a new local image with the layered package.

“BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.”

From the Borg website

Additionally, the main way to interact with Borg is via the command line. Reading the Quick Start guide it becomes apparent that Borg is well suited to scripting. In fact, it is pretty much necessary to use some form of shell script when performing repeated thorough backup’s of a system. A basic script is provided in the Borg Quick Start guide , as a point to get started.

Installing Borg

In a terminal, type the following command to install BorgBackup as a layered package:

This installs BorgBackup to the Fedora Silverblue system. To use it, reboot into the new ostree with:

Now Borg is installed, and ready to use.

Some notes about Silverblue and it’s file system, layered packages and flatpaks

The file system

Silverblue is an immutable operating system based on ostree, with support for layering rpm’s through the use of rpm-ostree. At the user level, this means the path that appears as /home in a flatpak, will actually be /var/home to the system. For programs like Borg, and other backup tools this is important to remember since they often require the actual path, so in this example that would be /var/home instead of just /home.

Before starting a backup it’s a good idea to understand where potential data could be stored, and then if that data should be backed up. Silverblue’s file system layout is very specific with respect to what is writable and what is not. On Silverblue /etc and /var are the only places that are not immutable, therefore writable. On a single user system, typically the user home directory would be a likely choice for data backup. Normally excluding Downloads, but including Documents and more. Also, /etc is a logical choice for some configuration options you don’t want to go through again. Take notes of what to exclude from your home directory and from /etc. Some files and subdirectories of /etc you need root or sudo privileges to access.

Flatpaks

Flatpak applications store data in your home directory under $HOME/.var/app/flatpakapp, regardless of whether they were installed as user or system. If installed at a user level, there is also data found in $HOME/.local/share/flatpak/app/, or if installed at a system level it will be found in /var/lib/flatpak/app For the purposes of this article, it was enough to list the flatpak’s installed and redirect the output to a file for backing up. Reasoning that if there is a need to reinstall them (flatpaks) the list file could be used to do it from. For a more robust approach, examining the flatpak file system layouts can be done here.

Layering and rpm-ostree

There is no easy way for a user to retrieve the layered package information aside from the

command. Which shows the current and previous ostree commit’s layered packages, and if any commits are pinned they would be listed too. Below is the output on my system, note the LayeredPackages label at the end of each commit listing.

The command

is useful to retrieve a history of commits for the system. Type it in your terminal to see the output.

Preparing the backup repo

In order to use Borg to back up a system, you need to first initialize a Borg repo. Before initializing, the decision must be made to use encryption (or not) and if so, what mode.

With Borg the data can be protected using 256-bit AES encryption. The integrity and authenticity of the data, which is encrypted on the clientside, is verified using HMAC-SHA256. The encryption modes are listed below.

Encryption modes

The encryption mode decided on was keyfile-blake2, which requires a passphrase to be entered as well as the keyfile being needed.

Borg can use the following compression types which you can specify at backup creation time.

• lz4 (super fast, low compression)
• zstd (wide range from high speed and low compression to high compression and lower speed)
• zlib (medium speed and compression)
• lzma (low speed, high compression)

For compression lzma was chosen at setting 6, the highest sensible compression level. The initial backup took 4 minutes 59.98 seconds to complete, while subsequent ones have taken less than 20 seconds as a rule.

Borg init

To be able to perform backups with Borg, first, create a directory for your Borg repo:

and then change to it.

Next, initialize the Borg repo with the borg init command:

Borg will prompt for your passphrase, which is case sensitive, and at creation must be entered twice. A suitable passphrase of alpha-numeric characters and symbols, and of a reasonable length should be created. It can be changed later on if needed without affecting the keyfile, or your encrypted data. The keyfile can be exported and should be for backup purposes, along with the passphrase, and stored somewhere secure.

Creating a backup

Next, create a test backup of the Documents directory, remember on Silverblue the actual path to the user Documents directory is /var/home/username/Documents. In practice on Silverblue, it is suitable to use ~/ or $HOME to indicate your home directory. The distinction between the actual path and environment variables being the real path does not change whereas the environment variable can be changed. From within the Borg repo, type the following command

and that will create a backup of the Documents directory named borgtest. To break down the command a bit; create requires a repo location, in this case . since we are in the top level of the repo. That makes the path .::borgtest for the backup name. Finally /var/home/username/Documents is the location of the data we are backing up.

The following command

returns a listing of your backups, after a few days it look similar to this:

To delete the test backup, type the following in the terminal

at this time Borg will prompt for the encryption passphrase in order to delete the backup.

Pulling it together into a shell script

As mentioned Borg is an eminently script friendly tool. The Borg documentation links provided are great places to find out more about BorgBackup, and there is more. The example script provided by Borg was modified to suit this article. Below is a version with the basic parts that others could use as a starting point if desired. It tries to capture the three information pieces of the system and apps mentioned earlier. The output of flatpak list, rpm-ostree status, and ostree log as human readable files given the same names each time so overwritten each time. The repo setup had to be changed since the original example is for a remote server login with ssh, and this was intended to be used locally. The other changes mostly involved correcting directory paths, tailoring the excluded content to suit this systems home directory, and choosing the compression.

 
#!/bin/sh



# This gets the ostree commit data, this file is overwritten each time

sudo ostree log fedora-workstation:fedora/29/x86_64/silverblue > ostree.log



rpm-ostree status > rpm-ostree-status.lst



# Flatpaks get listed too

flatpak list > flatpak.lst



# Setting this, so the repo does not need to be given on the commandline:

export BORG_REPO=/var/home/usernamehere/borg_testdir



# Setting this, so you won't be asked for your repository passphrase:(Caution advised!)

export BORG_PASSPHRASE='usercomplexpassphrasehere'



# some helpers and error handling:

info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }

trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERM



info "Starting backup"



# Backup the most important directories into an archive named after

# the machine this script is currently running on:

borg create                         \

    --verbose                       \

    --filter AME                    \

    --list                          \

    --stats                         \

    --show-rc                       \

    --compression auto,lzma,6       \

    --exclude-caches                \

    --exclude '/var/home/*/borg_testdir'\

    --exclude '/var/home/*/Downloads/'\

    --exclude '/var/home/*/.var/'   \

    --exclude '/var/home/*/Desktop/'\

    --exclude '/var/home/*/bin/'    \

                                    \

    ::'{hostname}-{now}'            \

    /etc                            \

    /var/home/ssnow                 \



    backup_exit=$?



    info "Pruning repository"



    # Use the `prune` subcommand to maintain 7 daily, 4 weekly and 6 monthly

    # archives of THIS machine. The '{hostname}-' prefix is very important to

    # limit prune's operation to this machine's archives and not apply to

    # other machines' archives also:



    borg prune                          \

        --list                          \

        --prefix '{hostname}-'          \

        --show-rc                       \

        --keep-daily    7               \

        --keep-weekly   4               \

        --keep-monthly  6               \



    prune_exit=$?



    # use highest exit code as global exit code

    global_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))



    if [ ${global_exit} -eq 0 ]; then

        info "Backup and Prune finished successfully"

    elif [ ${global_exit} -eq 1 ]; then

        info "Backup and/or Prune finished with warnings"

    else

        info "Backup and/or Prune finished with errors"

    fi



    exit ${global_exit}
 

This listing is missing some more excludes that were specific to the test system setup and backup intentions, and is very basic with room for customization and improvement. For this test to write an article it wasn’t a problem having the passphrase inside of a shell script file. Under normal use it is better to enter the passphrase each time when performing the backup.