Posted on Leave a comment

Automating Network Devices with Ansible

Ansible is a great automation tool for system and network engineers, with Ansible we can automate small network to a large scale enterprise network. I have been using Ansible to automate both Aruba, and Cisco switches from my Fedora powered laptops for a couple of years. This article covers the requirements and executing a couple of playbooks.

Configuring Ansible

If Ansible is not installed, it can be installed using the command below

$ sudo dnf -y install ansible

Once installed, create a folder in your home directory or a directory of your preference and copy the ansible configuration file. For this demonstration, I will be using the following.

$ mkdir -pv /home/$USER/network_automation
$ sudo cp -v /etc/ansible.cfg /home/$USER/network_automation
$ cd /home/$USER/network_automation
$ sudo chown $USER.$USER && chmod 0600 ansible.cfg

To prevent lengthy commands from failing, edit the ansible.cfg and append the following lines. We must add the persistent connection and set the desired time in seconds for the command_timeout as demonstrated below. A use case where this is useful is when you are performing backups of a network device that has a lengthy configuration.

$ vim ansible.cfg
[persistent_connection]
command_timeout = 300
connection_timeout = 30

Requirements

If SELinux is enabled, you will need to install SELinux binding, which is required when using the copy module.

# Install SELinux bindings
dnf -y install python3-libselinux python3-libsemanage

Creating the inventory

The inventory holds the names of the network assets, and grouping of the assets are in square brackets [], below is a  sample inventory.

[site_a]
Core_A ansible_host=192.168.122.200
Distro_A ansible_host=192.168.122.201
Distro_B ansible_host=192.168.122.202

Group vars can be used to address the common variables, for example, credentials, network operating system, and so on. Ansible document on inventory provides additional details.

Playbook

Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process.
Ansible Playbook

Read Operations

Let us create a simple playbook to run a show command to read the configuration on a few switches.

 
  1 ---
  2 - name: Basic Playbook
  3   hosts: site_a
  4   connection: local
  5
  6   tasks:
  7   - name: Get Interface Brief
  8     ios_command:
  9       commands:
 10         - show ip interface brief | e una
 11     register: interfaces
 12
 13   - name: Print results
 14     debug:
 15       msg: "{{ interfaces.stdout[0] }}
Without Debug

With Debug

The above images show the differences without and with the debug module respectively.

Let’s break the playbook into three blocks, starting with lines 1 to 4.

  • The three dashes/hyphens starts the YAML document
  • The hosts defines the hosts or host groups, multiple groups are comma-separated
  • Connection defines the methodology to connect to the network devices. Another option is network_cli (recommended method) and will be used later in this article. See IOS Platform Options for more details.

Lines 6 to 11 starts the tasks, we will be using ios_command and ios_config. This play will execute the show command show ip interface brief | e una and save the output from the command into the interfaces variable, with the register key.

Lines 13 to 15, by default, when you execute a show command you will not see the output, though this is not used during automation. It is very useful for debugging; therefore, the debug module was used.

The below video shows the execution of the playbook. There are a couple of ways you can execute the playbook.

  • Passing arguments to the command line, for example, include -u <username> -k to prompt for the remote user credentials
 
ansible-playbook -i inventory show_demo.yaml -u admin -k
  • Include the credentials in the host or group vars
 
ansible-playbook -i inventory show_demo.yaml

Never store passwords in plain text. We recommend using SSH keys to authenticate SSH connections. Ansible supports ssh-agent to manage your SSH keys. If you must use passwords to authenticate SSH connections, we recommend encrypting them with
Using Vault in Playbooks

Passing arguments to the command line
Credentials in the inventory

If we want to save the output to a file, we will use the copy module as shown in the playbook below. In addition to using the copy module, we will include the backup_dir variable to specify the directory path.

 
---
- name: Get System Infomation
  hosts: site_a
  connection: network_cli
  gather_facts: no
 
  vars:
    backup_dir: /home/eramirez/dev/ansible/fedora_magazine
 
  tasks:
  - name: get system interfaces
    ios_command:
      commands:
        - show ip int br | e una
    register: interface
   
  - name: Save result to disk
    copy:
      content: "{{ interface.stdout[0] }}"
      dest: "{{ backup_dir }}/{{ inventory_hostname }}.txt"

To demonstrate the use of variables in the inventory, we will use plain text. This method Must not be used in production.

 
[site_a]
Core_A ansible_host=192.168.122.200
Distro_A ansible_host=192.168.122.201
Distro_B ansible_host=192.168.122.202
[all:vars]
ansible_connection=network_cli
ansible_network_os=ios
ansible_user=admin
ansible_password=fedora
ansible_become=yes
ansible_become_password=yes
ansible_become_method=enable

Write Operations

In the previous section, we saw that we could get information from the network devices; in this section, we will write (add/modify) the configuration on these network devices. To make changes to the network device, we will be using the ios config module.

Let us create a playbook to configure a couple of interfaces in all of the network devices in site_a. We will first take a backup of the current configuration of all devices in site_a. Lastly, we will save the configuration.

 
---
- name: Get System Infomation
  hosts: site_a
  connection: network_cli
  gather_facts: no
 
  vars:
    backup_dir: /home/eramirez/dev/ansible/fedora_magazine
 
  tasks:
  - name: Backup configs
    ios_config:
      backup: yes
      backup_options:
        filename: "{{ inventory_hostname }}_running_cfg.txt"
        dir_path: "{{ backup_dir }}"
   
  - name: get system interfaces
    ios_config:
      lines:
        - description Raspberry Pi
        - switchport mode access
        - switchport access vlan 100
        - spanning-tree portfast
        - logging event link-status
        - no shutdown
      parents: "{{ item }}"
    with_items:
      - interface FastEthernet1/12
      - interface FastEthernet1/13
     
  - name: Save switch configuration
    ios_config:
      save_when: modified

Before we execute the playbook, we will first validate the interface configuration. We will then run the playbook and confirm the changes as illustrated below.

Conclusion

This article is a basic introduction to whet your appetite that demonstrates how Ansible is used to manage network devices. Ansible is capable of automating a vast network, which includes MPLS routing and performing validation before executing the next task.

Posted on Leave a comment

Protect your system with fail2ban and firewalld blacklists

If you run a server with a public-facing SSH access, you might have experienced malicious login attempts. This article shows how to use two utilities to keep the intruder out of our systems.

To protect against repeated ssh login attempts, we’ll look at fail2ban. And if you don’t travel much, and perhaps stay in one or two countries, you can configure firewalld to only allow access from the countries you choose.

First let’s work through a little terminology for those not familiar with the various applications we’ll need to make this work:

fail2ban: Daemon to ban hosts that cause multiple authentication errors.

fail2ban will monitor the SystemD journal to look for failed authentication attempts for whichever jails have been enabled. After the number of failed attempts specified it will add a firewall rule to block that specific IP address for an amount of time configured.

firewalld: A firewall daemon with D-Bus interface providing a dynamic firewall.

Unless you’ve manually decided to use traditional iptables, you’re already using firewalld on all supported releases of Fedora and CentOS.

Assumptions

  • The host system has an internet connection and is either fully exposed directly, through a DMZ (both REALLY bad ideas unless you know what you’re doing), or has a port being forwarded to it from a router.
  • While most of this might apply to other systems, this article assumes a current version of Fedora (31 and up) or RHEL/CentOS 8. On CentOS you must enable the Fedora EPEL repo with
    sudo dnf install epel-release

Install & Configuration

Fail2Ban

More than likely whichever FirewallD zone is set already allows SSH access but the sshd service itself is not enabled by default. To start it manually and without permanently enabling on boot:

$ sudo systemctl start sshd

Or to start and enable on boot:

$ sudo systemctl enable --now sshd

The next step is to install, configure, and enable fail2ban. As usual the install can be done from the command line:

$ sudo dnf install fail2ban

Once installed the next step is to configure a jail (a service you want to monitor and ban at whatever thresholds you’ve set). By default IPs are banned for 1 hour (which is not near long enough). The best practice is to override the system defaults using *.local files instead of directly modifying the *.config files. If we look at my jail.local we see:

# cat /etc/fail2ban/jail.local
[DEFAULT] # "bantime" is the number of seconds that a host is banned.
bantime = 1d # A host is banned if it has generated "maxretry" during the last "findtime"
findtime = 1h # "maxretry" is the number of failures before a host get banned.
maxretry = 5

Turning this into plain language, after 5 attempts within the last hour the IP will be blocked for 1 day. There’s also options for increasing the ban time for IPs that get banned multiple times, but that’s the subject for another article.

The next step is to configure a jail. In this tutorial sshd is shown but the steps are more or less the same for other services. Create a configuration file inside /etc/fail2ban/jail.d. Here’s mine:

# cat /etc/fail2ban/jail.d/sshd.local
[sshd]
enabled = true

It’s that simple! A lot of the configuration is already handled within the package built for Fedora (Hint: I’m the current maintainer). Next enable and start the fail2ban service.

$ sudo systemctl enable --now fail2ban

Hopefully there were not any immediate errors, if not, check the status of fail2ban using the following command:

$ sudo systemctl status fail2ban

If it started without errors it should look something like this:

$ systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2020-06-16 07:57:40 CDT; 5s ago
Docs: man:fail2ban(1)
Process: 11230 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
Main PID: 11235 (f2b/server)
Tasks: 5 (limit: 4630)
Memory: 12.7M
CPU: 109ms
CGroup: /system.slice/fail2ban.service
└─11235 /usr/bin/python3 -s /usr/bin/fail2ban-server -xf start
Jun 16 07:57:40 localhost.localdomain systemd[1]: Starting Fail2Ban Service…
Jun 16 07:57:40 localhost.localdomain systemd[1]: Started Fail2Ban Service.
Jun 16 07:57:41 localhost.localdomain fail2ban-server[11235]: Server ready

If recently started, fail2ban is unlikely to show anything interesting going on just yet but to check the status of fail2ban and make sure the jail is enabled enter:

$ sudo fail2ban-client status
Status
|- Number of jail:	1
`- Jail list:	sshd

And the high level status of the sshd jail is shown. If multiple jails were enabled they would show up here.

To check the detailed status a jail, just add the jail to the previous command. Here’s the output from my system which has been running for a while. I have removed the banned IPs from the output:

$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed:	8
| |- Total failed:	4399
| `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions |- Currently banned:	101 |- Total banned:	684 `- Banned IP list: ...

Monitoring the fail2ban log file for intrusion attempts can be achieved by “tailing” the log:

$ sudo tail -f /var/log/fail2ban.log

Tail is a nice little command line utility which by default shows the last 10 lines of a file. Adding the “-f” tells it to follow the file which is a great way to watch a file that’s still being written to.

Since the output has real IPs in it, a sample won’t be provided but it’s pretty human readable. The INFO lines will usually be attempts at a login. If enough attempts are made from a specific IP address you will see a NOTICE line showing an IP address was banned. After the ban time has been reached you will see an NOTICE unban line.

Lookout for several WARNING lines. Most often this happens when a ban is added but fail2ban finds the IP address already in its ban database, which means banning may not be working correctly. If recently installed the fail2ban package it should be setup for FirewallD rich rules. The package was only switched from “ipset” to “rich rules” as of fail2ban-0.11.1-6 so if you have an older install of fail2ban it may still be trying to use the ipset method which utilizes legacy iptables and is not very reliable.

FirewallD Configuration

Reactive or Proactive?

There are two strategies that can be used either separately or together. Reactive or proactive permanent blacklisting of individual IP address or subnets based on country of origin.

For the reactive approach once fail2ban has been running for a while it’s a good idea to take a look at how “bad is bad” by running sudo fail2ban-client status sshd again. There most likely will be many banned IP addresses. Just pick one and try running whois on it. There can be quite a bit of interesting information in the output but for this method, only the country of origin is of importance. To keep things simple, let’s filter out everything but the country.

For this example a few well known domain names will be used:

$ whois google.com | grep -i country
Registrant Country: US
Admin Country: US
Tech Country: US
$ whois rpmfusion.org | grep -i country
Registrant Country: FR
$ whois aliexpress.com | grep -i country
Registrant Country: CN

The reason for the grep -i is to make grep non-case sensitive while most entries use “Country”, some are in all lower case so this method matches regardless.

Now that the country of origin of an intrusion attempt is known the question is, “Does anyone from that country have a legitimate reason to connect to this computer?” If the answer is NO, then it should be acceptable to block the entire country.

Functionally the proactive approach it not very different from the reactive approach, however, there are countries from which intrusion attempts are very common. If the system neither resides in one of those countries, nor has any customers originating from them, then why not add them to the blacklist now rather than waiting?

Blacklisting Script and Configuration

So how do you do that? With FirewallD ipsets. I developed the following script to automate the process as much as possible:

#!/bin/bash
# Based on the below article
# https://www.linode.com/community/questions/11143/top-tip-firewalld-and-ipset-country-blacklist # Source the blacklisted countries from the configuration file
. /etc/blacklist-by-country # Create a temporary working directory
ipdeny_tmp_dir=$(mktemp -d -t blacklist-XXXXXXXXXX)
pushd $ipdeny_tmp_dir # Download the latest network addresses by country file
curl -LO http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz
tar xf all-zones.tar.gz # For updates, remove the ipset blacklist and recreate
if firewall-cmd -q --zone=drop --query-source=ipset:blacklist; then firewall-cmd -q --permanent --delete-ipset=blacklist
fi # Create the ipset blacklist which accepts both IP addresses and networks
firewall-cmd -q --permanent --new-ipset=blacklist --type=hash:net \ --option=family=inet --option=hashsize=4096 --option=maxelem=200000 \ --set-description="An ipset list of networks or ips to be dropped." # Add the address ranges by country per ipdeny.com to the blacklist
for country in $countries; do firewall-cmd -q --permanent --ipset=blacklist \ --add-entries-from-file=./$country.zone && \ echo "Added $country to blacklist ipset."
done # Block individual IPs if the configuration file exists and is not empty
if [ -s "/etc/blacklist-by-ip" ]; then echo "Adding IPs blacklists." firewall-cmd -q --permanent --ipset=blacklist \ --add-entries-from-file=/etc/blacklist-by-ip && \ echo "Added IPs to blacklist ipset."
fi # Add the blacklist ipset to the drop zone if not already setup
if firewall-cmd -q --zone=drop --query-source=ipset:blacklist; then echo "Blacklist already in firewalld drop zone."
else echo "Adding ipset blacklist to firewalld drop zone." firewall-cmd --permanent --zone=drop --add-source=ipset:blacklist
fi firewall-cmd -q --reload popd
rm -rf $ipdeny_tmp_dir

This should be installed to /usr/local/sbin and don’t forget to make it executable!

$ sudo chmod +x /usr/local/sbin/firewalld-blacklist

Then create a configure file: /etc/blacklist-by-country:

# Which countries should be blocked?
# Use the two letter designation separated by a space.
countries=""

And another configuration file /etc/blacklist-by-ip, which is just one IP per line without any additional formatting.

For this example 10 random countries were selected from the ipdeny zones:

# ls | shuf -n 10 | sed "s/\.zone//g" | tr '\n' ' '
nl ee ie pk is sv na om gp bn

Now as long as at least one country has been added to the config file it’s ready to run!

$ sudo firewalld-blacklist % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 142 100 142 0 0 1014 0 --:--:-- --:--:-- --:--:-- 1014
100 662k 100 662k 0 0 989k 0 --:--:-- --:--:-- --:--:-- 989k
Added nl to blacklist ipset.
Added ee to blacklist ipset.
Added ie to blacklist ipset.
Added pk to blacklist ipset.
Added is to blacklist ipset.
Added sv to blacklist ipset.
Added na to blacklist ipset.
Added om to blacklist ipset.
Added gp to blacklist ipset.
Added bn to blacklist ipset.
Adding ipset blacklist to firewalld drop zone.
success

To verify that the firewalld blacklist was successful, check the drop zone and blacklist ipset:

$ sudo firewall-cmd --info-zone=drop
drop (active) target: DROP icmp-block-inversion: no interfaces: sources: ipset:blacklist services: ports: protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: $ sudo firewall-cmd --info-ipset=blacklist | less
blacklist type: hash:net options: family=inet hashsize=4096 maxelem=200000 entries: 

The second command will output all of the subnets that were added based on the countries blocked and can be quite lengthy.

So now what do I do?

While it will be a good idea to monitor things more frequently at the beginning, over time the number of intrusion attempts should decline as the blacklist grows. Then the goal should be maintenance rather than active monitoring.

To this end I created a SystemD service file and timer so that on a monthly basis the by country subnets maintained by ipdeny are refreshed. In fact everything discussed here can be downloaded from my pagure.io project:

https://pagure.io/firewalld-blacklist

Aren’t you glad you read the whole article? Now just download the service file and timer to /etc/systemd/system/ and enable the timer:

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now firewalld-blacklist.timer

Posted on Leave a comment

Contribute at the Fedora Test Week for Kernel 5.7

The kernel team is working on final integration for kernel 5.7. This version was just recently released, and will arrive soon in Fedora. As a result, the Fedora kernel and QA teams have organized a test week from Monday, June 22, 2020 through Monday, June 29, 2020. Refer to the wiki page for links to the test images you’ll need to participate. Read below for details.

How does a test week work?

A test week is an event where anyone can help make sure changes in Fedora work well in an upcoming release. Fedora community members often participate, and the public is welcome at these events. If you’ve never contributed before, this is a perfect way to get started.

To contribute, you only need to be able to do the following things:

  • Download test materials, which include some large files
  • Read and follow directions step by step

The wiki page for the kernel test day has a lot of good information on what and how to test. After you’ve done some testing, you can log your results in the test day web application. If you’re available on or around the day of the event, please do some testing and report your results. We have a document which provides all the steps written.

Happy testing, and we hope to see you on test day.

Posted on Leave a comment

Fedora Silverblue, an introduction for developers

The Fedora Silverblue project takes Fedora workstation, libostree and podman, puts them in a blender, and creates a new Immutable Fedora Workstation. Fedora Silverblue is an OS that stops you from changing the core system files arbitrarily, and readily allows you to change the environment system files. The article What is Silverblue describes the big picture, and this article drills down into details for the developer.

Fedora Silverblue ties together a few different projects to make a system that is a git-like object, capable of layering packages, and has a container focused work flow. Silverblue is not the only distribution going down this road. It is the desktop equivalent of CoreOS, the server OS used by Red Hat Openshift.

Silverblue’s idea of ‘immutable’ has nothing to do with immutable layers in a container. Silverblue keeps system files immutable by making them read-only.

Why immutable?

Has an upgrade left your system in an unusable state? Have you wondered why one server in a pool of identical machines is being weird? These problems can happen when one system library – one tiny little file out of hundreds – is corrupted, badly configured or the wrong version. Or maybe your upgrade works fine but it’s not what you’d hoped for, and you want to roll back to the previous state.

An immutable OS is intended to stop problems like these biting you. This is not an easy thing to achieve – simple changes, like flipping the file system between read-write and read-only, may only change a fault-finding headache to a maintenance headache.

Freezing the system is good news for sysadmins, but what about developers? Setting up a development environment means heavily customizing the system, and filling it with living code that changes over time. The answer is partly a case of combining components, and partly the ability to swap between OS versions.

How it works

So how do you get the benefits of immutability without losing the ability to do your work? If you’re thinking ‘containers’, good guess – part of the solution uses podman. But much of the work happens underneath the container layer, at the OS level.

Fedora Silverblue ties together a few different projects to turn an immutable OS into a usable workstation. Silverblue uses libostree to provide the base system, lets you edit config files in /etc/, and provides three different ways to install packages.

  • rpm-ostree installs RPM packages, similar to DNF in the traditional Fedora workstation. Use this for things that shouldn’t go in containers, like KVM/libvirt.
  • flatpak installs packages from a central flathub repo. This is the one-stop shop for graphical desktop apps like LibreOffice.
  • The traditional dnf install still works, but only inside a toolbox (a Fedora container). A developer’s workbench goes in a toolbox.

If you want to know more about these components, check out Pieces of Silverblue.

Rolling back and pinning upgrades

All operating systems need upgrades. Features are added, security holes are plugged and bugs are squashed. But sometimes an upgrade is not a developer’s friend.

A developer depends on many things to get the job done. A good development environment is stuffed with libraries, editors, toolchains and apps that are controlled by the OS, not the developer. An upgrade may cause trouble. Have any of these situations happened to you?

  • A new encryption library is too strict, and an upgrade stopped an API working.
  • Code works well, but has deprecated syntax. An upgrade brought error-throwing misery.
  • The development environment is lovingly hand-crafted. An upgrade broke dependencies and added conflicts.

In a traditional environment, unpicking a troublesome upgrade is hard. In Silverblue, it’s easy. Silverblue keeps two copies of the OS – your current upgrade and your previous version. Point the OS at the previous version, reboot, and you’ve got your old system files back.

You aren’t limited to two copies of your file system – you can keep more by pinning your favorite versions. Dusty Mabe, one of the engineers who has been working on the system since the Project Atomic days, describes how to pin extra copies of the OS in his article Pinning Deployments in OSTree Based Systems.

Your home directory is not affected by rolling back. Rpm-ostree does not touch /etc/ and /var/.

System updates and package installs

Silverblue’s rpm-ostree treats all the files as one object, stored in a repository. The working file system is a checked-out copy of this object. After a system update, you get two objects in that repository – one current object and one updated object. The updated object is checked out and becomes the new file system.

You install your workhorse applications in toolboxes, which provide container isolation. And you install your desktop applications using Flatpak.

This new OS requires a shift in approach. For instance, you don’t have to keep only one copy of your system files – you can store a few and select which one you use. That means you can swap back and forth between an old Fedora release and the rawhide (development) version in a matter of minutes.

Build your own Silverblue VM

You can safely install Fedora Silverblue in a VM on your workstation. If you’ve got a hypervisor and half an hour to spare (10 minutes for ISO download, and 20 minutes for the build), you can see for yourself.

  1. Download Fedora Silverblue ISO from
  2. https://silverblue.fedoraproject.org/download (not Fedora workstation from https://getfedora.org/).
  3. Boot a VM with the Fedora Silverblue ISO. You can squeeze Fedora into compute resources of 1 CPU, 1024MiB of memory and 12GiB of storage, but bigger is better.
  4. Answer Anaconda’s questions.
  5. Wait for the Gnome desktop to appear.
  6. Answer Initial Setup’s questions.

Then you’re ready to set up your developer’s tools. If you’re looking for an IDE, check these out. Use flatpak on the desktop to install them.

Finally, use the CLI to create your first toolbox. Load it with modules using npm, gem, pip, git or your other favorite tools.

Help!

If you get stuck, ask questions at the forum.

If you’re looking for ideas about how to use Silverblue, read articles in the magazine.

Is Silverblue for you?

Silverblue is full of shiny new tech. That in itself is enough to attract the cool kids, like moths to a flame. But this OS is not for everyone. It’s a young system, so some bugs will still be lurking in there. And pioneering tech requires a change of habit – that’s extra cognitive load that the new user may not want to take on.

The OS brings immutable benefits, like keeping your system files safe. It also brings some drawbacks, like the need to reboot after adding system packages. Silverblue also enables new ways of working. If you want to explore new directions in the OS, find out if Silverblue brings benefits to your work.

Posted on Leave a comment

How to rebase to Fedora 32 on Silverblue

Silverblue is an operating system for your desktop built on Fedora. It’s excellent for daily use, development, and container-based workflows. It offers numerous advantages such as being able to roll back in case of any problems. If you want to update to Fedora 32 on your Silverblue system, this article tells you how. It not only shows you what to do, but also how to revert back things if anything unforeseen happens.

Prior to actually doing the rebase to Fedora 32, it is recommended to perform any pending updates. This is accomplished by entering the following at the terminal

rpm-ostree update

or installing updates through GNOME Software and following with a system reboot.

Rebasing using GNOME Software

The GNOME Software shows you that there is new version of Fedora available on the Updates screen.

Fedora 32 is available

First thing you need to do is to download the new image, so click on the Download button. This will take some time and after it’s done you will see that update is ready for install.

Fedora 32 is ready for installation

Click on the Install button. This step will take only a few moments and then you will be prompted to restart your computer.

Restart is needed to rebase to Fedora 32 Silverblue

Click on Restart button and you are done. After restart you will end up in new and shiny release of Fedora 32. Easy, isn’t it?

Rebasing using terminal

If you prefer to do everything in a terminal, than this next guide is for you.

Rebasing to Fedora 32 using terminal is easy. First, check if the 32 branch is available, which should be true now:

$ ostree remote refs fedora

You should see the following in the output:

fedora:fedora/32/x86_64/silverblue

Next, rebase your system to the Fedora 32 branch.

$ rpm-ostree rebase fedora:fedora/32/x86_64/silverblue

Finally, the last thing to do is restart your computer and boot to Fedora 32.

How to revert things back

If anything bad happens — for instance, if you can’t boot to Fedora 32 at all — it’s easy to go back. Just pick the previous entry in GRUB, and your system will start in its previous state before switching to Fedora 32. To make this change permanent, use the following command:

$ rpm-ostree rollback

That’s it. Now you know how to rebase Silverblue to Fedora 32 and back. So why not do it today?

Posted on Leave a comment

Fedora Classroom Session: IRC 101

The Fedora Classroom is a project to help people by spreading knowledge on subjects related to Fedora for others, If you would like to propose a session, feel free to open a ticket here with the tag “classroom.” If you’re interested in taking a proposed session, kindly let us know and once you take it, you will be awarded the Sensei Badge too as a token of appreciation. Recordings from the previous sessions can be found here.

We’re back with another awesome classroom on IRC 101 led by Pac23.

About the Session: A Beginners Guide to Internet Relay Chat

In short, the IRC 101 session will be a guide for newcomers on how to get started with IRC with the Fedora community and hang out with other contributors in IRC. After finishing the session you will have the knowledge to setup your IRC client and start communicating with other Fedora people.

When and where

The Classroom session will be organized on May 9th, 16:00 UTC. Here’s a link to see what time it is in your timezone. The session will be streamed on Fedora Project’s YouTube channel.

Topics covered in the session

  • Why IRC & How it works?
  • How to install an IRC Client.
  • Registering your nick in IRC
  • Some basic commands, modes & access controls
  • Joining fedora channels
  • Brownie Topic: Fedora bots in IRC.

About the instructor

Pac23’s been around in the Fedora community and contributing to the project for around a year. He’s started with volunteering to package a custom kernel. He’s also a Computer Engineering undergrad at the University of Mumbai. His interests mostly reside in DevOps, IoT & system design. Outside computer science, he loves traveling, airplanes and history. He can be found as pac23 in IRC channels including #fedora-neuro, #fedora-devel, and #fedora-kernel.

If you miss the session, no worries. The recording will also be uploaded in the Fedora Project‘s YouTube channel.

We hope you can attend and enjoy this experience from some of the awesome people that work in Fedora Project. We look forward to seeing you in the Classroom session.


Photograph used in feature image is San Simeon School House by Anita RitenourCC-BY 2.0.

Posted on Leave a comment

Using mergerfs to increase your virtual storage

What happens if you have multiple disks or partitions that you’d like to use for a media project and you don’t want to lose any of your existing data, but you’d like to have everything located or mounted under one drive. That’s where mergerfs can come to your rescue!

mergerfs is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices.

You will need to grab the latest RPM from their github page here. The releases for Fedora have fc and the version number in the name. For example here is the version for Fedora 31:

mergerfs-2.29.0-1.fc31.x86_64.rpm

Installing and configuring mergerfs

Install the mergerfs package that you’ve downloaded using sudo:

$ sudo dnf install mergerfs-2.29.0-1.fc31.x86_64.rpm

You will now be able to mount multiple disks as one drive. This comes in handy if you have a media server and you’d like all of your media files to show up under one location. If you upload new files to your system, you can copy them to your mergerfs directory and mergerfs will automatically copy them to which ever drive has enough free space available.

Here is an example to make it easier to understand:

$ df -hT | grep disk
/dev/sdb1 ext4 23M 386K 21M 2% /disk1
/dev/sdc1 ext4 44M 1.1M 40M 3% /disk2 $ ls -l /disk1/Videos/
total 1
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv $ ls -l /disk2/Videos/
total 2
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv

In this example there are two disks mounted as disk1 and disk2. Both drives have a Videos directory with existing files.

Now we’re going to mount those drives using mergerfs to make them appear as one larger drive.

$ sudo mergerfs -o defaults,allow_other,use_ino,category.create=mfs,moveonenospc=true,minfreespace=1M /disk1:/disk2 /media

The mergerfs man page is quite extensive and complex so we’ll break down the options that were specified.

  • defaults: This will use the default settings unless specified.
  • allow_other: allows users besides sudo or root to see the filesystem.
  • use_ino: Causes mergerfs to supply file/directory inodes rather than libfuse. While not a default it is recommended it be enabled so that linked files share the same inode value.
  • category.create=mfs: Spreads files out across your drives based on available space.
  • moveonenospc=true: If enabled, if writing fails, a scan will be done looking for the drive with the most free space.
  • minfreespace=1M: The minimum space value used.
  • disk1: First hard drive.
  • disk2: Second hard drive.
  • /media: The directory folder where the drives are mounted.

Here is what it looks like:

$ df -hT | grep disk /dev/sdb1 ext4 23M 386K 21M 2% /disk1 /dev/sdc1 ext4 44M 1.1M 40M 3% /disk2 $ df -hT | grep media 1:2 fuse.mergerfs 66M 1.4M 60M 3% /media 

You can see that the mergerfs mount now shows a total capacity of 66M which is the combined total of the two hard drives.

Continuing with the example:

There is a 30Mb video called Baby’s second Xmas.mkv. Let’s copy it to the /media folder which is the mergerfs mount.

$ ls -lh "Baby's second Xmas.mkv"
-rw-rw-r--. 1 curt curt 30M Apr 20 08:45 Baby's second Xmas.mkv
$ cp "Baby's second Xmas.mkv" /media/Videos/

Here is the end result:

$ df -hT | grep disk
/dev/sdb1 ext4 23M 386K 21M 2% /disk1
/dev/sdc1 ext4 44M 31M 9.8M 76% /disk2 $ df -hT | grep media
1:2 fuse.mergerfs 66M 31M 30M 51% /media

You can see from the disk space utilization that mergerfs automatically copied the file to disk2 because disk1 did not have enough free space.

Here is a breakdown of all of the files:

$ ls -l /disk1/Videos/
total 1
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv $ ls -l /disk2/Videos/
total 30003
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 30720000 Apr 20 08:47 Baby's second Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv $ ls -l /media/Videos/
total 30004
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 30720000 Apr 20 08:47 Baby's second Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv

When you copy files to your mergerfs mount, it will always copy the files to the hard disk that has enough free space. If none of the drives in the pool have enough free space, then you won’t be able to copy them.

Posted on Leave a comment

Upgrading Fedora 31 to Fedora 32

Fedora 32 is available now. You’ll likely want to upgrade your system to get the latest features available in Fedora. Fedora Workstation has a graphical upgrade method. Alternatively, Fedora offers a command-line method for upgrading Fedora 30 to Fedora 31.

Before upgrading, visit the wiki page of common Fedora 32 bugs to see if there’s an issue that might affect your upgrade. Although the Fedora community tries to ensure upgrades work well, there’s no way to guarantee this for every combination of hardware and software that users might have.

Upgrading Fedora 31 Workstation to Fedora 32

Soon after release time, a notification appears to tell you an upgrade is available. You can click the notification to launch the GNOME Software app. Or you can choose Software from GNOME Shell.

Choose the Updates tab in GNOME Software and you should see a screen informing you that Fedora 32 is Now Available.

If you don’t see anything on this screen, try using the reload button at the top left. It may take some time after release for all systems to be able to see an upgrade available.

Choose Download to fetch the upgrade packages. You can continue working until you reach a stopping point, and the download is complete. Then use GNOME Software to restart your system and apply the upgrade. Upgrading takes time, so you may want to grab a coffee and come back to the system later.

Using the command line

If you’ve upgraded from past Fedora releases, you are likely familiar with the dnf upgrade plugin. This method is the recommended and supported way to upgrade from Fedora 31 to Fedora 32. Using this plugin will make your upgrade to Fedora 32 simple and easy.

1. Update software and back up your system

Before you do start the upgrade process, make sure you have the latest software for Fedora 31. This is particularly important if you have modular software installed; the latest versions of dnf and GNOME Software include improvements to the upgrade process for some modular streams. To update your software, use GNOME Software or enter the following command in a terminal.

sudo dnf upgrade --refresh

Additionally, make sure you back up your system before proceeding. For help with taking a backup, see the backup series on the Fedora Magazine.

2. Install the DNF plugin

Next, open a terminal and type the following command to install the plugin:

sudo dnf install dnf-plugin-system-upgrade

3. Start the update with DNF

Now that your system is up-to-date, backed up, and you have the DNF plugin installed, you can begin the upgrade by using the following command in a terminal:

sudo dnf system-upgrade download --releasever=32

This command will begin downloading all of the upgrades for your machine locally to prepare for the upgrade. If you have issues when upgrading because of packages without updates, broken dependencies, or retired packages, add the ‐‐allowerasing flag when typing the above command. This will allow DNF to remove packages that may be blocking your system upgrade.

4. Reboot and upgrade

Once the previous command finishes downloading all of the upgrades, your system will be ready for rebooting. To boot your system into the upgrade process, type the following command in a terminal:

sudo dnf system-upgrade reboot

Your system will restart after this. Many releases ago, the fedup tool would create a new option on the kernel selection / boot screen. With the dnf-plugin-system-upgrade package, your system reboots into the current kernel installed for Fedora 31; this is normal. Shortly after the kernel selection screen, your system begins the upgrade process.

Now might be a good time for a coffee break! Once it finishes, your system will restart and you’ll be able to log in to your newly upgraded Fedora 32 system.

Upgrading Fedora: Upgrade complete!

Resolving upgrade problems

On occasion, there may be unexpected issues when you upgrade your system. If you experience any issues, please visit the DNF system upgrade quick docs for more information on troubleshooting.

If you are having issues upgrading and have third-party repositories installed on your system, you may need to disable these repositories while you are upgrading. For support with repositories not provided by Fedora, please contact the providers of the repositories.

Posted on Leave a comment

Fedora 32 is officially here!

It’s here! We’re proud to announce the release of Fedora 32. Thanks to the hard work of thousands of Fedora community members and contributors, we’re celebrating yet another on-time release.

If you just want to get to the bits without delay, head over to https://getfedora.org/ right now. For details, read on!

All of Fedora’s Flavors

Fedora Editions are targeted outputs geared toward specific “showcase” uses.

Fedora Workstation focuses on the desktop. In particular, it’s geared toward software developers who want a “just works” Linux operating system experience. This release features GNOME 3.36, which has plenty of great improvements as usual. My favorite is the new lock screen!

Fedora Server brings the latest in cutting-edge open source server software to systems administrators in an easy-to-deploy fashion. For edge computing use cases, Fedora IoT provides a strong foundation for IoT ecosystems.

Fedora CoreOS is an emerging Fedora Edition. It’s an automatically-updating, minimal operating system for running containerized workloads securely and at scale. It offers several update streams that can be followed for automatic updates that occur roughly every two weeks. Currently the next stream is based on Fedora 32, with the testing and stable streams to follow. You can find information about released artifacts that follow the next stream from the download page and information about how to use those artifacts in the Fedora CoreOS Documentation.

Of course, we produce more than just the editions. Fedora Spins and Labs target a variety of audiences and use cases, including the Fedora Astronomy Lab, which brings a complete open source toolchain to both amateur and professional astronomers, and desktop environments like KDE Plasma and Xfce. New in Fedora 32 is the Comp Neuro Lab, developed by our Neuroscience Special Interest Group to enable computational neuroscience.

And, don’t forget our alternate architectures: ARM AArch64, Power, and S390x. Of particular note, we have improved support for Pine64 devices, NVidia Jetson 64 bit platforms, and the Rockchip system-on-a-chip devices including the Rock960, RockPro64, and Rock64.

General improvements

No matter what variant of Fedora you use, you’re getting the latest the open source world has to offer. Following our “First” foundation, we’ve updated key programming language and system library packages, including GCC 10, Ruby 2.7, and Python 3.8. Of course, with Python 2 past end-of-life, we’ve removed most Python 2 packages from Fedora. A legacy python27 package is provided for developers and users who still need it. In Fedora Workstation, we’ve enabled the EarlyOOM service by default to improve the user experience in low-memory situations.

We’re excited for you to try out the new release! Go to https://getfedora.org/ and download it now. Or if you’re already running a Fedora operating system, follow the easy upgrade instructions. For more information on the new features in Fedora 32, see the release notes.

In the unlikely event of a problem….

If you run into a problem, check out the Fedora 32 Common Bugs page, and if you have questions, visit our Ask Fedora user-support platform.

Thank you everyone

Thanks to the thousands of people who contributed to the Fedora Project in this release cycle, and especially to those of you who worked extra hard to make this another on-time release during a pandemic. Fedora is a community, and it’s great to see how much we’ve supported each other. I invite you to join us in the Red Hat Summit Virtual Experience 28-29 April to learn more about Fedora and other communities.

Edited 1800 UTC on 28 April to add a link to the release notes.

Posted on Leave a comment

What’s new in Fedora 32 Workstation

Fedora 32 Workstation is the latest release of our free, leading-edge operating system. You can download it from the official website here right now. There are several new and noteworthy changes in Fedora 32 Workstation. Read more details below.

GNOME 3.36

Fedora 32 Workstation includes the latest release of GNOME Desktop Environment for users of all types. GNOME 3.36 in Fedora 32 Workstation includes many updates and improvements, including:

Redesigned Lock Screen

The lock screen in Fedora 32 is a totally new experience. The new design removes the “window shade” metaphor used in previous releases, and focuses on ease and speed of use.

Unlock screen in Fedora 32

New Extensions Application

Fedora 32 features the new Extensions application, to easily manage your GNOME Extensions. In the past, extensions were installed, configured, and enabled using either the Software application and / or the Tweak Tool.

The new Extensions application in Fedora 32

Note that the Extensions application is not installed by default on Fedora 32. To either use the Software application to search and install, or use the following command in the terminal:

sudo dnf install gnome-extensions-app

Reorganized Settings

Eagle-eyed Fedora users will notice that the Settings application has been re-organized. The structure of the settings categories is a lot flatter, resulting in more settings being visible at a glance.

Additionally, the About category now has a more information about your system, including which windowing system you are running (e.g. Wayland)

The reorganized settings application in Fedora 32

Redesigned Notifications / Calendar popover

The Notifications / Calendar popover — toggled by clicking on the Date and Time at the top of your desktop — has had numerous small style tweaks. Additionally, the popover now has a Do Not Disturb switch to quickly disable all notifications. This quick access is useful when presenting your screen, and not wanting your personal notifications appearing.

The new Notification / Calendar popover in Fedora 32

Redesigned Clocks Application

The Clocks application is totally redesigned in Fedora 32. It features a design that works better on smaller windows.

The Clocks application in Fedora 32

GNOME 3.36 also provides many additional features and enhancements. Check out the GNOME 3.36 Release Notes for further information


Improved Out of Memory handling

Previously, if a system encountered a low-memory situation, it may have encountered heavy swap usage (aka swap thrashing)– sometimes resulting in the Workstation UI slowing down, or becoming unresponsive for periods of time. Fedora 32 Workstation now ships and enables EarlyOOM by default. EarlyOOM enables users to more quickly recover and regain control over their system in low-memory situations with heavy swap usage.