Tag: Fedora Project community

As part of the 8.0 pre-release announcement, the OpenSSH project stated that they consider the scp protocol outdated, inflexible, and not readily fixed. They then go on to recommend the use of sftp or rsync for file transfer instead.

Many users grew up on the scp command, however, and so are not familiar with rsync. Additionally, rsync can do much more than just copy files, which can give a beginner the impression that it’s complicated and opaque. Especially when broadly the scp flags map directly to the cp flags while the rsync flags do not.

This article will provide an introduction and transition guide for anyone familiar with scp. Let’s jump into the most common scenarios: Copying Files and Copying Directories.

Copying files

For copying a single file, the scp and rsync commands are effectively equivalent. Let’s say you need to ship foo.txt to your home directory on a server named server.

$ scp foo.txt me@server:/home/me/

The equivalent rsync command requires only that you type rsync instead of scp:

$ rsync foo.txt me@server:/home/me/

Copying directories

For copying directories, things do diverge quite a bit and probably explains why rsync is seen as more complex than scp. If you want to copy the directory bar to server the corresponding scp command looks exactly like the cp command except for specifying ssh information:

$ scp -r bar/ me@server:/home/me/

With rsync, there are more considerations, as it’s a more powerful tool. First, let’s look at the simplest form:

$ rsync -r bar/ me@server:/home/me/

Looks simple right? For the simple case of a directory that contains only directories and regular files, this will work. However, rsync cares a lot about sending files exactly as they are on the host system. Let’s create a slightly more complex, but not uncommon, example.

# Create a multi-level directory structure
$ mkdir -p bar/baz
# Create a file at the root directory
$ touch bar/foo.txt
# Now create a symlink which points back up to this file
$ cd bar/baz
$ ln -s ../foo.txt link.txt
# Return to our original location
$ cd -

We now have a directory tree that looks like the following:

bar
├── baz
│   └── link.txt -> ../foo.txt
└── foo.txt 1 directory, 2 files

If we try the commands from above to copy bar, we’ll notice very different (and surprising) results. First, let’s give scp a go:

$ scp -r bar/ me@server:/home/me/

If you ssh into your server and look at the directory tree of bar you’ll notice an important and subtle difference from your host system:

bar
├── baz
│   └── link.txt
└── foo.txt 1 directory, 2 files

Note that link.txt is no longer a symlink. It is now a full-blown copy of foo.txt. This might be surprising behavior if you’re used to cp. If you did try to copy the bar directory using cp -r, you would get a new directory with the exact symlinks that bar had. Now if we try the same rsync command from before we’ll get a warning:

$ rsync -r bar/ me@server:/home/me/
skipping non-regular file "bar/baz/link.txt"

Rsync has warned us that it found a non-regular file and is skipping it. Because you didn’t tell it to copy symlinks, it’s ignoring them. Rsync has an extensive manual section titled “SYMBOLIC LINKS” that explains all of the possible behavior options available to you. For our example, we need to add the –links flag.

$ rsync -r --links bar/ me@server:/home/me/

On the remote server we see that the symlink was copied over as a symlink. Note that this is different from how scp copied the symlink.

bar/
├── baz
│   └── link.txt -> ../foo.txt
└── foo.txt 1 directory, 2 files

To save some typing and take advantage of more file-preserving options, use the –archive (-a for short) flag whenever copying a directory. The archive flag will do what most people expect as it enables recursive copy, symlink copy, and many other options.

$ rsync -a bar/ me@server:/home/me/

The rsync man page has in-depth explanations of what the archive flag enables if you’re curious.

Caveats

There is one caveat, however, to using rsync. It’s much easier to specify a non-standard ssh port with scp than with rsync. If server was using port 8022 SSH connections, for instance, then those commands would look like this:

$ scp -P 8022 foo.txt me@server:/home/me/

With rsync, you have to specify the “remote shell” command to use. This defaults to ssh. You do so using the -e flag.

$ rsync -e 'ssh -p 8022' foo.txt me@server:/home/me/

Rsync does use your ssh config; however, so if you are connecting to this server frequently, you can add the following snippet to your ~/.ssh/config file. Then you no longer need to specify the port for the rsync or ssh commands!

Host server Port 8022

Alternatively, if every server you connect to runs on the same non-standard port, you can configure the RSYNC_RSH environment variable.

Why else should you switch to rsync?

Now that we’ve covered the everyday use cases and caveats for switching from scp to rsync, let’s take some time to explore why you probably want to use rsync on its own merits. Many people have made the switch to rsync long before now on these merits alone.

In-flight compression

If you have a slow or otherwise limited network connection between you and your server, rsync can spend more CPU cycles to save network bandwidth. It does this by compressing data before sending it. Compression can be enabled with the -z flag.

Delta transfers

Rsync also only copies a file if the target file is different than the source file. This works recursively through directories. For instance, if you took our final bar example above and re-ran that rsync command multiple times, it would do no work after the initial transfer. Using rsync even for local copies is worth it if you know you will repeat them, such as backing up to a USB drive, for this feature alone as it can save a lot of time with large data sets.

Syncing

As the name implies, rsync can do more than just copy data. So far, we’ve only demonstrated how to copy files with rsync. If you instead want rsync to make the target directory look like your source directory, you can add the –delete flag to rsync. The delete flag makes it so rsync will copy files from the source directory which don’t exist on the target directory. Then it will remove files on the target directory which do not exist in the source directory. The result is the target directory is identical to the source directory. By contrast, scp will only ever add files to the target directory.

Conclusion

For simple use cases, rsync is not significantly more complicated than the venerable scp tool. The only significant difference being the use of -a instead of -r for recursive copying of directories. However, as we saw rsync’s -a flag behaves more like cp’s -r flag than scp’s -r flag does.

Hopefully, with these new commands, you can speed up your file transfer workflow!

If you run a server with a public-facing SSH access, you might have experienced malicious login attempts. This article shows how to use two utilities to keep the intruder out of our systems.

To protect against repeated ssh login attempts, we’ll look at fail2ban. And if you don’t travel much, and perhaps stay in one or two countries, you can configure firewalld to only allow access from the countries you choose.

First let’s work through a little terminology for those not familiar with the various applications we’ll need to make this work:

fail2ban: Daemon to ban hosts that cause multiple authentication errors.

fail2ban will monitor the SystemD journal to look for failed authentication attempts for whichever jails have been enabled. After the number of failed attempts specified it will add a firewall rule to block that specific IP address for an amount of time configured.

firewalld: A firewall daemon with D-Bus interface providing a dynamic firewall.

Unless you’ve manually decided to use traditional iptables, you’re already using firewalld on all supported releases of Fedora and CentOS.

Assumptions

• The host system has an internet connection and is either fully exposed directly, through a DMZ (both REALLY bad ideas unless you know what you’re doing), or has a port being forwarded to it from a router.
• While most of this might apply to other systems, this article assumes a current version of Fedora (31 and up) or RHEL/CentOS 8. On CentOS you must enable the Fedora EPEL repo with
  sudo dnf install epel-release

Install & Configuration

Fail2Ban

More than likely whichever FirewallD zone is set already allows SSH access but the sshd service itself is not enabled by default. To start it manually and without permanently enabling on boot:

$ sudo systemctl start sshd

Or to start and enable on boot:

$ sudo systemctl enable --now sshd

The next step is to install, configure, and enable fail2ban. As usual the install can be done from the command line:

$ sudo dnf install fail2ban

Once installed the next step is to configure a jail (a service you want to monitor and ban at whatever thresholds you’ve set). By default IPs are banned for 1 hour (which is not near long enough). The best practice is to override the system defaults using *.local files instead of directly modifying the *.config files. If we look at my jail.local we see:

# cat /etc/fail2ban/jail.local
[DEFAULT] # "bantime" is the number of seconds that a host is banned.
bantime = 1d # A host is banned if it has generated "maxretry" during the last "findtime"
findtime = 1h # "maxretry" is the number of failures before a host get banned.
maxretry = 5

Turning this into plain language, after 5 attempts within the last hour the IP will be blocked for 1 day. There’s also options for increasing the ban time for IPs that get banned multiple times, but that’s the subject for another article.

The next step is to configure a jail. In this tutorial sshd is shown but the steps are more or less the same for other services. Create a configuration file inside /etc/fail2ban/jail.d. Here’s mine:

# cat /etc/fail2ban/jail.d/sshd.local
[sshd]
enabled = true

It’s that simple! A lot of the configuration is already handled within the package built for Fedora (Hint: I’m the current maintainer). Next enable and start the fail2ban service.

$ sudo systemctl enable --now fail2ban

Hopefully there were not any immediate errors, if not, check the status of fail2ban using the following command:

$ sudo systemctl status fail2ban

If it started without errors it should look something like this:

$ systemctl status fail2ban
● fail2ban.service - Fail2Ban Service
Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2020-06-16 07:57:40 CDT; 5s ago
Docs: man:fail2ban(1)
Process: 11230 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
Main PID: 11235 (f2b/server)
Tasks: 5 (limit: 4630)
Memory: 12.7M
CPU: 109ms
CGroup: /system.slice/fail2ban.service
└─11235 /usr/bin/python3 -s /usr/bin/fail2ban-server -xf start
Jun 16 07:57:40 localhost.localdomain systemd[1]: Starting Fail2Ban Service…
Jun 16 07:57:40 localhost.localdomain systemd[1]: Started Fail2Ban Service.
Jun 16 07:57:41 localhost.localdomain fail2ban-server[11235]: Server ready

If recently started, fail2ban is unlikely to show anything interesting going on just yet but to check the status of fail2ban and make sure the jail is enabled enter:

$ sudo fail2ban-client status
Status
|- Number of jail:	1
`- Jail list:	sshd

And the high level status of the sshd jail is shown. If multiple jails were enabled they would show up here.

To check the detailed status a jail, just add the jail to the previous command. Here’s the output from my system which has been running for a while. I have removed the banned IPs from the output:

$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed:	8
| |- Total failed:	4399
| `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions |- Currently banned:	101 |- Total banned:	684 `- Banned IP list: ...

Monitoring the fail2ban log file for intrusion attempts can be achieved by “tailing” the log:

$ sudo tail -f /var/log/fail2ban.log

Tail is a nice little command line utility which by default shows the last 10 lines of a file. Adding the “-f” tells it to follow the file which is a great way to watch a file that’s still being written to.

Since the output has real IPs in it, a sample won’t be provided but it’s pretty human readable. The INFO lines will usually be attempts at a login. If enough attempts are made from a specific IP address you will see a NOTICE line showing an IP address was banned. After the ban time has been reached you will see an NOTICE unban line.

Lookout for several WARNING lines. Most often this happens when a ban is added but fail2ban finds the IP address already in its ban database, which means banning may not be working correctly. If recently installed the fail2ban package it should be setup for FirewallD rich rules. The package was only switched from “ipset” to “rich rules” as of fail2ban-0.11.1-6 so if you have an older install of fail2ban it may still be trying to use the ipset method which utilizes legacy iptables and is not very reliable.

FirewallD Configuration

Reactive or Proactive?

There are two strategies that can be used either separately or together. Reactive or proactive permanent blacklisting of individual IP address or subnets based on country of origin.

For the reactive approach once fail2ban has been running for a while it’s a good idea to take a look at how “bad is bad” by running sudo fail2ban-client status sshd again. There most likely will be many banned IP addresses. Just pick one and try running whois on it. There can be quite a bit of interesting information in the output but for this method, only the country of origin is of importance. To keep things simple, let’s filter out everything but the country.

For this example a few well known domain names will be used:

$ whois google.com | grep -i country
Registrant Country: US
Admin Country: US
Tech Country: US

$ whois rpmfusion.org | grep -i country
Registrant Country: FR

$ whois aliexpress.com | grep -i country
Registrant Country: CN

The reason for the grep -i is to make grep non-case sensitive while most entries use “Country”, some are in all lower case so this method matches regardless.

Now that the country of origin of an intrusion attempt is known the question is, “Does anyone from that country have a legitimate reason to connect to this computer?” If the answer is NO, then it should be acceptable to block the entire country.

Functionally the proactive approach it not very different from the reactive approach, however, there are countries from which intrusion attempts are very common. If the system neither resides in one of those countries, nor has any customers originating from them, then why not add them to the blacklist now rather than waiting?

Blacklisting Script and Configuration

So how do you do that? With FirewallD ipsets. I developed the following script to automate the process as much as possible:

#!/bin/bash
# Based on the below article
# https://www.linode.com/community/questions/11143/top-tip-firewalld-and-ipset-country-blacklist # Source the blacklisted countries from the configuration file
. /etc/blacklist-by-country # Create a temporary working directory
ipdeny_tmp_dir=$(mktemp -d -t blacklist-XXXXXXXXXX)
pushd $ipdeny_tmp_dir # Download the latest network addresses by country file
curl -LO http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz
tar xf all-zones.tar.gz # For updates, remove the ipset blacklist and recreate
if firewall-cmd -q --zone=drop --query-source=ipset:blacklist; then firewall-cmd -q --permanent --delete-ipset=blacklist
fi # Create the ipset blacklist which accepts both IP addresses and networks
firewall-cmd -q --permanent --new-ipset=blacklist --type=hash:net \ --option=family=inet --option=hashsize=4096 --option=maxelem=200000 \ --set-description="An ipset list of networks or ips to be dropped." # Add the address ranges by country per ipdeny.com to the blacklist
for country in $countries; do firewall-cmd -q --permanent --ipset=blacklist \ --add-entries-from-file=./$country.zone && \ echo "Added $country to blacklist ipset."
done # Block individual IPs if the configuration file exists and is not empty
if [ -s "/etc/blacklist-by-ip" ]; then echo "Adding IPs blacklists." firewall-cmd -q --permanent --ipset=blacklist \ --add-entries-from-file=/etc/blacklist-by-ip && \ echo "Added IPs to blacklist ipset."
fi # Add the blacklist ipset to the drop zone if not already setup
if firewall-cmd -q --zone=drop --query-source=ipset:blacklist; then echo "Blacklist already in firewalld drop zone."
else echo "Adding ipset blacklist to firewalld drop zone." firewall-cmd --permanent --zone=drop --add-source=ipset:blacklist
fi firewall-cmd -q --reload popd
rm -rf $ipdeny_tmp_dir

This should be installed to /usr/local/sbin and don’t forget to make it executable!

$ sudo chmod +x /usr/local/sbin/firewalld-blacklist

Then create a configure file: /etc/blacklist-by-country:

# Which countries should be blocked?
# Use the two letter designation separated by a space.
countries=""

And another configuration file /etc/blacklist-by-ip, which is just one IP per line without any additional formatting.

For this example 10 random countries were selected from the ipdeny zones:

# ls | shuf -n 10 | sed "s/\.zone//g" | tr '\n' ' '
nl ee ie pk is sv na om gp bn

Now as long as at least one country has been added to the config file it’s ready to run!

$ sudo firewalld-blacklist % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed
100 142 100 142 0 0 1014 0 --:--:-- --:--:-- --:--:-- 1014
100 662k 100 662k 0 0 989k 0 --:--:-- --:--:-- --:--:-- 989k
Added nl to blacklist ipset.
Added ee to blacklist ipset.
Added ie to blacklist ipset.
Added pk to blacklist ipset.
Added is to blacklist ipset.
Added sv to blacklist ipset.
Added na to blacklist ipset.
Added om to blacklist ipset.
Added gp to blacklist ipset.
Added bn to blacklist ipset.
Adding ipset blacklist to firewalld drop zone.
success

To verify that the firewalld blacklist was successful, check the drop zone and blacklist ipset:

$ sudo firewall-cmd --info-zone=drop
drop (active) target: DROP icmp-block-inversion: no interfaces: sources: ipset:blacklist services: ports: protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich rules: $ sudo firewall-cmd --info-ipset=blacklist | less
blacklist type: hash:net options: family=inet hashsize=4096 maxelem=200000 entries: 

The second command will output all of the subnets that were added based on the countries blocked and can be quite lengthy.

So now what do I do?

While it will be a good idea to monitor things more frequently at the beginning, over time the number of intrusion attempts should decline as the blacklist grows. Then the goal should be maintenance rather than active monitoring.

To this end I created a SystemD service file and timer so that on a monthly basis the by country subnets maintained by ipdeny are refreshed. In fact everything discussed here can be downloaded from my pagure.io project:

https://pagure.io/firewalld-blacklist

Aren’t you glad you read the whole article? Now just download the service file and timer to /etc/systemd/system/ and enable the timer:

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now firewalld-blacklist.timer

The Fedora Silverblue project takes Fedora workstation, libostree and podman, puts them in a blender, and creates a new Immutable Fedora Workstation. Fedora Silverblue is an OS that stops you from changing the core system files arbitrarily, and readily allows you to change the environment system files. The article What is Silverblue describes the big picture, and this article drills down into details for the developer.

Fedora Silverblue ties together a few different projects to make a system that is a git-like object, capable of layering packages, and has a container focused work flow. Silverblue is not the only distribution going down this road. It is the desktop equivalent of CoreOS, the server OS used by Red Hat Openshift.

Silverblue’s idea of ‘immutable’ has nothing to do with immutable layers in a container. Silverblue keeps system files immutable by making them read-only.

Why immutable?

Has an upgrade left your system in an unusable state? Have you wondered why one server in a pool of identical machines is being weird? These problems can happen when one system library – one tiny little file out of hundreds – is corrupted, badly configured or the wrong version. Or maybe your upgrade works fine but it’s not what you’d hoped for, and you want to roll back to the previous state.

An immutable OS is intended to stop problems like these biting you. This is not an easy thing to achieve – simple changes, like flipping the file system between read-write and read-only, may only change a fault-finding headache to a maintenance headache.

Freezing the system is good news for sysadmins, but what about developers? Setting up a development environment means heavily customizing the system, and filling it with living code that changes over time. The answer is partly a case of combining components, and partly the ability to swap between OS versions.

How it works

So how do you get the benefits of immutability without losing the ability to do your work? If you’re thinking ‘containers’, good guess – part of the solution uses podman. But much of the work happens underneath the container layer, at the OS level.

Fedora Silverblue ties together a few different projects to turn an immutable OS into a usable workstation. Silverblue uses libostree to provide the base system, lets you edit config files in /etc/, and provides three different ways to install packages.

• rpm-ostree installs RPM packages, similar to DNF in the traditional Fedora workstation. Use this for things that shouldn’t go in containers, like KVM/libvirt.
• flatpak installs packages from a central flathub repo. This is the one-stop shop for graphical desktop apps like LibreOffice.
• The traditional dnf install still works, but only inside a toolbox (a Fedora container). A developer’s workbench goes in a toolbox.

If you want to know more about these components, check out Pieces of Silverblue.

Rolling back and pinning upgrades

All operating systems need upgrades. Features are added, security holes are plugged and bugs are squashed. But sometimes an upgrade is not a developer’s friend.

A developer depends on many things to get the job done. A good development environment is stuffed with libraries, editors, toolchains and apps that are controlled by the OS, not the developer. An upgrade may cause trouble. Have any of these situations happened to you?

• A new encryption library is too strict, and an upgrade stopped an API working.
• Code works well, but has deprecated syntax. An upgrade brought error-throwing misery.
• The development environment is lovingly hand-crafted. An upgrade broke dependencies and added conflicts.

In a traditional environment, unpicking a troublesome upgrade is hard. In Silverblue, it’s easy. Silverblue keeps two copies of the OS – your current upgrade and your previous version. Point the OS at the previous version, reboot, and you’ve got your old system files back.

You aren’t limited to two copies of your file system – you can keep more by pinning your favorite versions. Dusty Mabe, one of the engineers who has been working on the system since the Project Atomic days, describes how to pin extra copies of the OS in his article Pinning Deployments in OSTree Based Systems.

Your home directory is not affected by rolling back. Rpm-ostree does not touch /etc/ and /var/.

System updates and package installs

Silverblue’s rpm-ostree treats all the files as one object, stored in a repository. The working file system is a checked-out copy of this object. After a system update, you get two objects in that repository – one current object and one updated object. The updated object is checked out and becomes the new file system.

You install your workhorse applications in toolboxes, which provide container isolation. And you install your desktop applications using Flatpak.

This new OS requires a shift in approach. For instance, you don’t have to keep only one copy of your system files – you can store a few and select which one you use. That means you can swap back and forth between an old Fedora release and the rawhide (development) version in a matter of minutes.

Build your own Silverblue VM

You can safely install Fedora Silverblue in a VM on your workstation. If you’ve got a hypervisor and half an hour to spare (10 minutes for ISO download, and 20 minutes for the build), you can see for yourself.

1. Download Fedora Silverblue ISO from
2. https://silverblue.fedoraproject.org/download (not Fedora workstation from https://getfedora.org/).
3. Boot a VM with the Fedora Silverblue ISO. You can squeeze Fedora into compute resources of 1 CPU, 1024MiB of memory and 12GiB of storage, but bigger is better.
4. Answer Anaconda’s questions.
5. Wait for the Gnome desktop to appear.
6. Answer Initial Setup’s questions.

Then you’re ready to set up your developer’s tools. If you’re looking for an IDE, check these out. Use flatpak on the desktop to install them.

• Gnome Builder (Gnome’s official IDE)
• Eclipse
• Code::Blocks

Finally, use the CLI to create your first toolbox. Load it with modules using npm, gem, pip, git or your other favorite tools.

Help!

If you get stuck, ask questions at the forum.

If you’re looking for ideas about how to use Silverblue, read articles in the magazine.

Is Silverblue for you?

Silverblue is full of shiny new tech. That in itself is enough to attract the cool kids, like moths to a flame. But this OS is not for everyone. It’s a young system, so some bugs will still be lurking in there. And pioneering tech requires a change of habit – that’s extra cognitive load that the new user may not want to take on.

The OS brings immutable benefits, like keeping your system files safe. It also brings some drawbacks, like the need to reboot after adding system packages. Silverblue also enables new ways of working. If you want to explore new directions in the OS, find out if Silverblue brings benefits to your work.

The Fedora Classroom is a project to help people by spreading knowledge on subjects related to Fedora for others, If you would like to propose a session, feel free to open a ticket here with the tag “classroom.” If you’re interested in taking a proposed session, kindly let us know and once you take it, you will be awarded the Sensei Badge too as a token of appreciation. Recordings from the previous sessions can be found here.

We’re back with another awesome classroom on IRC 101 led by Pac23.

About the Session: A Beginners Guide to Internet Relay Chat

In short, the IRC 101 session will be a guide for newcomers on how to get started with IRC with the Fedora community and hang out with other contributors in IRC. After finishing the session you will have the knowledge to setup your IRC client and start communicating with other Fedora people.

When and where

The Classroom session will be organized on May 9th, 16:00 UTC. Here’s a link to see what time it is in your timezone. The session will be streamed on Fedora Project’s YouTube channel.

Topics covered in the session

• Why IRC & How it works?
• How to install an IRC Client.
• Registering your nick in IRC
• Some basic commands, modes & access controls
• Joining fedora channels
• Brownie Topic: Fedora bots in IRC.

About the instructor

Pac23’s been around in the Fedora community and contributing to the project for around a year. He’s started with volunteering to package a custom kernel. He’s also a Computer Engineering undergrad at the University of Mumbai. His interests mostly reside in DevOps, IoT & system design. Outside computer science, he loves traveling, airplanes and history. He can be found as pac23 in IRC channels including #fedora-neuro, #fedora-devel, and #fedora-kernel.

If you miss the session, no worries. The recording will also be uploaded in the Fedora Project‘s YouTube channel.

We hope you can attend and enjoy this experience from some of the awesome people that work in Fedora Project. We look forward to seeing you in the Classroom session.

Photograph used in feature image is San Simeon School House by Anita Ritenour — CC-BY 2.0.