Category: Linux, FreeBSD, and Unix types

If you’re a developer, then you know your way around development tools. You’ve spent years studying one or more programming languages and have perfected your skills. You can develop with GUI tools or from the command line. On your own, nothing can stop you. You code as if your mind and your fingers are one to create elegant, perfectly commented, source for an app you know will take the world by storm.

But what happens when you’re tasked with collaborating on a project? Or what about when that app you’ve developed becomes bigger than just you? What’s the next step? If you want to successfully collaborate with other developers, you’ll want to make use of a distributed version control system. With such a system, collaborating on a project becomes incredibly efficient and reliable. One such system is Git. Along with Git comes a handy repository called GitHub, where you can house your projects, such that a team can check out and check in code.

I will walk you through the very basics of getting Git up and running and using it with GitHub, so the development on your game-changing app can be taken to the next level. I’ll be demonstrating on Ubuntu 18.04, so if your distribution of choice is different, you’ll only need to modify the Git install commands to suit your distribution’s package manager.

Git and GitHub

The first thing to do is create a free GitHub account. Head over to the GitHub signup page and fill out the necessary information. Once you’ve done that, you’re ready to move on to installing Git (you can actually do these two steps in any order).

Installing Git is simple. Open up a terminal window and issue the command:

sudo apt install git-all

This will include a rather large number of dependencies, but you’ll wind up with everything you need to work with Git and GitHub.

On a side note: I use Git quite a bit to download source for application installation. There are times when a piece of software isn’t available via the built-in package manager. Instead of downloading the source files from a third-party location, I’ll often go the project’s Git page and clone the package like so:

git clone ADDRESS

Where ADDRESS is the URL given on the software’s Git page.
Doing this most always ensures I am installing the latest release of a package.

Create a local repository and add a file

The next step is to create a local repository on your system (we’ll call it newproject and house it in ~/). Open up a terminal window and issue the commands:

cd ~/ mkdir newproject cd newproject

Now we must initialize the repository. In the ~/newproject folder, issue the command git init. When the command completes, you should see that the empty Git repository has been created (Figure 1).

Next we need to add a file to the project. From within the root folder (~/newproject) issue the command:

touch readme.txt

You will now have an empty file in your repository. Issue the command git status to verify that Git is aware of the new file (Figure 2).

Even though Git is aware of the file, it hasn’t actually been added to the project. To do that, issue the command:

git add readme.txt

Once you’ve done that, issue the git status command again to see that readme.txt is now considered a new file in the project (Figure 3).

Your first commit

With the new file in the staging environment, you are now ready to create your first commit. What is a commit? Easy: A commit is a record of the files you’ve changed within the project. Creating the commit is actually quite simple. It is important, however, that you include a descriptive message for the commit. By doing this, you are adding notes about what the commit contains (such as what changes you’ve made to the file). Before we do this, however, we have to inform Git who we are. To do this, issue the command:

git config --global user.email EMAIL git config --global user.name “FULL NAME”

Where EMAIL is your email address and FULL NAME is your name.

Now we can create the commit by issuing the command:

git commit -m “Descriptive Message”

Where Descriptive Message is your message about the changes within the commit. For example, since this is the first commit for the readme.txt file, the commit could be:

git commit -m “First draft of readme.txt file”

You should see output indicating that 1 file has changed and a new mode was created for readme.txt (Figure 4).

Create a branch and push it to GitHub

Branches are important, as they allow you to move between project states. Let’s say you want to create a new feature for your game-changing app. To do that, create a new branch. Once you’ve completed work on the feature you can merge this feature from the branch to the master branch. To create the new branch, issue the command:

git checkout -b BRANCH

where BRANCH is the name of the new branch. Once the command completes, issue the command git branch to see that it has been created (Figure 5).

Next we need to create a repository on GitHub. If you log into your GitHub account, click the New Repository button from your account main page. Fill out the necessary information and click Create repository (Figure 6).

After creating the repository, you will be presented with a URL to use for pushing our local repository. To do this, go back to the terminal window (still within ~/newproject) and issue the commands:

git remote add origin URL git push -u origin master

Where URL is the url for our new GitHub repository.

You will be prompted for your GitHub username and password. Once you successfully authenticate, the project will be pushed to your GitHub repository and you’re ready to go.

Pulling the project

Say your collaborators make changes to the code on the GitHub project and have merged those changes. You will then need to pull the project files to your local machine, so the files you have on your system match those on the remote account. To do this, issue the command (from within ~/newproject):

git pull origin master

The above command will pull down any new or changed files to your local repository.

The very basics

And that is the very basics of using Git from the command line to work with a project stored on GitHub. There is quite a bit more to learn, so I highly recommend you issue the commands man git, man git-push, and man git-pull to get a more in-depth understanding of what the git command can do.

Happy developing!

Learn more about Linux through the free “Introduction to Linux” course from The Linux Foundation and edX.

Linux container technology has been proposed by companies like Resin.io as a simpler and more secure way to deploy embedded devices. And, Daynix Computing has developed an open source framework called Rebuild that uses Linux containers in the build management process of embedded IoT development. At the 2017 Open Source Summit, Daynix “virtualization expert” Yan Vugenfirer gave a presentation on Rebuild called “How Linux Containers can Help to Manage Development Environments for IoT and Embedded Systems.”

Vugenfirer started by reminding the audience of the frustrations of embedded development, especially when working with large, complex projects. “You’re dealing with different toolchains, SDKs, and compilers all with different dependencies,” he said. “It gets more complicated if you need to update packages, or change SDKs, or run a codebase over several devices. The code may compile on your machine, but there may be problems in the build server or in the CI (continuous integration) server.”

Rebuild offers an easier way to manage build environments by leveraging the power of Linux containers, said Vugenfirer. “Rebuild allows seamless integration between the containers and your existing build environment and enables easy sharing of environments between team members. We are mapping the local file system into the container in order to preserve all permissions and ownership when working with source code. The code is stored locally on our file system while we work with different environments.”

Based on Ruby 2.0+ Gem and Docker, Rebuild supports Linux, Windows, and Mac OS environments. It’s available in free and commercial versions.

The software lets you run multiple platforms on same machine and gives you the assurance that it will run the same on the build or CI servers as it does on your development workstation, said Vugenfirer. The developer can choose whether actions performed on the build environment change the original.

Build management designed for teams

The software is particularly useful for sharing environments between team members. It provides a “unified experience,” and offers features like track versioning,” said Vugenfirer. Other benefits include easier technical support, as well as the ability to “reproduce an environment in the future with different developers.” Rebuild also “eases preparation for certification audits for medical or automotive devices by letting you show what libraries are certified,” he added.

Developers control Rebuild via the Rebuild CLI interface, which “acts as a gateway for scripts on CI or build servers,” explained Vugenfirer. Below that in the architecture hierarchy sits Docker, and on the base level is the Environments Registry, which works with Docker to manage and share the container images. Rebuild currently supports DockerHub as the default, as well as Docker private registry and Daynix’s own Rebuild Native Registry.

Developers can use existing scripts to build the software by simply prefacing it with the “rbld run” command. “You don’t need to know about Docker or Docker files,” said Vugenfirer.

Daynix CTO Dmitry Fleytman joined Vugenfirer to give a demo of Rebuild that showed how to search a registry for available environments such as Raspbian, Ubuntu, BeagleBone, and QEMU. He then deployed the Raspbian environment and issued a Run command to compile source code. There’s also an interactive mode alternative to issuing a single run command. In either case, the process “needs to be done only once for each environment version,” said Vugenfirer.

Although you can run an environment without modifying it, you can also do the opposite by using the “rbld modify” command, which lets you update packages. Other commands include “rbld commit” for a local commit and “rbld publish” to share it. You can also revert changes with “rbld checkout.”

Rebuild lets you create a new environment from scratch using the file system or build on a base image from one of the distros in environment repository. After using the “rbld create” command, you can then modify, commit, and publish it.

Future plans for Rebuild include adding role management for the Rebuild Native Registry to assign who can create an environment vs. simply running them. Daynix also plans to add more registries, provide IDE extensions, and offer more tracking of environment usage. They have just begun working on Yocto Project integration.

More information can be found in the conference video below:

[embedded content]

Join us at Open Source Summit + Embedded Linux Conference Europe in Edinburgh, UK on October 22-24, 2018, for 100+ sessions on Linux, Cloud, Containers, AI, Community, and more.

Learn the basics of grep with this tutorial from our archives.

The grep command is a very powerful way to find documents on your computer. You can use grep to see if a file contains a word or use one of many forms of regular expression to search for a pattern instead. Grep can check the file that you specify or can search an entire tree of your filesystem recursively looking for matching files.

One of the most basic ways to use grep is shown below, looking for the lines of a file that match a pattern. I limit the search to only text files in the current directory *.txt and the -i option makes the search case-insensitive. As you can see, the only matches for the string “this” are the capitalized string “This”.

$ cat sample.txt
This is the sample file.
It contains a few lines of text
that we can use to search for things.
Samples of text
and seeking those samples
there can be many matches
but not all of them are fun
so start searching for samples
start looking for text that matches $ grep -i this sample.txt 
This is the sample file.

The -A, -B, and -C options to grep let you see a little bit more context than a single line that matched. These options let you specify the number of trailing, preceding, and both trailing and preceding lines to print, respectively. Matches are shown separated with a “—” line so you can clearly see the context for each match in the presented results. Notice that the last example using -C 1 to grab both the preceding line and trailing line shows four results in the last match. This is because there are two matches (the middle two lines) that share the same context.

$ grep -A 2 It sample.txt 
It contains a few lines of text
that we can use to search for things.
Samples of text $ grep -C 1 -i the sample.txt 
This is the sample file.
It contains a few lines of text
--
and seeking those samples
there can be many matches
but not all of them are fun
so start searching for samples

The -n option can be used to show the line number that is being presented. Below I grab one line before and one line after the match and see the line numbers, too.

$ grep -n -C 1 tha sample.txt 
2-It contains a few lines of text
3:that we can use to search for things.
4-Samples of text
--
8-so start searching for samples
9:start looking for text that matches

Digging through a bunch of files

You can get grep to recurse into a directory using the -R option. When you use this, the matching file name is shown on the output as well as the match itself. When you combine -R with -n the file name is first shown, then the line number, and then the matching line.

 $ grep -R sample .
./subdir/sample3.txt:another sample in a sub directory
./sample.txt:This is the sample file.
./sample.txt:and seeking those samples
./sample.txt:so start searching for samples
./sample2.txt:This is the second sample file $ grep -n -R sample .
./subdir/sample3.txt:1:another sample in a sub directory
...

If you have some subdirectories that you don’t want searched, then the –exclude-dir can tell grep to skip over them. Notice that I have used single quotes around the sub* glob below. The difference can be seen in the last commands where I use echo to show the command itself rather than execute it. Notice that the shell has expanded the sub* into ‘subdir’ for me in the last command. If you have subdir1 and subdir2 and use the pattern sub* then your shell will likely expand that glob into the two directory names, and that will confuse grep which is expecting a single glob. If in doubt, enclose the directory to exclude in single quotes as shown in the first command below.

$ grep -R --exclude-dir 'sub*' sample .
./sample.txt:This is the sample file.
./sample.txt:and seeking those samples
./sample.txt:so start searching for samples
./sample2.txt:This is the second sample file $ echo grep -R --exclude-dir 'sub*' sample .
grep -R --exclude-dir sub* sample . $ echo grep -R --exclude-dir sub* sample .
grep -R --exclude-dir subdir sample .

Although the recursion built into grep is handy, you might like to combine the find and grep commands. It can be useful to use the find command by itself to see what files you will be executing grep on. The find command below uses regular expressions on the file names to limit the files to consider to only those with the number 2 or 3 in their name and only text files. The -type f limits the output to only files.

$ find . -name '*[23]*txt' -type f
./subdir/sample3.txt
./sample2.txt

You then tell find to execute a command for each file that is found instead of just printing the file name using the -exec option to find. It is convenient to use the -H option to grep to print the filename for each match. You may recall that grep will give you -H by default when run on many files. Using -H can be handy in case find only finds a single file; if that file matches, it is good to know what the file name is as well as the matches.

$ find . -name '*[23]*txt' -type f -exec grep -H sampl {} +

For dealing with common file types, like source code, it might be convenient to use a bash alias such as the one below to “Recursively Grep SRC code”. The search is limited to C/C++ source using file name matching. Many possible extensions are chained together using the -o argument to find meaning “OR”. The “$1” argument passed to the grep command takes the first argument to RGSRC and passes it to grep. The last command searches for the string “Ferris” in any C/C++ source code in the current directory or any subdirectory.

$ cat ~/.bashrc
...
RGSRC() {
 find . \( -name "*.hh" -o -name "*.cpp" -o -name "*.hpp" -o -name "*.h" -o -name "*.c" \) \
 -exec grep -H "$1" {} +
}
... $ RGSRC Ferris
...
./Ferris.cpp:using namespace Ferris::RDFCore;
...

Regular Expressions

While I have been searching for a single word using grep in the above, you can define what you want using regular expressions. There is support in grep for basic, extended, and Perl compatible regular expressions. Basic regular expressions are the default.

Regular expressions let you define a pattern for what you are after. For example, the regular expression ‘[Ss]imple’ will match the strings ‘simple’ and ‘Simple’. This is different from using -i to perform a case-insensitive search, because ‘sImple’ will not be considered a match for the above regular expression. Each character inside the square brackets can match, and only one of ‘S’ or ‘s’ is allowed before the remaining string ‘imple’. You can have many characters inside the square brackets and also define the an inversion. For example, [^F]oo will match any character than ‘F’ followed by two lower case ‘o’ characters. If you want to find the ‘[‘ character you have to escape it’s special meaning by preceding it with a backslash.

To match any character use the full stop. If you follow a character or square bracketed match with ‘*’ it will match zero or more times. To match one or more use ‘+’ instead. So ‘[B]*ar’ will match ‘ar’, ‘Bar’, ‘BBar’, ‘BBBar’, and so on. You can also use {n} to match n times and {n,m} to match at least n times but no more than m times. To use the ‘+’ and {n,m} modifiers you will have to enable extended regular expressions using the -E option.

These are some of the more fundamental parts of a regular expression, there are more and you can defined some very sophisticated patterns to find exactly what you are after. The first command below will find sek, seek, seeek in the sample file. The second command will find the strings ‘many’ or ‘matches’ in the file.

$ grep -E 's[e]{1,3}k' sample.txt 
and seeking those samples $ grep -E 'ma(ny|tches)' sample.txt 
there can be many matches
start looking for text that matches

Looking across lines

The grep command works on a line-by-line basis. This means that if you are looking for two words together, then you will have some trouble matching one word at the end of one line and the second word at the start of the next line. So finding the person ‘John Doe’ will work unless the Doe happens to be the first word of the next line.

Although there are other tools, such as awk and Perl, that will allow you to search over multiple lines, you might like to use pcregrep to get the job done. On Fedora, you will have to install the pcre-tools package.

The below command will find the string ‘text that’ with the words separated by any amount of whitespace. In this case, whitespace also includes the newline.

$ pcregrep -M 'text[\s]*that' sample.txt
It contains a few lines of text
that we can use to search for things.
start looking for text that matches

A few other things

Another grep option that might be handy is -m, which limits the number of matches sought in a file. The -v will invert the matches, so you see only the lines which do not match the pattern you gave. An example of an inverted match is shown below.

$ grep -vi sampl sample.txt 
It contains a few lines of text
that we can use to search for things.
there can be many matches
but not all of them are fun
start looking for text that matches

Final words

Using grep with either -R to directly inspect an area of your filesystem or in combination with a complicated find command will let you search through large amounts of text fairly quickly. You will likely find grep already installed on many machines. The pcregrep allows you to search multiple lines fairly easily. Next time, I’ll take a look at some other grep-like commands that let you search PDF documents and XML files.

Learn more about Linux through the free “Introduction to Linux” course from The Linux Foundation and edX.

What Is Machine Learning

I am partial towards this definition by Nvidia:

“Machine Learning at its most basic is the practice of using algorithms to parse data, learn from it, and then make a determination or prediction about something in the world.”

The first step to understanding machine learning is understanding what kinds of problems it intends to solve, based on the foregoing definition. It is principally concerned with mapping data to mathematical models — allowing us to make inferences (predictions) about measurable phenomena in the world. From the machine learning model’s predictions, we can then make rational, informed decisions with increased empirical certainty.

Take, for example, the adaptive brightness on your phone screen. Modern phones have front- and rear-facing cameras that allow the phone to constantly detect the intensity of ambient light, and then adjust the brightness of the screen to make it more pleasant for viewing. But, depending on an individuals taste, they might not like the gradient preselected by the software, and have to constantly fiddle with the brightness by hand. In the end, they turn off adaptive brightness altogether!

What if, instead, the phone employed machine learning software, that registered the intensity of ambient light, and the brightness that the user selected by hand as one example of their preference. Over time, the phone could build a model of an individuals preferences, and then make predictions of how to set the screen brightness on a full continuum of ambient light conditions. (This is a real thing in the next version of Android).

As you can imagine, machine learning could be deployed for all kinds of data relationships susceptible to modeling, which would allow programmers and inventors to increasingly automate decision-making. To be sure, a perfect understand of the methodology of machine learning is to have a fairly deep appreciation for statistical sampling methods, data modeling, linear algebra, and other specialized disciplines.

I will instead try to offer a brief overview of the important terms and concepts in machine learning, and offer an operative (but fictional) example from my own time as a discovery lawyer. In closing, I will discuss a few of the common problems associated with machine learning.

Nuts & Bolts: Key Terms

Feature. A feature represents the the variable that you change in order to observe and measure the outcome. Features are the inputs to a machine learning scheme; in other words, a feature is like the independent variable in a science experience, or the x variable on a line graph.

If you were designing a machine learning model that would classify emails as “important” (e.g., as Gmail does with labels), the model could take more than one feature as input to the model: whether the sender is in your address book (yes or no); whether the email contains a particular phrase, like “urgent”; whether the receiver has previously marked emails from the sender as important, among potential measurable features relating to “importance.”

Features need to be measurable, in that they can be mapped to numeric values for the data model.

Label. The label refers to the variable that is observed or measured in response to the various features in a model. For example, if a model meant to predict college applicant acceptance/rejection based off two features — SAT Score & GPA— the label would indicate yes (1) or no (0) for each example fed into the model. A label is like the dependent variable in a science experiment, or the Y variable on a line graph.

Example. An example is one data entry (like a line on an excel spreadsheet) that includes all features and their associated values. An example can be labeled (includes the label, i.e., the Y variable, with it’s value), or unlabeled (the value of the Y variable is unknown).

Training: Broadly, training is the act of feeding a mathematical model with labeled examples, so that the model can infer and make predictions for unlabeled examples.

Loss. Loss is the difference between the models prediction and the label on a single example. Statistical models aim to reduce loss as much as possible. For example, if you are to fit a line through a cloud of data points to show the linear growth on the Y-axis as X varies, a model would want a line that fits through all the points such that the sum of every loss is minimized as much as possible. Humans can do this intuitively, but computers can be automated to try different slopes until it arrives at the best mathematical answer.

Generalization/Overfitting: Overfitting is an outcome in which a model does not accurately predict testing data (i.e., doesnt “generalize” well) because the model tries to fit the training data too precisely. These problems can occur from poor sampling.

Types of Models

Regression. A regression model is a model that tries to predict a value along some continuum. For example, a model might try to predict the number of people that will move to California; or the probability that a person will get a dog; or the resell price of a used bicycle on craigslist.

Classification. A classification model is a model that predicts discrete outcomes — in a sense, it sorts inputs into various buckets. For example, a model might look at an image and determine if it is a donut, or not a donut.

Model Design: Linear, or…?

Conceptually, the simplest models are those in which the label can be predicted with a line (i.e., are linear). As you can imagine, some distributions cannot naturally be mapped along a continuous line, and therefore you need other mathematical tools to fit a model to the data. One simple machine learning tool to deal with nonlinear classification problems is feature crosses, which is merely adding a new feature that is the cross product (multiplication) of other existing features.

On the more complex side, models can rely on “neural networks”, so called because they mirror the architecture of neurons in humans cognitive architecture, to model complex linear and non-linear relationships. These networks consists of stacked layers of nodes, each represented a weighted sum with some bias from various input features (with potentially a non-linear layer added in), that ultimately yields an output after the series of transformations is complete. Now onto a (simpler) real life example.

Real Life Example: An Attorney’s Perspective

Perhaps only 20+ years ago, the majority of business records (documents) were kept on paper, and it was the responsibility of junior attorneys to sift through literal reams of paper (most of it irrelevant) to find the proverbial “smoking gun” evidence in all manner of cases. In 2018, nearly all business records are electronic. The stage of litigation in which documents are produced, exchanged, and examined is called “discovery.”

As most business records are now electronic, the process of discovery is now aided and facilitated by the use of computers. This is not a fringe issue for a businesses. Every industry is subject to one or more federal or state stricture relating to varied document retention requirements — prohibitions on the destruction of business records — in order to check and enforce compliance with whatever regulatory schema (tax, environmental, public health, occupational safety, etc.)

Document retention also becomes extremely important in litigation — when a dispute or lawsuit arises between two parties — because there are evidentiary and procedural rules that aim to preserve all documents (information) that are relevant and responsive to the issues in the litigation. Critically, a party that fails to preserve business records in accordance with the court’s rules is subject to stiff penalties, including fines, adverse instructions to a jury, or the forfeit of some or all claims.

A Common but High-Stakes Regulatory Problem Solved by Machine Learning

So, imagine the government is investigating the merger between two broadband companies, and the government suspects that the two competitors engaged in illegal coordination to raise the price of broadband service and simultaneously lower quality. Before they approve the merger, the government wants to be certain that the two parties did not engage in unfair and deceptive business practices.

So, the government commands the two parties to produce for inspection, electronically, every business record (email, internal documents, transcripts of meetings…) that include communications directly between the two parties, has discussions relating to the merger, and/or is related to the pricing of broadband services.

As you might imagine, the total corpus of ALL documents controlled by the two companies borders on the hundreds of millions. That is a lot of paper to sift through for the junior attorneys, and given the government’s very specific criteria, it will take a long time for a human to read each document, determine its purpose, and decide whether its contents merit inclusion in the discovery set. The companies lawyers are also concerned that if they over-include non-responsive documents (i.e., just dump 100’s of millions of documents on the government investigator), they will be deemed to not have complied with the order, and lose out on the merger.

Aha! But our documents can be stored and searched electronically, so maybe they can just design a bunch of keyword searches pick out every document that has the term “pricing”, among other features, before having to review the document for relevancy. This is a huge improvement, but it is still slow, as the lawyers have to anticipate a lot of keyword searches, and still need to read the documents themselves. Enter the machine learning software.

With modern tools, lawyers can load the entire body of documents into one database. First, they will code a representative sample (a number of “examples”) for whether the document should or should not be included in the production of records to the government. These labeled examples will form the basis of the training material to be fed to the model. After the model has been trained, you can provide unlabeled examples (i.e., emails that haven’t been coded for relevance yet), and the machine learning model will predict the probability that the document would have been coded as relevant by a person from all the historical examples it has been fed.

On that basis, you might be able to confidently produce/exclude a million documents, but only have human-coded .1% of those as a training set. In practice, this might mean automatically producing all the documents above some probability threshold, and having a human manually review anything that the model is unsure about. This can result in huge cost savings in time and human capital. Anecdotally, I recall someone expressed doubt to me that the FBI could not have reviewed all of Hillary Clinton’s emails in a scant week or two. With machine learning and even a small team of people, the task is actually relatively trivial.

Upside, Downsides

Bias. It is important to underscore that machine learning techniques are not infallible. Biased selection of the training examples can result in bad predictions. Consider, for example, our discovery example above. Let’s say one of our errant document reviewers rushed through his batch of 1000 documents, and just haphazardly picked yes or no, nearly at random. Now, we might no longer expect the prediction model will accurately identify future examples as fitting the government’s express criteria, or not.

The discussion of “bias” in machine learning can also relate to human invidious discrimination. In one famous example, a twitter chatbot began to make racist and xenophobic tweets, which is a socially unacceptable outcome, even though the statistical model itself cannot be ascribed to have evil intent. Although this short primer is not an appropriate venue for the topic, policymakers should remain wary of drawing conclusions based on models whose inputs are not scrutinized and understood.

Job Displacement. In the case of our junior attorneys sequestered in the basement sifting through physical paper, machine learning has enabled them to shift to more intellectual redeeming tasks (like brewing coffee and filling out time-sheets). But, on the flip side, you no longer need so many junior lawyers, since their previous scope of work can now largely be automated. And, in fact, the entire industry of contract document review attorneys is seeing incredible consolidation and shrinkage. Looking towards the future, our leaders will have to contemplate how to either protect or redistribute human labor in light of such disruption.

Links/Resources/Sources

Framing: Key ML Terminology | Machine Learning Crash Course | Google Developers — developers.google.com

What is Machine Learning? – An Informed Definition — www.techemergence.com

The Risk of Machine-Learning Bias (and How to Prevent It) — sloanreview.mit.edu

People might not think about the Linux kernel all that much when talking about containers, serverless, and other hot technologies, but none of them would be possible without Linux as a solid base to build on, says Greg Kroah-Hartman.  He should know. Kroah-Hartman maintains the stable branch of the Linux kernel along with several subsystems.  He is also co-author of the Linux Kernel Development Report, a Fellow at The Linux Foundation, and he serves on the program committee for Open Source Summit.

In this article, we talk with Kroah-Hartman about his long involvement with Linux, the importance of community interaction, and the upcoming Open Source Summit.

The Linux Foundation: New technologies (cloud, containers, machine learning, serverless) are popping up on weekly basis, what’s the importance of Linux in the changing landscape?

Greg K-H: There’s the old joke, “What’s a cloud made of? Linux servers.” That is truer than most people realize. All of those things you mention rely on Linux as a base technology to build on top of.  So while people might not think about “Linux the kernel” all that much when talking about containers, serverless and the other “buzzwords of the day,” none of them would be possible without Linux being there to ensure that there is a rock-solid base for everyone to build on top of.  

The goal of an operating system is to provide a computing platform to userspace that looks the same no matter what hardware it runs on top of.  Because of this, people can build these other applications and not care if they are running it locally on a Raspberry Pi or in a cloud on a shared giant PowerPC cluster as everywhere the application API is the same.

So, Linux is essential for all of these new technologies to work properly and scale and move to different places as needed.  Without it, getting any of those things working would be a much more difficult task.

LF: You have been involved with Linux for a very long time. Has your role changed within the community? You seem to focus a lot on security these days.

Greg K-H: I originally started out as a driver writer, then helped write the security layer in the kernel many many years ago.  From there I started to maintain the USB subsystem and then co-created the driver model. From there I ended up taking over more driver subsystems and when the idea for the stable kernel releases happened back in 2005, I was one of the developers who volunteered for that.

So for the past 13 years, I’ve been doing pretty much the same thing, not much has changed since then except the increased number of stable trees I maintain at the same time to try to keep devices in the wild more secure.

I’ve been part of the kernel security team I think since it was started back in the early 2000’s but that role is more of a “find who to point the bug at” type of thing.  The kernel security team is there to help

take security problem reports and route them to the correct developer who maintains or knows that part of the kernel best.  The team has grown over the years as we have added the people that ended up getting called on the most to reduce the latency between reporting a bug and getting it fixed.

LF: We agree that Linux is being created by people all over the map, but once in a while it’s great to meet people in person. So, what role does Open Source Summit play in bringing these people together?

Greg K-H: Because open source projects are all developed by people who work for different companies and who live in different places, it’s important to get together when ever possible to actually meet the people behind the email if at all possible.  Development is an interaction that depends on trust, if I accept patches from you, then I am now responsible for those changes as well. If you disappear I am on the hook for them, so either I need to ensure they are correct, or even better, I can know that you will be around to fix the code if there is a problem.  By meeting people directly, you can establish a face behind the email to help smooth over any potential disagreements that can easily happen due to the lack of “tone” in online communication.

It’s also great to meet developers of other projects to hear of ways they are abusing your project to get it to bend to their will, or learn of problems they are having that you did not know about.  Or just learn about new things that are being developed in totally different development groups.  The huge range of talks at a conference like this makes it easy to pick up on what is happening in a huge range of different developer communities easily.

LF: You obviously meet a lot of people during the event. Have you ever come across an incident where someone ended up becoming a contributor or maintainer because of the exposure such an event provided?

Greg K-H: At one of the OSS conferences last year, I met a college student who was attending the conference for the first time.  They mentioned that they were looking of any project ideas that someone with their skill level could help out with. At a talk later that day a new idea for how to unify a specific subsystem of the kernel came up and how it was going “just take a bunch of grunt work” to accomplish.  Later that night, at the evening event, I saw the student again and mentioned the project to them and pointed them at the developer who asked for the help. They went off to talk in the corner about the specifics that would be needed to be done.

A few weeks later, a lot of patches started coming from the student and after a few rounds of review, were accepted by the maintainer.  More patches followed and eventually the majority of the work was done, which was great to see, the kernel really benefited from their contribution.

This year, I ran into the student again at another OSS conference and asked them what they were doing now.  Turns out they had gotten a job offer and were working for a Linux kernel company doing development on new products during their summer break.  Without that first interaction, meeting the developers directly that worked on the subsystem that needed the help, getting a job like that would have been much more difficult.

So, while I’m not saying that everyone who attends one of these types of conferences will instantly get a job, you will interact with developers who know what needs to be done in different areas of their open source projects.  And from there it is almost an easy jump to getting solid employment with one of the hundreds of companies that rely on these projects for their business.

LF: Are you also giving any talks at Open Source Summit?

Greg K-H:  I’m giving a talk about the Spectre and Meltdown problems that have happened this year.  It is a very high-level overview, going into the basics of what they are, and describing when the many different variants were announced and fixed in Linux.  This is a new security type of problem that is going to be with us for a very long time and I give some good tips on how to stay on top of the problem and ensure that your machines are safe.

Sign up to receive updates on Open Source Summit North America:

This article originally appeared at The Linux Foundation.

Learn how to easily check Linux logs in this article from our archives.

At some point in your career as a Linux administrator, you are going to have to view log files. After all, they are there for one very important reason…to help you troubleshoot an issue. In fact, every seasoned administrator will immediately tell you that the first thing to be done, when a problem arises, is to view the logs.

And there are plenty of logs to be found: logs for the system, logs for the kernel, for package managers, for Xorg, for the boot process, for Apache, for MySQL… For nearly anything you can think of, there is a log file.

Most log files can be found in one convenient location: /var/log. These are all system and service logs, those which you will lean on heavily when there is an issue with your operating system or one of the major services. For desktop app-specific issues, log files will be written to different locations (e.g., Thunderbird writes crash reports to ‘~/.thunderbird/Crash Reports’). Where a desktop application will write logs will depend upon the developer and if the app allows for custom log configuration.

We are going to be focus on system logs, as that is where the heart of Linux troubleshooting lies. And the key issue here is, how do you view those log files?

Fortunately there are numerous ways in which you can view your system logs, all quite simply executed from the command line.

/var/log

This is such a crucial folder on your Linux systems. Open up a terminal window and issue the command cd /var/log. Now issue the command ls and you will see the logs housed within this directory (Figure 1).

Now, let’s take a peek into one of those logs.

Viewing logs with less

One of the most important logs contained within /var/log is syslog. This particular log file logs everything except auth-related messages. Say you want to view the contents of that particular log file. To do that, you could quickly issue the command less /var/log/syslog. This command will open the syslog log file to the top. You can then use the arrow keys to scroll down one line at a time, the spacebar to scroll down one page at a time, or the mouse wheel to easily scroll through the file.

The one problem with this method is that syslog can grow fairly large; and, considering what you’re looking for will most likely be at or near the bottom, you might not want to spend the time scrolling line or page at a time to reach that end. Will syslog open in the less command, you could also hit the [Shift]+[g] combination to immediately go to the end of the log file. The end will be denoted by (END). You can then scroll up with the arrow keys or the scroll wheel to find exactly what you want.

This, of course, isn’t terribly efficient.

Viewing logs with dmesg

The dmesg command prints the kernel ring buffer. By default, the command will display all messages from the kernel ring buffer. From the terminal window, issue the command dmesg and the entire kernel ring buffer will print out (Figure 2).

Fortunately, there is a built-in control mechanism that allows you to print out only certain facilities (such as daemon).

Say you want to view log entries for the user facility. To do this, issue the command dmesg –facility=user. If anything has been logged to that facility, it will print out.

Unlike the less command, issuing dmesg will display the full contents of the log and send you to the end of the file. You can always use your scroll wheel to browse through the buffer of your terminal window (if applicable). Instead, you’ll want to pipe the output of dmesg to the less command like so:

dmesg | less

The above command will print out the contents of dmesg and allow you to scroll through the output just as you did viewing a standard log with the less command.

Viewing logs with tail

The tail command is probably one of the single most handy tools you have at your disposal for the viewing of log files. What tail does is output the last part of files. So, if you issue the command tail /var/log/syslog, it will print out only the last few lines of the syslog file.

But wait, the fun doesn’t end there. The tail command has a very important trick up its sleeve, by way of the -f option. When you issue the command tail -f /var/log/syslog, tail will continue watching the log file and print out the next line written to the file. This means you can follow what is written to syslog, as it happens, within your terminal window (Figure 3).

Using tail in this manner is invaluable for troubleshooting issues.

To escape the tail command (when following a file), hit the [Ctrl]+[x] combination.

You can also instruct tail to only follow a specific amount of lines. Say you only want to view the last five lines written to syslog; for that you could issue the command:

tail -f -n 5 /var/log/syslog

The above command would follow input to syslog and only print out the most recent five lines. As soon as a new line is written to syslog, it would remove the oldest from the top. This is a great way to make the process of following a log file even easier. I strongly recommend not using this to view anything less than four or five lines, as you’ll wind up getting input cut off and won’t get the full details of the entry.

There are other tools

You’ll find plenty of other commands (and even a few decent GUI tools) to enable the viewing of log files. Look to more, grep, head, cat, multitail, and System Log Viewer to aid you in your quest to troubleshooting systems via log files.   

Advance your career with Linux system administration skills. Check out the Essentials of System Administration course from The Linux Foundation.

In today’s technology arena, open source is pervasive. The 2018 Open Source Jobs Report found that hiring open source talent is a priority for 83 percent of hiring managers, and half are looking for candidates holding certifications. And yet, 87 percent of hiring managers also cite difficulty in finding the right open source skills and expertise. This article is the second in a weekly series on the growing importance of open source certification.

In the first article, we focused on why certification matters now more than ever. Here, we’ll focus on the kinds of certifications that are making a difference, and what is involved in completing necessary training and passing the performance-based exams that lead to certification, with tips from Clyde Seepersad, General Manager of Training and Certification at The Linux Foundation.

Performance-based exams

So, what are the details on getting certified and what are the differences between major types of certification? Most types of open source credentials and certification that you can obtain are performance-based. In many cases, trainees are required to demonstrate their skills directly from the command line.

“You’re going to be asked to do something live on the system, and then at the end, we’re going to evaluate that system to see if you were successful in accomplishing the task,” said Seepersad. This approach obviously differs from multiple choice exams and other tests where candidate answers are put in front of you. Often, certification programs involve online self-paced courses, so you can learn at your own speed, but the exams can be tough and require demonstration of expertise. That’s part of why the certifications that they lead to are valuable.

Certification options

Many people are familiar with the certifications offered by The Linux Foundation, including the Linux Foundation Certified System Administrator (LFCS) and Linux Foundation Certified Engineer (LFCE) certifications. The Linux Foundation intentionally maintains separation between its training and certification programs and uses an independent proctoring solution to monitor candidates. It also requires that all certifications be renewed every two years, which gives potential employers confidence that skills are current and have been recently demonstrated.

“Note that there are no prerequisites,” Seepersad said. “What that means is that if you’re an experienced Linux engineer, and you think the LFCE, the certified engineer credential, is the right one for you…, you’re allowed to do what we call ‘challenge the exams.’ If you think you’re ready for the LFCE, you can sign up for the LFCE without having to have gone through and taken and passed the LFCS.”

Seepersad noted that the LFCS credential is great for people starting their careers, and the LFCE credential is valuable for many people who have experience with Linux such as volunteer experience, and now want to demonstrate the breadth and depth of their skills for employers. He also said that the LFCS and LFCE coursework prepares trainees to work with various Linux distributions. Other certification options, such as the Kubernetes Fundamentals and Essentials of OpenStack Administration courses and exams, have also made a difference for many people, as cloud adoption has increased around the world.

Seepersad added that certification can make a difference if you are seeking a promotion. “Being able show that you’re over the bar in terms of certification at the engineer level can be a great way to get yourself into the consideration set for that next promotion,” he said.

Tips for Success

In terms of practical advice for taking an exam, Seepersad offered a number of tips:

• Set the date, and don’t procrastinate.

• Look through the online exam descriptions and get any training needed to be able to show fluency with the required skill sets.

• Practice on a live Linux system. This can involve downloading a free terminal emulator or other software and actually performing tasks that you will be tested on.

Seepersad also noted some common mistakes that people make when taking their exams. These include spending too long on a small set of questions, wasting too much time looking through documentation and reference tools, and applying changes without testing them in the work environment.

With open source certification playing an increasingly important role in securing a rewarding career, stay tuned for more certification details in this article series, including how to prepare for certification.

Learn more about Linux training and certification.

About a year ago, we started a project to lay the foundation for full-scale power management for applications involving the Xen Project Hypervisor on Arm architectures. We intend to make Xen on Arm’s power management the open source reference design for other Arm hypervisors in need of power management capabilities.

Looking at Previous Examples for Initial Approach

We looked at the older ACPI-based power management for Xen on x86, which features CPU idling (cpu-idle), CPU frequency scaling (cpu-freq), and suspend-to-RAM. We also looked at the PSCI platform management and pass-through capabilities of Xen on Arm, which already existed, but did not have any power management support. We decided to take a different path compared to x86 because we could not rely on ACPI for Arm, which is not widespread in the Arm embedded community. Xen on Arm already used PSCI for booting secondary CPUs, system shutdown, restart and other miscellaneous platform functions; thus, we decided to follow the trend, and base our implementation on PSCI.

Among the typical power management features, such as cpu-idle, cpu-freq, suspend-to-RAM, hibernate and others, we concluded that suspend-to-RAM would be the one best suited for our initial targets, systems-on-chips (SoCs). Most SoCs allow the CPU voltage domain to be completely powered off while the processor subsystem is suspended, and the state preserved in the RAM self-refresh mode, thereby significantly cutting the power consumption, often down to just tens of milliwatts.

Our Design Approach

Our solution provides a framework that is well suited for embedded applications. In our suspend-to-RAM approach, each unprivileged guest is given a chance to suspend on its own and to configure its own wake-up devices. At the same time, the privileged guest (Dom0) is considered to be a decision maker for the whole system: it can trigger the suspend of Xen, regardless of the states of the unprivileged guests.

These two features allow for different Xen embedded configurations and use-cases. They make it possible to freeze an unprivileged guest due to an ongoing suspend procedure, or to inform it about the suspend intent, giving it a chance to cooperate and suspend itself. These features are the foundation for higher level coordination mechanisms and use-case specific policies.

Solution Support

Our solution relies on the PSCI interface to allow guests to suspend themselves, and to enable the hypervisor to suspend the physical system. It further makes use of EEMI to enable guest notifications when the suspend-to-RAM procedure is initiated. EEMI stands for Embedded Energy Management Interface, and it is used to communicate with the power management controller on Xilinx devices. On the Xilinx Zynq UltraScale+ MPSoC we were able to suspend the whole application subsystem with Linux and Xen and put the MPSoC into its deep-sleep state, where it consumes only 35 mW. Resuming from this state is triggered by a wake-up interrupt that can be owned by either Dom0 or an unprivileged guest.

After the successful implementation of suspend-to-ram, the logical next step is to introduce CPU frequency scaling and CPU idling based on the aggregate load and performance requirements of all VMs.

While an individual VM may be aware of its own performance need, its utilization level, and the resulting CPU load, this information only applies to the virtual CPUs assigned to the guest. Since the VMs are not aware of the virtual to physical CPU mappings, while also lacking awareness of all the other VMs and their performance needs, a VM is not in a position to make suitable decisions regarding the power and performance states of the SoC.

The hypervisor, on the other hand, is scheduling the virtual CPUs and needs to be aware of their utilization of the physical CPUs. Having this visibility, the hypervisor is well suited to make power management decisions concerning the frequency and idle states of the physical CPUs. In our vision, the hypervisor scheduler will become energy aware and allocate energy consumption slots to guests, rather than time slots.

Currently, our work is focused on testing the new Xen suspend-to-RAM feature on Xilinx Zynq UltraScale+ MPSoC. We are calling the Xen Project developers to join the Xen power management activity and implement and test the new feature on other Arm architectures, so we accelerate the upstreaming effort and the accompanying cleanup.  

Authors

Mirela Grujic, Principal Engineer at AGGIOS

Davorin Mista, VP Engineering and Co-Founder at AGGIOS

Stefano Stabellini, Principal Engineer at Xilinx and Xen Project Maintainer

Vojin Zivojnovic, CEO and Co-Founder at AGGIOS

In this ongoing tour of Linux, we’ve looked at how to manipulate folders/directories, and now we’re continuing our discussion of permissions, users and groups, which are necessary to establish who can manipulate which files and directories. Last time, we showed how to create new users, and now we’re going to dive right back in:

You can create new groups and then add users to them at will with the groupadd command. For example, using:

sudo groupadd photos

will create the photos group.

You’ll need to create a directory hanging off the root directory:

sudo mkdir /photos

If you run ls -l /, one of the lines will be:

drwxr-xr-x 1 root root 0 jun 26 21:14 photos

The first root in the output is the user owner and the second root is the group owner.

To transfer the ownership of the /photos directory to the photos group, use

chgrp photos /photos

The chgrp command typically takes two parameters, the first parameter is the group that will take ownership of the file or directory and the second is the file or directory you want to give over to the the group.

Next, run ls -l / and you’ll see the line has changed to:

drwxr-xr-x 1 root photos 0 jun 26 21:14 photos

You have successfully transferred the ownership of your new directory over to the photos group.

Then, add your own user and the guest user to the photos group:

sudo usermod <your username here> -a -G photos
sudo usermod guest -a -G photos

You may have to log out and log back in to see the changes, but, when you do, running groups will show photos as one of the groups you belong to.

A couple of things to point out about the usermod command shown above. First: Be careful not to use the -g option instead of -G. The -g option changes your primary group and could lock you out of your stuff if you use it by accident. -G, on the other hand, adds you to the groups listed and doesn’t mess with the primary group. If you want to add your user to more groups than one, list them one after another, separated by commas, no spaces, after -G:

sudo usermod <your username> -a -G photos,pizza,spaceforce

Second: Be careful not to forget the -a parameter. The -a parameter stands for append and attaches the list of groups you pass to -G to the ones you already belong to. This means that, if you don’t include -a, the list of groups you already belong to, will be overwritten, again locking you out from stuff you need.

Neither of these are catastrophic problems, but it will mean you will have to add your user back manually to all the groups you belonged to, which can be a pain, especially if you have lost access to the sudo and wheel group.

Permits, Please!

There is still one more thing to do before you can copy images to the /photos directory. Notice how, when you did ls -l / above, permissions for that folder came back as drwxr-xr-x.

If you read the article I recommended at the beginning of this post, you’ll know that the first d indicates that the entry in the file system is a directory, and then you have three sets of three characters (rwx, r-x, r-x) that indicate the permissions for the user owner (rwx) of the directory, then the group owner (r-x), and finally the rest of the users (r-x). This means that the only person who has write permissions so far, that is, the only person who can copy or create files in the /photos directory, is the root user.

But that article I mentioned also tells you how to change the permissions for a directory or file:

sudo chmod g+w /photos

Running ls -l / after that will give you /photos permissions as drwxrwxr-x which is what you want: group members can now write into the directory.

Now you can try and copy an image or, indeed, any other file to the directory and it should go through without a problem:

cp image.jpg /photos

The guest user will also be able to read and write from the directory. They will also be able to read and write to it, and even move or delete files created by other users within the shared directory.

Conclusion

The permissions and privileges system in Linux has been honed over decades. inherited as it is from the old Unix systems of yore. As such, it works very well and is well thought out. Becoming familiar with it is essential for any Linux sysadmin. In fact, you can’t do much admining at all unless you understand it. But, it’s not that hard.

Next time, we’ll be dive into files and see the different ways of creating, manipulating, and destroying them in creative ways. Always fun, that last one.

See you then!

Learn more about Linux through the free “Introduction to Linux” course from The Linux Foundation and edX.

There are so many Linux distributions available, some of which are all-purpose and some that have a more singular focus. Truth be told, you can take most general distributions and turn them into purpose-driven platforms. But, when it comes to things like cloud and IoT, most prefer distributions built with that specific use in mind. That’s where the likes of Clear Linux comes in. This particular flavor of Linux was designed for the cloud, and it lets you install either an incredibly bare OS or one with exactly what you need to start developing for cloud and/or IoT.

What is Clear Linux?

Clear Linux comes from Intel’s Open Source Technology Center, which focuses primarily on the cloud. With that in mind, it should come as no surprise that Clear Linux was designed specifically for the cloud while best leveraging Intel hardware. Because Clear Linux focuses primarily on Intel hardware, it can make best use of power management features and performance optimizations. Clear Linux also features:

• FUSE file system debugging tool (for complete debug info)

• Automatic feedback-directed optimizer

• Function multi-versioning (to assist in developing platforms that can run anywhere)

• Autoproxy (no need to manually configure proxies)

• Telemetry (detect and respond to quality issues)

• Mixer tool (for composing a specific use-case OS)

• Software Update

• Stateless (separates the OS configuration, per-system configuration, and VT user-data)

Supported hardware

Clear Linux can run on very minimal hardware (e.g., a single core CPU with 128MB of RAM and 600MB of storage). But it should be known (as you might already suspect), Clear Linux can only run on Intel 64-bit architecture and the hardware must support UEFI (otherwise it won’t boot). The following processor families have been verified to run Clear Linux:

• 2nd Generation, or later, Intel® Core™ processor family.

• Intel® Xeon® Processor E3

• Intel® Xeon® Processor E5

• Intel® Xeon® Processor E7

• Intel® Atom™ processor C2000 product family for servers – Q3 2013 version or later.

• Intel® Atom™ processor E3800 series – Q4 2013 version or later.

Beyond Intel architecture, the minimum system requirements are:

I want to show you how to get Clear Linux up and running. I’ll be demonstrating on a VirtualBox VM. The installation isn’t terribly difficult, but there are certain things you need to know.

Installation

If you are going the route of VirtualBox virtual machine, create the VM as per normal, but you must enable EFI. To do this, open the Settings for the VM and click on the System tab. In this tab (Figure 1), click the checkbox for Enable EFI (special OSes only).

Once you have the VM setup, download the clear-23550-installer.iso.xz file. You will then need to uncompress the file with the command:

unxz clear-23550-installer.iso.xz

Once the file is uncompressed, you’ll see the clear-23550-installer.iso file. If you are going to use this as a virtual machine, you can attach that to the VM. If you’re installing on bare metal, burn that file to either a CD/DVD or USB flash drive as a bootable media. Boot the media or start the VirtualBox VM to be greeted by the text-based installer (Figure 2).

The installer is fairly straightforward. However, you do need to know that if you don’t go the Manual/Advanced route, you’ll wind up with a very bare system. Most of the questions asked during the installation are self explanatory. Just make sure when you reach the Choose Installation Type screen to select Manual (Figure 3).

If you go with the default (Automatic), you cannot select any additional packages. Chances are, you will want to go the Manual route (Figure 4).

Another task you are able to take care of in the Manual installation is the ability to create an admin user. If you don’t do that, the only user available is root. Of course, even if you go with the minimal installation, you can add users manually with the useradd command.

Post installation

After the installation completes, reboot and login. If you didn’t install the graphical environment (which probably will be the case, as this is geared toward the cloud), you might want to install more applications. This is done via bundles, using the swupd command. Say you want to run container applications with Docker. For this, you’d add the containers-basic bundle (for a list of all available bundles, see this page). To install the containers-basic bundle, the command is:

sudo swupd bundle-add containers-basic

After that command runs, you can now deploy container applications from Dockerhub. There are quite a large amount of bundles you can add. You can even install the GNOME desktop with the command:

sudo swupd bundle-add desktop-autostart

In fact, using bundles, you can pretty much define exactly how Clear Linux is going to function. Just remember, the base install is pretty empty, so you’ll want to go through the list of bundles and install everything you’ll need to make Clear Linux serve your specific purpose.

It should also be noted that the swupd command also takes care of the updating of bundles. This process is handled automatically. The autoupdate process can be enabled and disabled with the following two commands:

sudo swupd autoupdate --enable sudo swupd autoupdate --disable

You can also force a manual update with the command:

sudo swupd update

Once you have everything installed and updated, you can start developing for the cloud and/or IoT; Clear Linux will serve you well in that regard.

Make it yours

If you want a cloud/IoT-specific Linux distribution that lets you build a distribution for a very specific need, Clear Linux might be just what you’re looking for. Give it a go for yourself.