Posted on Leave a comment

Leveraging BI and Big Data in Modern Business

Companies today are collecting data at an unprecedented rate, but how much of the collected data actually makes an impact on their business? According to ODPi, by 2020, the accumulated volume of Big Data will increase from 4.4 zettabytes to roughly 44 zettabytes or 44 trillion GB.

It’s a tall order for companies to translate this data into ROI, and many businesses still don’t know how to combine Business Intelligence (BI) with Big Data to get insightful business value.

Cupid Chan, CTO of Index Analytics and ODPi lead for the BI & AI Special Interest Group (SIG), tells his clients, “It doesn’t matter how much data you have; unless you can get the insight from it, it is just bits and bytes occupying the storage.”

To help such businesses gain insight into how BI can be addressed by Big Data through multi-structured data and advanced data analytics, ODPi has released a new whitepaper called “BI”g Data – How Business Intelligence and Big Data Work Together.

The latest whitepaper shares best practices for combining BI and Big Data. It also shares real end-user perspectives on how businesses are using Big Data tools, the challenges they face, and where they are looking to enhance their investments.

Read more at The Linux Foundation

Posted on Leave a comment

Embedded Linux Software Highlights from Embedded World

In my day job at LinuxGizmos, I’ve been neck deep recently in embedded Linux hardware news from the Embedded World show in Nuremberg. There are plenty of new SBCs and compute modules — many based on NXP’s newly shipping i.MX8M Mini — as well as a new Qualcomm Robotics RB3 Platform, more IoT gateways, and Linux-ready chips like ST’s STM32MP1 and Octavo SiP version of the SoC.

Yet, Embedded World has produced some embedded Linux software news, as well. Here we take a brief look at some highlights, including Google open sourcing its Cloud IoT Device SDK, the Linux Foundation launching an ELISA project for open source safety-critical systems, and a new long-term kernel from the Civil Infrastructure Platform project.

In other news, Siemens has spun a Debian-based binary version of Mentor Embedded Linux (MEL), and AMD and Advantech are collaborating with Mentor to develop a machine-learning savvy implementation of MEL. Finally, Wind River announced a “Helix Platform” that combines Wind River Linux and VxWorks, and MontaVista has launched MontaVista Carrier Grade eXpress 2.6.

Google releases open source Device SDK

Google has released a Cloud IoT Device SDK under open source license designed to connect microcontroller devices and IoT-oriented Linux gizmos to its Google Cloud IoT platform. The SDK can be considered a lower-end, MCU endpoint-oriented counterpart to its Linux-focused Cloud IoT Edge stack for IoT gateways that integrate Google’s AI-accelerating Cloud TPU chips.

The Cloud IoT Device SDK comprises client libraries written in Embedded C to “enable developers to securely connect, provision, and manage devices with Cloud IoT Core,” says Google. Target devices range from handhelds to low-end smart home devices. OS support includes Zephyr, Mbed OS, FreeRTOS, and POSIX-compliant platforms like Linux. Early partners include Arm, Cypress, Nordic, Espressif, Microchip, and NXP.

The open source release presents an alternative strategy to Google’s proprietary, higher-end Android Things IoT platform. Google recently announced that Android Things would be limited to OEM partners developing smart speakers and displays with Google Assistant.

Linux Foundation launches ELISA safety-critical project

The Linux Foundation, which this week welcomed 34 new members including HP, also announced a project called Enabling Linux in Safety Applications (ELISA) to develop open source tools and processes that help companies build and certify Linux-based safety-critical applications and systems. Targeted applications include robotics, medical, smart factories, transportation, and autonomous cars.

ELISA is building on work done by the SIL2LinuxMP project from the Open Source Automation Development Lab (OSADL), as well as the Linux Foundation’s Real-Time Linux project. Founding ELISA members include Arm, BMW Car IT GmbH, Linutronix, and Toyota, which is a leading member of the LF’s Automotive Grade Linux project. The roster also includes new LF member and robotics manufacturer KUKA.

ELISA project goals include working with certification authorities and standardization bodies “to establish how Linux can be used as a component in safety-critical systems.” The project will develop safety-related reference documentation and use cases, educate and collaborate with the open source community, provide members with incident and hazard monitoring of critical components, and encourage best practices.

CIP releases first SLTS kernel

ELISA is related to the LF’s Civil Infrastructure Platform (CIP) project, which this week announced the release of its promised Super Long Term Support (SLTS) Linux Kernel with 64-bit Arm support. The key enhancement of the SLTS kernel is its unprecedented 10-year plus support. The kernel is also designed for the higher safety, security, and reliability requirements of large infrastructure and industrial applications.

The CIP project also announced two new working groups. The first is a Software Update Working Group led by Toshiba. The second is a Security Working Group led by Renesas, whose new RZ-G2 SoCs are the first to support the SLTS.

Mentor Embedded Linux goes binary

Like Wind River Linux and MontaVista, Mentor Graphics’ Mentor Embedded Linux (MEL) has been one of the leading commercial embedded Linux distros. It is also similarly based on Yocto Project code. Now, almost two years after Siemens acquired Mentor, Siemens PLM Software has announced a new version of MEL that ditches the Yocto foundation for Debian. The distro, which melds MEL with an inhouse Debian stack designed for Siemens automation equipment, is available as an “enterprise-class” binary.

Because it can load as a simple binary, the new Siemens enterprise version of MEL is easier to install and use than the Yocto-based version, claims Siemens. (The Yocto version will continue to be available.)

Siemens partner Xilinx is also sold on the binary approach: “By combining the capabilities of an embedded Linux distribution with those from the Debian binary desktop Linux distribution, today’s developers — many of whom have honed their skills in the Linux desktop development — can easily extend those same skills into fully featured embedded systems,” stated Simon George, director of system software and SoC Solution Marketing, Xilinx.

The new Linux solution provides a stable kernel, a robust toolchain, broad community support, secure field updates, and application isolation, says Siemens. It offers up-to-date cloud support and familiar MEL features such as Sourcery Analyzer tools. Improved multi-core support enables heterogeneous systems that also run Mentor’s Nucleus RTOS.

AMD and Advantech collaborate on ML-focused MEL version

In other MEL news, AMD, Advantech, and Mentor announced a customized version of MEL that runs on Advantech’s SOM-5871 compute module based on AMD’s Ryzen Embedded V1000 SoC. The solution will “make it easier for customers to implement machine vision applications within their IoT or edge compute ecosystem, helping to improve efficiency and accuracy of machine vision solutions,” says AMD. The chipmaker hints that the platform will align with the LF’s EdgeX Foundry project for edge computing.

Wind River goes cross-platform with Helix Platform

Wind River, which is no longer owned by Intel, has unveiled a Wind River Helix Virtualization Platform, an umbrella framework that integrates both Wind River Linux and the company’s VxWorks RTOS. The Helix Platform provides an integrated edge compute platform for applications ranging from industrial infrastructure to autonomous driving.

Helix Platform uses Wind River Hypervisor to enable time and space partitioning that leverages RTOS and virtualization technology, safety functionality, and COTS certification. Linux, VxWorks, and even third-party OSes such as Windows and Android can coexist together on multi-processor and multi-core systems, all orchestrated by the common Helix Cloud platform.

MontaVista unveils CGX 2.6

Finally, MontaVista has announced version 2.6 of its MontaVista Carrier Grade eXpress (CGX), the 12th generation of its Carrier Grade Linux certified distribution. Like Wind River Linux and the original MEL, CGX is a commercial embedded distro based on Yocto Project code and aimed at industrial and networking customers.

Due for release in mid-2019 with BSPs for x86 and ARMv8, MontaVista CGX 2.6 is based on Yocto 2.6, Linux kernel 4.19, and GCC 8.2 toolchain. Highlights include improved security features such as OpenSSL FIPS, OPTEE/Trustzone, Secure Boot, and SWUpdate.

CGX 2.6 provides protocol support for BLE, 4G/LTE, Zigbee, LoRA, CANbus, Modbus, and Profibus. Cloud support has been updated with APIs for the latest Amazon AWS IoT, Microsoft Azure IoT, Google Cloud IoT, and ARM mBed Client. Naturally, Kubernetes is also supported.

MontaVista was instrumental in the early development of embedded Linux, was owned by networking chip maker Cavium for several years before being spun back out as an independent company when Marvell acquired Cavium. Like its old rival Wind River, MontaVista is once again unhitched and ready for action.

Posted on Leave a comment

All about {Curly Braces} in Bash

At this stage of our Bash basics series, it would be hard not to see some crossover between topics. For example, you have already seen a lot of brackets in the examples we have shown over the past several weeks, but the focus has been elsewhere.

For the next phase of the series, we’ll take a closer look at brackets, curly, curvy, or straight, how to use them, and what they do depending on where you use them. We will also tackle other ways of enclosing things, like when to use quotes, double-quotes, and backquotes.

This week, we’re looking at curly brackets or braces: {}.

Array Builder

You have already encountered curly brackets before in The Meaning of Dot. There, the focus was on the use of the dot/period (.), but using braces to build a sequence was equally important.

As we saw then:

 echo {0..10}

prints out the numbers from 0 to 10. Using:

 echo {10..0}

prints out the same numbers, but in reverse order. And,

 echo {10..0..2}

prints every second number, starting with 10 and making its way backwards to 0.

Then,

 echo {z..a..2}

prints every second letter, starting with z and working its way backwards until a.

And so on and so forth.

Another thing you can do is combine two or more sequences:

 echo {a..z}{a..z}

This prints out all the two letter combinations of the alphabet, from aa to zz.

Is this useful? Well, actually it is. You see, arrays in Bash are defined by putting elements between parenthesis () and separating each element using a space, like this:

 month=("Jan" "Feb" "Mar" "Apr" "May" "Jun" "Jul" "Aug" "Sep" "Oct" "Nov" "Dec")

To access an element within the array, you use its index within brackets []:

 $ echo ${month[3]} # Array indexes start at [0], so [3] points to the fourth item Apr

You can accept all those brackets, parentheses, and braces on faith for a moment. We’ll talk about them presently.

Notice that, all things being equal, you can create an array with something like this:

 letter_combos=({a..z}{a..z})

and letter_combos points to an array that contains all the 2-letter combinations of the entire alphabet.

You can also do this:

 dec2bin=({0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}{0..1}) 

This last one is particularly interesting because dec2bin now contains all the binary numbers for an 8-bit register, in ascending order, starting with 00000000, 00000001, 00000010, etc., until reaching 11111111. You can use this to build yourself an 8-bit decimal-to-binary converter. Say you want to know what 25 is in binary. You can do this:

 $ echo ${dec2bin[25]} 00011001

Yes, there are better ways of converting decimal to binary as we saw in the article where we discussed & as a logical operator, but it is still interesting, right?

Parameter expansion

Getting back to

 echo ${month[3]}

Here the braces {} are not being used as apart of a sequence builder, but as a way of generating parameter expansion. Parameter expansion involves what it says on the box: it takes the variable or expression within the braces and expands it to whatever it represents.

In this case, month is the array we defined earlier, that is:

 month=("Jan" "Feb" "Mar" "Apr" "May" "Jun" "Jul" "Aug" "Sep" "Oct" "Nov" "Dec")

And, item 3 within the array points to "Apr" (remember: the first index in an array in Bash is [0]). That means that echo ${month[3]}, after the expansion, translates to echo "Apr".

Interpreting a variable as its value is one way of expanding it, but there are a few more you can leverage. You can use parameter expansion to manipulate what you read from variable, say, by cutting a chunk off the end.

Suppose you have a variable like:

 a="Too longgg"

The command:

 echo ${a%gg}

chops off the last two gs and prints “Too long“.

Breaking this down,

  • ${...} tells the shell to expand whatever is inside it
  • a is the variable you are working with
  • % tells the shell you want to chop something off the end of the expanded variable (“Too longgg”)
  • and gg is what you want to chop off.

This can be useful for converting files from one format to another. Allow me to explain with a slight digression:

ImageMagick is a set of command line tools that lets you manipulate and modify images. One of its most useful tools ImageMagick comes with is convert. In its simplest form convert allows you to, given an image in a certain format, make a copy of it in another format.

The following command takes a JPEG image called image.jpg and creates a PNG copy called image.png:

 convert image.jpg image.png

ImageMagick is often pre-installed on most Linux distros. If you can’t find it, look for it in your distro’s software manager.

Okay, end of digression. On to the example:

With variable expansion, you can do the same as shown above like this:

 i=image.jpg convert $i ${i%jpg}png

What you are doing here is chopping off the extension jpg from i and then adding png, making the command convert image.jpg image.png.

You may be wondering how this is more useful than just writing in the name of the file. Well, when you have a directory containing hundreds of JPEG images, you need to convert to PNG, run the following in it:

 for i in *.jpg; do convert $i ${i%jpg}png; done 

… and, hey presto! All the pictures get converted automatically.

If you need to chop off a chunk from the beginning of a variable, instead of %, use #:

 $ a="Hello World!" $ echo Goodbye${a#Hello} Goodbye World!

There’s quite a bit more to parameter expansion, but a lot of it makes sense only when you are writing scripts. We’ll explore more on that topic later in this series.

Output Grouping

Meanwhile, let’s finish up with something simple: you can also use { ... } to group the output from several commands into one big blob. The command:

 echo "I found all these PNGs:"; find . -iname "*.png"; echo "Within this bunch of files:"; ls > PNGs.txt

will execute all the commands but will only copy into the PNGs.txt file the output from the last ls command in the list. However, doing

 { echo "I found all these PNGs:"; find . -iname "*.png"; echo "Within this bunch of files:"; ls; } > PNGs.txt

creates the file PNGs.txt with everything, starting with the line “I found all these PNGs:“, then the list of PNG files returned by find, then the line “Within this bunch of files:” and finishing up with the complete list of files and directories within the current directory.

Notice that there is space between the braces and the commands enclosed within them. That’s because { and } are reserved words here, commands built into the shell. They would roughly translate to “group the outputs of all these commands together” in plain English.

Also notice that the list of commands has to end with a semicolon (;) or the whole thing will bork.

Next Time

In our next installment, we’ll be looking at more things that enclose other things, but of different shapes. Until then, have fun!

Read more:

And, Ampersand, and & in Linux

Ampersands and File Descriptors in Bash

Logical & in Bash

Posted on Leave a comment

Embedded Linux Conference Europe

Embedded Linux Conference

October 28, 2019

Lyon Convention Centre

69463 Lyon

France

Embedded Linux Conference (ELC) is the premier vendor-neutral technical conference where developers working on embedded Linux and industrial IoT products and deployments gather for education and collaboration, paving the way for innovation. Attend, and join 800+ technical experts paving the way for transformation in these key areas from across the globe for education, collaboration and deep dive learning opportunities.  Learn More

Click Here!

Posted on Leave a comment

Open Source Summit Europe

Open Source Summit

October 28, 2019

Lyon Convention Centre

69463 Lyon

France

Open Source Summit Europe (OSSEU) is the leading conference for developers, architects, and other technologists – as well as open source community and industry leaders – to collaborate, share information, learn about the latest technologies and gain a competitive advantage by using innovative open solutions. Over 2,000 will gather for OSSEU in 2019. Learn More

Click Here!

Posted on Leave a comment

Open Networking Summit Europe

Open Networking Summit is the industry’s premier open networking event, enabling collaborative development and innovation across enterprises, service providers and cloud providers and discussing the future of Open Source Networking, including SDN, NFV, VNF, CNF/Cloud Native Networking, Orchestration, Automation of Cloud, Core Network, Edge, Access & IoT services.  Learn More

Posted on Leave a comment

5 Linux GUI Cloud Backup Tools

We have reached a point in time where most every computer user depends upon the cloud … even if only as a storage solution. What makes the cloud really important to users, is when it’s employed as a backup. Why is that such a game changer? By backing up to the cloud, you have access to those files, from any computer you have associated with your cloud account. And because Linux powers the cloud, many services offer Linux tools.

Let’s take a look at five such tools. I will focus on GUI tools, because they offer a much lower barrier to entry to many of the CLI tools. I’ll also be focusing on various, consumer-grade cloud services (e.g., Google Drive, Dropbox, Wasabi, and pCloud). And, I will be demonstrating on the Elementary OS platform, but all of the tools listed will function on most Linux desktop distributions.

Note: Of the following backup solutions, only Duplicati is licensed as open source. With that said, let’s see what’s available.

Insync

I must confess, Insync has been my cloud backup of choice for a very long time. Since Google refuses to release a Linux desktop client for Google Drive (and I depend upon Google Drive daily), I had to turn to a third-party solution. Said solution is Insync. This particular take on syncing the desktop to Drive has not only been seamless, but faultless since I began using the tool.

The cost of Insync is a one-time $29.99 fee (per Google account). Trust me when I say this tool is worth the price of entry. With Insync you not only get an easy-to-use GUI for managing your Google Drive backup and sync, you get a tool (Figure 1) that gives you complete control over what is backed up and how it is backed up. Not only that, but you can also install Nautilus integration (which also allows you to easy add folders outside of the configured Drive sync destination).

You can download Insync for Ubuntu (or its derivatives), Linux Mint, Debian, and Fedora from the Insync download page. Once you’ve installed Insync (and associated it with your account), you can then install Nautilus integration with these steps (demonstrating on Elementary OS):

  1. Open a terminal window and issue the command sudo nano /etc/apt/sources.list.d/insync.list.

  2. Paste the following into the new file: deb http://apt.insynchq.com/ubuntu precise non-free contrib.

  3. Save and close the file.

  4. Update apt with the command sudo apt-get update.

  5. Install the necessary package with the command sudo apt-get install insync-nautilus.

Allow the installation to complete. Once finished, restart Nautilus with the command nautilus -q (or log out and back into the desktop). You should now see an Insync entry in the Nautilus right-click context menu (Figure 2).

Dropbox

Although Dropbox drew the ire of many in the Linux community (by dropping support for all filesystems but unencrypted ext4), it still supports a great deal of Linux desktop deployments. In other words, if your distribution still uses the ext4 file system (and you do not opt to encrypt your full drive), you’re good to go.

The good news is the Dropbox Linux desktop client is quite good. The tool offers a system tray icon that allows you to easily interact with your cloud syncing. Dropbox also includes CLI tools and a Nautilus integration (by way of an additional addon found here).

The Linux Dropbox desktop sync tool works exactly as you’d expect. From the Dropbox system tray drop-down (Figure 3) you can open the Dropbox folder, launch the Dropbox website, view recently changed files, get more space, pause syncing, open the preferences window, find help, and quite Dropbox.

The Dropbox/Nautilus integration is an important component, as it makes quickly adding to your cloud backup seamless and fast. From the Nautilus file manager, locate and right-click the folder to bad added, and select Dropbox > Move to Dropbox (Figure 4).

The only caveat to the Dropbox/Nautilus integration is that the only option is to move a folder to Dropbox. To some this might not be an option. The developers of this package would be wise to instead have the action create a link (instead of actually moving the folder).

Outside of that one issue, the Dropbox cloud sync/backup solution for Linux is a great route to go.

pCloud

pCloud might well be one of the finest cloud backup solutions you’ve never heard of. This take on cloud storage/backup includes features like:

  • Encryption (subscription service required for this feature);

  • Mobile apps for Android and iOS;

  • Linux, Mac, and Windows desktop clients;

  • Easy file/folder sharing;

  • Built-in audio/video players;

  • No file size limitation;

  • Sync any folder from the desktop;

  • Panel integration for most desktops; and

  • Automatic file manager integration.

pCloud offers both Linux desktop and CLI tools that function quite well. pCloud offers both a free plan (with 10GB of storage), a Premium Plan (with 500GB of storage for a one-time fee of $175.00), and a Premium Plus Plan (with 2TB of storage for a one-time fee of $350.00). Both non-free plans can also be paid on a yearly basis (instead of the one-time fee).

The pCloud desktop client is quite user-friendly. Once installed, you have access to your account information (Figure 5), the ability to create sync pairs, create shares, enable crypto (which requires an added subscription), and general settings.

The one caveat to pCloud is there’s no file manager integration for Linux. That’s overcome by the Sync folder in the pCloud client.

CloudBerry

The primary focus for CloudBerry is for Managed Service Providers. The business side of CloudBerry does have an associated cost (one that is probably well out of the price range for the average user looking for a simple cloud backup solution). However, for home usage, CloudBerry is free.

What makes CloudBerry different than the other tools is that it’s not a backup/storage solution in and of itself. Instead, CloudBerry serves as a link between your desktop and the likes of:

  • AWS

  • Microsoft Azure

  • Google Cloud

  • BackBlaze

  • OpenStack

  • Wasabi

  • Local storage

  • External drives

  • Network Attached Storage

  • Network Shares

  • And more

In other words, you use CloudBerry as the interface between the files/folders you want to share and the destination with which you want send them. This also means you must have an account with one of the many supported solutions.
Once you’ve installed CloudBerry, you create a new Backup plan for the target storage solution. For that configuration, you’ll need such information as:

  • Access Key

  • Secret Key

  • Bucket

What you’ll need for the configuration will depend on the account you’re connecting to (Figure 6).

The one caveat to CloudBerry is that it does not integrate with any file manager, nor does it include a system tray icon for interaction with the service.

Duplicati

Duplicati is another option that allows you to sync your local directories with either locally attached drives, network attached storage, or a number of cloud services. The options supported include:

  • Local folders

  • Attached drives

  • FTP/SFTP

  • OpenStack

  • WebDAV

  • Amazon Cloud Drive

  • Amazon S3

  • Azure Blob

  • Box.com

  • Dropbox

  • Google Cloud Storage

  • Google Drive

  • Microsoft OneDrive

  • And many more

Once you install Duplicati (download the installer for Debian, Ubuntu, Fedora, or RedHat from the Duplicati downloads page), click on the entry in your desktop menu, which will open a web page to the tool (Figure 7), where you can configure the app settings, create a new backup, restore from a backup, and more.

To create a backup, click Add backup and walk through the easy-to-use wizard (Figure 8). The backup service you choose will dictate what you need for a successful configuration.

For example, in order to create a backup to Google Drive, you’ll need an AuthID. For that, click the AuthID link in the Destination section of the setup, where you’ll be directed to select the Google Account to associate with the backup. Once you’ve allowed Duplicati access to the account, the AuthID will fill in and you’re ready to continue. Click Test connection and you’ll be asked to okay the creation of a new folder (if necessary). Click Next to complete the setup of the backup.

More Where That Came From

These five cloud backup tools aren’t the end of this particular rainbow. There are plenty more options where these came from (including CLI-only tools). But any of these backup clients will do a great job of serving your Linux desktop-to-cloud backup needs.