Ampersands and File Descriptors in Bash

Ampersands and File Descriptors in Bash - Printable Version

+- Sick Gaming (https://www.sickgaming.net)
+-- Forum: Computers (https://www.sickgaming.net/forum-86.html)
+--- Forum: Linux, FreeBSD, and Unix types (https://www.sickgaming.net/forum-88.html)
+--- Thread: Ampersands and File Descriptors in Bash (/thread-88675.html)

Ampersands and File Descriptors in Bash - xSicKxBot - 02-13-2019

Ampersands and File Descriptors in Bash

<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/02/ampersands-and-file-descriptors-in-bash.png" width="1969" height="1034" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/02/ampersands-and-file-descriptors-in-bash.png" class="ff-og-image-inserted" /></div>
In our quest to examine all the clutter (<code>&</code>, <code>|</code>, <code>;</code>, <code>></code>, <code><</code>, <code>{</code>, <code>[</code>, <code>(</code>, ), <code>]</code>, <code>}</code>, etc.) that is peppered throughout most chained Bash commands, <a href="https://www.linux.com/blog/learn/2019/2/and-ampersand-and-linux">we have been taking a closer look at the ampersand symbol (<code>&</code>)</a>.
<a href="https://www.linux.com/blog/learn/2019/2/and-ampersand-and-linux">Last time, we saw how you can use <code>&</code> to push processes that may take a long time to complete into the background</a>. But, the &, in combination with angle brackets, can also be used to pipe output and input elsewhere.
In the <a href="https://www.linux.com/blog/learn/2019/1/understanding-angle-brackets-bash">previous tutorials on</a> <a href="https://www.linux.com/blog/learn/2019/1/more-about-angle-brackets-bash">angle brackets</a>, you saw how to use <code>></code> like this:
<pre>
ls > list.txt
</pre>
to pipe the output from <code>ls</code> to the list.txt file.
Now we see that this is really shorthand for
<pre>
ls 1> list.txt
</pre>
And that <code>1</code>, in this context, is a file descriptor that points to the standard output (<code>stdout</code>).
In a similar fashion <code>2</code> points to standard error (<code>stderr</code>), and in the following command:
<pre>
ls 2> error.log
</pre>
all error messages are piped to the error.log file.
To recap: <code>1></code> is the standard output (<code>stdout</code>) and <code>2></code> the standard error output (<code>stderr</code>).
There is a third standard file descriptor, <code>0<</code>, the standard input (<code>stdin</code>). You can see it is an input because the arrow (<code><</code>) is pointing into the <code>0</code>, while for <code>1</code> and <code>2</code>, the arrows (<code>></code>) are pointing outwards.
<h3>What are the standard file descriptors good for?</h3>
If you are following this series in order, you have already used the standard output (<code>1></code>) several times in its shorthand form: <code>></code>.
Things like <code>stderr</code> (<code>2</code>) are also handy when, for example, you know that your command is going to throw an error, but what Bash informs you of is not useful and you don’t need to see it. If you want to make a directory in your home/ directory, for example:
<pre>
mkdir newdir
</pre>
and if newdir/ already exists, <code>mkdir</code> will show an error. But why would you care? (Ok, there some circumstances in which you may care, but not always.) At the end of the day, newdir will be there one way or another for you to fill up with stuff. You can supress the error message by pushing it into the void, which is /dev/null:
<pre>
mkdir newdir 2> /dev/null
</pre>
This is not just a matter of “let’s not show ugly and irrelevant error messages because they are annoying,” as there may be circumstances in which an error message may cause a cascade of errors elsewhere. Say, for exapmple, you want to find all the .service files under /etc. You could do this:
<pre>
find /etc -iname "*.service"
</pre>
But it turns out that on most systems, many of the lines spat out by <code>find</code> show errors because a regular user does not have read access rights to some of the folders under /etc. It makes reading the correct output cumbersome and, if <code>find</code> is part of a larger script, it could cause the next command in line to bork.
Instead, you can do this:
<pre>
find /etc -iname "*.service" 2> /dev/null
</pre>
And you get only the results you are looking for.
<h3>A Primer on File Descriptors</h3>
There are some caveats to having separate file descriptors for <code>stdout</code> and <code>stderr</code>, though. If you want to store the output in a file, doing this:
<pre>
find /etc -iname "*.service" 1> services.txt
</pre>
would work fine because <code>1></code> means “send standard output, and only standard output (NOT standard error) somewhere“.
But herein lies a problem: what if you *do* want to keep a record within the file of the errors along with the non-erroneous results? The instruction above won’t do that because it ONLY writes the correct results from <code>find</code>, and
<pre>
find /etc -iname "*.service" 2> services.txt
</pre>
will ONLY write the errors.
How do we get both? Try the following command:
<pre>
find /etc -iname "*.service" &> services.txt
</pre>
… and say hello to <code>&</code> again!
We have been saying all along that <code>stdin</code> (<code>0</code>), <code>stdout</code> (<code>1</code>), and <code>stderr</code> (<code>2</code>) are file descriptors. A file descriptor is a special construct that points to a channel to file, either for reading, or writing, or both. This comes from the old UNIX philosophy of treating everything as a file. Want to write to a device? Treat it as a file. Want to write to a socket and send data over a network? Treat it as a file. Want to read from and write to a file? Well, obviously, treat it as a file.
So, when managing where the output and errors from a command goes, treat the destination as a file. Hence, when you open them to read and write to them, they all get file descriptors.
This has interesting effects. You can, for example, pipe contents from one file descriptor to another:
<pre>
find /etc -iname "*.service" 1> services.txt 2>&1
</pre>
This pipes <code>stderr</code> to <code>stdout</code> and <code>stdout</code> is piped to a file, services.txt.
And there it is again: the <code>&</code>, signaling to Bash that <code>1</code> is the destination file descriptor.
Another thing with the standard file descriptors is that, when you pipe from one to another, the order in which you do this is a bit counterintuitive. Take the command above, for example. It looks like it has been written the wrong way around. You may be reading it like this: “pipe the output to a file and then pipe errors to the output.” It would seem the error output comes to late and is sent when <code>1</code> is already done.
But that is not how file descriptors work. A file descriptor is not a placeholder for the file, but for the input and/or output channel to the file. In this case, when you do <code>1> services.txt</code>, you are saying “open a write channel to services.txt and leave it open“. <code>1</code> is the name of the channel you are going to use, and it remains open until the end the line.
If you still think it is the wrong way around, try this:
<pre>
find /etc -iname "*.service" 2&>1 1>services.txt
</pre>
And notice how it doesn’t work, notice how errors get piped to the terminal and only the non-erroneous output (that is <code>stdout</code>) gets pushed to <code>services.txt</code>.
That is because Bash processes every result from <code>find</code> from left to right. Think about it like this: when Bash gets to <code>2&>1</code>, <code>stdout</code> (<code>1</code>) is still a channel that points to the terminal. If the result that <code>find</code> feeds Bash contains an error, it is popped into <code>2</code>, transferred to <code>1</code>, and, away it goes, off to the terminal!
Then at the end of the command, Bash sees you want to open <code>stdout</code> as a channel to the services.txt file. If no error has occurred, the result goes through <code>1</code> into the file.
By contrast, in
<pre>
find /etc -iname "*.service" 1>services.txt 2>&1
</pre>
<code>1</code> is pointing at <code>services.txt</code> right from the beginning, so anything that pops into <code>2</code> gets piped through <code>1</code>, which is already pointing to the final resting place in <code>services.txt</code>, and that is why it works.
In any case, as mentioned above <code>&></code> is shorthand for “both standard output and standard error“, that is, <code>2>&1</code>.
This is probably all a bit much, but don’t worry about it. Re-routing file descriptors here and there is commonplace in Bash command lines and scripts. And, you’ll be learning more about file descriptors as we progress through this series. See you next week!
</div>