{"id":82091,"date":"2019-02-12T16:32:00","date_gmt":"2019-02-12T16:32:00","guid":{"rendered":"http:\/\/www.sickgaming.net\/blog\/2019\/02\/12\/ampersands-and-file-descriptors-in-bash\/"},"modified":"2019-02-12T16:32:00","modified_gmt":"2019-02-12T16:32:00","slug":"ampersands-and-file-descriptors-in-bash","status":"publish","type":"post","link":"https:\/\/sickgaming.net\/blog\/2019\/02\/12\/ampersands-and-file-descriptors-in-bash\/","title":{"rendered":"Ampersands and File Descriptors in Bash"},"content":{"rendered":"
<\/div>\n

In our quest to examine\u00a0all the clutter\u00a0(&<\/code>, |<\/code>, ;<\/code>, ><\/code>, <<\/code>, {<\/code>, [<\/code>, (<\/code>, ), ]<\/code>, }<\/code>, etc.) that is peppered throughout most chained Bash commands, we have been taking a closer look at the ampersand symbol (&<\/code>)<\/a>.<\/p>\n

Last time, we saw how you can use &<\/code> to push processes that may take a long time to complete into the background<\/a>. But, the &,\u00a0in combination with angle brackets, can also be used to pipe output and input elsewhere.<\/p>\n

In the previous tutorials\u00a0on<\/a> angle brackets<\/a>, you saw how to\u00a0use ><\/code> like this:<\/p>\n

\nls > list.txt\n<\/pre>\n
to pipe the output from ls<\/code> to the list.txt<\/i> file.<\/p>\n
Now we see\u00a0that this\u00a0is really shorthand for<\/p>\n
\nls 1> list.txt\n<\/pre>\n
And that 1<\/code>, in this context, is a file descriptor that points to the standard output (stdout<\/code>).<\/p>\n
In a similar fashion 2<\/code> points to standard error (stderr<\/code>), and in the following command:<\/p>\n
\nls 2> error.log\n<\/pre>\n
all error messages are piped to the error.log<\/i> file.<\/p>\n
To recap: 1><\/code> is the standard output (stdout<\/code>) and 2><\/code> the standard error output (stderr<\/code>).<\/p>\n
There is a third standard file descriptor, 0<<\/code>, the standard input (stdin<\/code>). You can see it is an input because the arrow (<<\/code>) is pointing into the 0<\/code>, while for 1<\/code> and 2<\/code>, the arrows (><\/code>) are pointing outwards.<\/p>\n
What are the standard file descriptors good for?<\/h3>\n
If you are following this series in order, you have already used the standard output (1><\/code>) several times\u00a0in its shorthand form: ><\/code>.<\/p>\n
Things like stderr<\/code> (2<\/code>) are also handy when, for example, you know that your command is going to throw an error, but what Bash informs you of is not useful and you don’t need to see it. If you want to make a directory in your home\/<\/i> directory, for example:<\/p>\n
\nmkdir newdir\n<\/pre>\n
and if newdir\/<\/i> already exists, mkdir<\/code> will show an error. But why would you care? (Ok, there some circumstances in which you may care, but not always.) At the end of the day, newdir<\/i> will be there one way or another for you to fill up with stuff. You\u00a0can supress the error message by pushing it into the void, which is \/dev\/null<\/i>:<\/p>\n
\nmkdir newdir 2> \/dev\/null\n<\/pre>\n
This is not just a matter of “let’s not show ugly and irrelevant error messages because they are annoying,<\/i>”\u00a0as\u00a0there may be circumstances in which an error message may cause a cascade of errors elsewhere. Say, for exapmple, you want to find all the .service<\/i> files under \/etc<\/i>. You could do this:<\/p>\n
\nfind \/etc -iname \"*.service\"\n<\/pre>\n
But it turns out that on most systems, many of the lines spat out by find<\/code> show errors because a regular user does not have read access rights to some of the folders under \/etc<\/i>. It makes reading the correct output cumbersome and, if\u00a0find<\/code> is part of a larger script, it could cause the next command in line to bork.<\/p>\n
Instead, you can do this:<\/p>\n
\nfind \/etc -iname \"*.service\" 2> \/dev\/null\n<\/pre>\n
And you get only the results you are looking for.<\/p>\n
A Primer on File Descriptors<\/h3>\n
There are some caveats to having separate file descriptors for stdout<\/code> and stderr<\/code>, though. If you want to store the output in a file, doing this:<\/p>\n
\nfind \/etc -iname \"*.service\" 1> services.txt\n<\/pre>\n
would work fine because 1><\/code> means “send standard output, and only standard output (NOT standard error) somewhere<\/i>“.<\/p>\n
But herein lies a problem: what if you *do* want to keep a record within the file of the errors along with the non-erroneous results? The instruction above won’t do that because it ONLY writes the correct results from find<\/code>, and<\/p>\n
\nfind \/etc -iname \"*.service\" 2> services.txt\n<\/pre>\n
will ONLY write the errors.<\/p>\n
How do we get both? Try the following command:<\/p>\n
\nfind \/etc -iname \"*.service\" &> services.txt\n<\/pre>\n
… and say hello to &<\/code> again!<\/p>\n
We have been saying all along that stdin<\/code> (0<\/code>), stdout<\/code> (1<\/code>), and stderr<\/code> (2<\/code>) are file descriptors<\/i>. A file descriptor is a special construct that points to a channel to file, either for reading, or writing, or both. This comes from the old UNIX philosophy of treating everything as a file. Want to write to a device? Treat it as a file. Want to write to a socket and send data over a network? Treat it as a file. Want to read from and write to a file? Well, obviously, treat it as a file.<\/p>\n
So, when\u00a0managing where the output and errors from a command goes,\u00a0treat the destination as a file.\u00a0Hence, when you open them to read and write to them, they all get file descriptors.<\/p>\n
This has interesting effects. You can, for example, pipe contents from one file descriptor to another:<\/p>\n
\nfind \/etc -iname \"*.service\" 1> services.txt 2>&1\n<\/pre>\n
This pipes stderr<\/code> to stdout<\/code> and stdout<\/code> is piped to a file, services.txt<\/i>.<\/p>\n
And there it is again:\u00a0the &<\/code>, signaling to Bash that 1<\/code> is the destination file descriptor.<\/p>\n
Another thing with the standard file descriptors is that, when you pipe from one to another, the order in which you do this is a bit counterintuitive. Take the command above, for example.\u00a0It looks like it has been written the wrong way around. You may be reading it like this: “pipe the output to a file and then pipe errors to the output.<\/i>”\u00a0It would seem the error output comes to late and is sent when 1<\/code> is already done.<\/p>\n
But that is not how file descriptors work. A file descriptor is not a placeholder for the file, but for the input and\/or output channel<\/i> to the file. In this case, when you do 1> services.txt<\/code>, you are saying “open a write channel to services.txt and leave it open<\/i>“. 1<\/code> is the name of the channel you are going to use, and it remains open until the end the line.<\/p>\n
If you still think it is the wrong way around, try this:<\/p>\n
\nfind \/etc -iname \"*.service\" 2&>1 1>services.txt\n<\/pre>\n
And notice how it doesn’t work, notice how errors get piped to the terminal and only the non-erroneous output (that is stdout<\/code>) gets pushed to services.txt<\/code>.<\/p>\n
That is because Bash processes every result from find<\/code> from left to right. Think about it like this: when Bash gets to 2&>1<\/code>, stdout<\/code> (1<\/code>) is still a channel that points to the terminal. If the result that\u00a0find<\/code> feeds Bash contains an error, it is popped into 2<\/code>, transferred to 1<\/code>, and, away it goes, off to the terminal!<\/p>\n
Then at the end of the command, Bash sees you want to open stdout<\/code> as a channel to the services.txt<\/i> file. If no error has occurred, the result goes through 1<\/code> into the file.<\/p>\n
By contrast, in<\/p>\n
\nfind \/etc -iname \"*.service\" 1>services.txt 2>&1\n<\/pre>\n
1<\/code> is pointing at services.txt<\/code> right from the beginning, so anything that pops into 2<\/code> gets piped through 1<\/code>, which is already pointing to the\u00a0final resting place in services.txt<\/code>, and that is why it works.<\/p>\n
In any case, as mentioned above &><\/code> is shorthand for “both standard output and standard error<\/i>“, that is, 2>&1<\/code>.<\/p>\n
This is probably all a bit much, but don’t worry about it. Re-routing file descriptors here and there is commonplace in Bash command lines and scripts. And, you’ll be learning more about file descriptors as we progress through this series. See you next week!<\/p>\n","protected":false},"excerpt":{"rendered":"
In our quest to examine\u00a0all the clutter\u00a0(&, |, ;, >, <, {, [, (, ), ], }, etc.) that is peppered throughout most chained Bash commands, we have been taking a closer look at the ampersand symbol (&). Last time, we saw how you can use & to push processes that may take a long […]<\/p>\n","protected":false},"author":2,"featured_media":82092,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[40],"tags":[],"class_list":["post-82091","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux-freebsd-unix"],"_links":{"self":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/82091","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/comments?post=82091"}],"version-history":[{"count":0,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/posts\/82091\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media\/82092"}],"wp:attachment":[{"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/media?parent=82091"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/categories?post=82091"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sickgaming.net\/blog\/wp-json\/wp\/v2\/tags?post=82091"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}