03-18-2019, 07:05 PM
Finding Files with mlocate
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/finding-files-with-mlocate.jpg" width="1733" height="874" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/finding-files-with-mlocate.jpg" class="ff-og-image-inserted"></div>
<p><strong>Learn how to locate files in this tutorial from our archives. </strong></p>
<p>It’s not uncommon for a sysadmin to have to find needles buried deep inside haystacks. On a busy machine, there can be hundreds of thousands of files present on your filesystems. What do you do when you need to make sure one particular configuration file is up to date, but you can’t remember where it is located?</p>
<p>If you’ve used Unix-type machines for a while, then you’ve almost certainly come across the <em>find</em> command before. It is unquestionably exceptionally sophisticated and highly functional. Here’s an example that just searches for links inside a directory, ignoring files:</p>
<pre>
# find . -lname "*"</pre>
<p>You can do seemingly endless things with the <em>find</em><em> </em>command; there’s no denying that. The find command is nice and succinct when it wants to be, but it can also get complex very quickly. This is not necessarily because of the find command itself, but coupled with “xargs” you can pass it all sorts of options to tune your output, and indeed delete those files which you have found.</p>
<h3>Location, location, frustration</h3>
<p>There often comes a time when simplicity is the preferred route, however — especially when a testy boss is leaning over your shoulder, chatting away about how time is of the essence. And, imagine trying to vaguely guess the path of the file you’ve never seen but that your boss is certain lives somewhere on the busy /var partition.</p>
<p>Step forward <em>mlocate</em>. You may be aware of one of its close relatives: <em>slocate</em>, which securely (note the prepended letter s for secure) took note of the pertinent file permissions to prevent unprivileged users from seeing privileged files). Additionally, there is also the older, original locate command whence they came.</p>
<p>The differences between mlocate and other members of its family (according to mlocate at least) is that, when scanning your filesystems, mlocate doesn’t need to continually rescan all your filesystem(s). Instead, it merges its findings (note the prepended m for merge) with any existing file lists, making it much more performant and less heavy on system caches.</p>
<p>In this series of articles, we’ll look more closely at the mlocate tool (and simply refer to it as “locate” due to its popularity) and examine how to quickly and easily tune it to your heart’s content.</p>
<h3>Compact and Bijou</h3>
<p>If you’re anything like me unless you reuse complex commands frequently then ultimately you forget them and need to look them up.The beauty of the locate command is that you can query entire filesystems very quickly and without worrying about top-level, root, paths with a simple command using locate. </p>
<p>In the past, you might well have discovered that the <em>find </em>command can be very stubborn and cause you lots of unwelcome head-scratching. You know, a missing semicolon here or a special character not being escaped properly there. Let’s leave the complicated find command alone now, relax, and have a look into the clever little command that is locate.</p>
<p>You will most likely want to check that it’s on your system first by running these commands:</p>
<p>For Red Hat derivatives:</p>
<pre>
# yum install mlocate
</pre>
<p>For Debian derivatives:</p>
<pre>
# apt-get install mlocate
</pre>
<p>There shouldn’t be any differences between distributions, but there are almost certainly subtle differences between versions; beware.</p>
<p>Next, we’ll introduce a key component to the locate command, namely <em>updatedb</em>. As you can probably guess, this is the command which <em>updates</em> the locate command’s <em>db</em>. Hardly counterintuitive.</p>
<p>The db is the locate command’s file list, which I mentioned earlier. That list is held in a relatively simple and highly efficient database for performance. The updatedb runs periodically, usually at quiet times of the day, scheduled via a cron job. In Listing 1, we can see the innards of the file /etc/cron.daily/mlocate.cron (both the file’s path and its contents might possibly be distro and version dependent).</p>
<pre>
#!/bin/sh nodevs=$(< /proc/filesystems awk '$1 == "nodev" { print $2 }') renice +19 -p $$ >/dev/null 2>&1 ionice -c2 -n7 -p $$ >/dev/null 2>&1 /usr/bin/updatedb -f "$nodevs"
</pre>
<p><strong>Listing 1:</strong> How the “updatedb” command is triggered every day.</p>
<p>As you can see, the <em>mlocate.cron</em><em> </em>script makes careful use of the excellent <em>nice</em> commands in order to have as little impact as possible on system performance. I haven’t explicitly stated that this command runs at a set time every day (although if my addled memory serves, the original locate command was associated with a slow-down-your-computer run scheduled at midnight). This is thanks to the fact that on some “cron” versions delays are now introduced into overnight start times.</p>
<p>This is probably because of the so-called Thundering Herd of Hippos problem. Imagine lots of computers (or hungry animals) waking up at the same time to demand food (or resources) from a single or limited source. This can happen when all your hippos set their wristwatches using NTP (okay, this allegory is getting stretched too far, but bear with me). Imagine that exactly every five minutes (just as a “cron job” might) they all demand access to food or something otherwise being served.</p>
<p>If you don’t believe me then have a quick look at the config from — a version of cron called <em>anacron</em>, in Listing 2, which is the guts of the file<em> </em><em>/etc/anacrontab.</em></p>
<pre>
# /etc/anacrontab: configuration file for anacron # See anacron(8) and anacrontab(5) for details. SHELL=/bin/sh PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # the maximal random delay added to the base delay of the jobs RANDOM_DELAY=45 # the jobs will be started during the following hours only START_HOURS_RANGE=3-22 #period in days delay in minutes job-identifier command 1 5 cron.daily nice run-parts /etc/cron.daily 7 25 cron.weekly nice run-parts /etc/cron.weekly @monthly 45 cron.monthly nice run-parts /etc/cron.monthly </pre>
<p><strong>Listing 2:</strong> How delays are introduced into when “cron” jobs are run.</p>
<p>From Listing 2, you have hopefully spotted both “RANDOM_DELAY” and the “delay in minutes” column. If this aspect of cron is new to you, then you can find out more here:</p>
<pre>
# man anacrontab
</pre>
<p>Failing that, you can introduce a delay yourself if you’d like. An <a href="http://www.moundalexis.com/archives/000076.php">excellent web page</a> (now more than a decade old) discusses this issue in a perfectly sensible way. This website discusses using sleep to introduce a level of randomality, as seen in Listing 3.</p>
<pre>
#!/bin/sh # Grab a random value between 0-240.
value=$RANDOM
while [ $value -gt 240 ] ; do value=$RANDOM
done # Sleep for that time.
sleep $value # Syncronize.
/usr/bin/rsync -aqzC --delete --delete-after masterhost::master /some/dir/
</pre>
<p><strong>Listing 3: </strong>A shell script to introduce random delays before triggering an event, to avoid a <a href="http://www.moundalexis.com/archives/000076.php">Thundering Herd of Hippos.</a></p>
<p>The aim in mentioning these (potentially surprising) delays was to point you at the file<em> /etc/crontab</em>, or the root user’s own crontab file. If you want to change the time of when the locate command runs specifically because of disk access slowdowns, then it’s not too tricky. There may be a more graceful way of achieving this result, but you can also just move the file <em>/etc/cron.daily/mlocate.cron</em> somewhere else (I’ll use the<em> </em><em>/usr/local/etc</em><em> </em>directory), and as the root user add an entry into the root user’s crontab with this command and then paste the content as below:</p>
<pre>
# crontab -e 33 3 * * * /usr/local/etc/mlocate.cron
</pre>
<p>Rather than traipse through <em>/var/log/cron</em><em> </em>and its older, rotated, versions, you can quickly tell the last time your cron.daily jobs were fired, in the case of anacron at least, with:</p>
<pre>
# ls -hal /var/spool/anacron
</pre>
<p>Next time, we’ll look at more ways to use locate, updatedb, and other tools for finding files.</p>
<p><em>Learn more about essential sysadmin skills: Download the <a href="https://go.pardot.com/l/6342/2017-07-17/3vwshv?utm_source=linco&utm_medium=blog&utm_campaign=sysadmin&utm_content=promo">Future Proof Your SysAdmin Career</a> ebook now.</em></p>
<p><em>Chris Binnie’s latest book, Linux Server Security: Hack and Defend, shows how hackers launch sophisticated attacks to compromise servers, steal data, and crack complex passwords, so you can learn how to defend against these attacks. In the book, he also talks you through making your servers invisible, performing penetration testing, and mitigating unwelcome attacks. You can find out more about DevSecOps and Linux security via his website (<a href="http://www.devsecops.cc/">http://www.devsecops.cc</a>).</em></p>
</div>
<div style="margin: 5px 5% 10px 5%;"><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/finding-files-with-mlocate.jpg" width="1733" height="874" title="" alt="" /></div><div><div><img src="http://www.sickgaming.net/blog/wp-content/uploads/2019/03/finding-files-with-mlocate.jpg" class="ff-og-image-inserted"></div>
<p><strong>Learn how to locate files in this tutorial from our archives. </strong></p>
<p>It’s not uncommon for a sysadmin to have to find needles buried deep inside haystacks. On a busy machine, there can be hundreds of thousands of files present on your filesystems. What do you do when you need to make sure one particular configuration file is up to date, but you can’t remember where it is located?</p>
<p>If you’ve used Unix-type machines for a while, then you’ve almost certainly come across the <em>find</em> command before. It is unquestionably exceptionally sophisticated and highly functional. Here’s an example that just searches for links inside a directory, ignoring files:</p>
<pre>
# find . -lname "*"</pre>
<p>You can do seemingly endless things with the <em>find</em><em> </em>command; there’s no denying that. The find command is nice and succinct when it wants to be, but it can also get complex very quickly. This is not necessarily because of the find command itself, but coupled with “xargs” you can pass it all sorts of options to tune your output, and indeed delete those files which you have found.</p>
<h3>Location, location, frustration</h3>
<p>There often comes a time when simplicity is the preferred route, however — especially when a testy boss is leaning over your shoulder, chatting away about how time is of the essence. And, imagine trying to vaguely guess the path of the file you’ve never seen but that your boss is certain lives somewhere on the busy /var partition.</p>
<p>Step forward <em>mlocate</em>. You may be aware of one of its close relatives: <em>slocate</em>, which securely (note the prepended letter s for secure) took note of the pertinent file permissions to prevent unprivileged users from seeing privileged files). Additionally, there is also the older, original locate command whence they came.</p>
<p>The differences between mlocate and other members of its family (according to mlocate at least) is that, when scanning your filesystems, mlocate doesn’t need to continually rescan all your filesystem(s). Instead, it merges its findings (note the prepended m for merge) with any existing file lists, making it much more performant and less heavy on system caches.</p>
<p>In this series of articles, we’ll look more closely at the mlocate tool (and simply refer to it as “locate” due to its popularity) and examine how to quickly and easily tune it to your heart’s content.</p>
<h3>Compact and Bijou</h3>
<p>If you’re anything like me unless you reuse complex commands frequently then ultimately you forget them and need to look them up.The beauty of the locate command is that you can query entire filesystems very quickly and without worrying about top-level, root, paths with a simple command using locate. </p>
<p>In the past, you might well have discovered that the <em>find </em>command can be very stubborn and cause you lots of unwelcome head-scratching. You know, a missing semicolon here or a special character not being escaped properly there. Let’s leave the complicated find command alone now, relax, and have a look into the clever little command that is locate.</p>
<p>You will most likely want to check that it’s on your system first by running these commands:</p>
<p>For Red Hat derivatives:</p>
<pre>
# yum install mlocate
</pre>
<p>For Debian derivatives:</p>
<pre>
# apt-get install mlocate
</pre>
<p>There shouldn’t be any differences between distributions, but there are almost certainly subtle differences between versions; beware.</p>
<p>Next, we’ll introduce a key component to the locate command, namely <em>updatedb</em>. As you can probably guess, this is the command which <em>updates</em> the locate command’s <em>db</em>. Hardly counterintuitive.</p>
<p>The db is the locate command’s file list, which I mentioned earlier. That list is held in a relatively simple and highly efficient database for performance. The updatedb runs periodically, usually at quiet times of the day, scheduled via a cron job. In Listing 1, we can see the innards of the file /etc/cron.daily/mlocate.cron (both the file’s path and its contents might possibly be distro and version dependent).</p>
<pre>
#!/bin/sh nodevs=$(< /proc/filesystems awk '$1 == "nodev" { print $2 }') renice +19 -p $$ >/dev/null 2>&1 ionice -c2 -n7 -p $$ >/dev/null 2>&1 /usr/bin/updatedb -f "$nodevs"
</pre>
<p><strong>Listing 1:</strong> How the “updatedb” command is triggered every day.</p>
<p>As you can see, the <em>mlocate.cron</em><em> </em>script makes careful use of the excellent <em>nice</em> commands in order to have as little impact as possible on system performance. I haven’t explicitly stated that this command runs at a set time every day (although if my addled memory serves, the original locate command was associated with a slow-down-your-computer run scheduled at midnight). This is thanks to the fact that on some “cron” versions delays are now introduced into overnight start times.</p>
<p>This is probably because of the so-called Thundering Herd of Hippos problem. Imagine lots of computers (or hungry animals) waking up at the same time to demand food (or resources) from a single or limited source. This can happen when all your hippos set their wristwatches using NTP (okay, this allegory is getting stretched too far, but bear with me). Imagine that exactly every five minutes (just as a “cron job” might) they all demand access to food or something otherwise being served.</p>
<p>If you don’t believe me then have a quick look at the config from — a version of cron called <em>anacron</em>, in Listing 2, which is the guts of the file<em> </em><em>/etc/anacrontab.</em></p>
<pre>
# /etc/anacrontab: configuration file for anacron # See anacron(8) and anacrontab(5) for details. SHELL=/bin/sh PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # the maximal random delay added to the base delay of the jobs RANDOM_DELAY=45 # the jobs will be started during the following hours only START_HOURS_RANGE=3-22 #period in days delay in minutes job-identifier command 1 5 cron.daily nice run-parts /etc/cron.daily 7 25 cron.weekly nice run-parts /etc/cron.weekly @monthly 45 cron.monthly nice run-parts /etc/cron.monthly </pre>
<p><strong>Listing 2:</strong> How delays are introduced into when “cron” jobs are run.</p>
<p>From Listing 2, you have hopefully spotted both “RANDOM_DELAY” and the “delay in minutes” column. If this aspect of cron is new to you, then you can find out more here:</p>
<pre>
# man anacrontab
</pre>
<p>Failing that, you can introduce a delay yourself if you’d like. An <a href="http://www.moundalexis.com/archives/000076.php">excellent web page</a> (now more than a decade old) discusses this issue in a perfectly sensible way. This website discusses using sleep to introduce a level of randomality, as seen in Listing 3.</p>
<pre>
#!/bin/sh # Grab a random value between 0-240.
value=$RANDOM
while [ $value -gt 240 ] ; do value=$RANDOM
done # Sleep for that time.
sleep $value # Syncronize.
/usr/bin/rsync -aqzC --delete --delete-after masterhost::master /some/dir/
</pre>
<p><strong>Listing 3: </strong>A shell script to introduce random delays before triggering an event, to avoid a <a href="http://www.moundalexis.com/archives/000076.php">Thundering Herd of Hippos.</a></p>
<p>The aim in mentioning these (potentially surprising) delays was to point you at the file<em> /etc/crontab</em>, or the root user’s own crontab file. If you want to change the time of when the locate command runs specifically because of disk access slowdowns, then it’s not too tricky. There may be a more graceful way of achieving this result, but you can also just move the file <em>/etc/cron.daily/mlocate.cron</em> somewhere else (I’ll use the<em> </em><em>/usr/local/etc</em><em> </em>directory), and as the root user add an entry into the root user’s crontab with this command and then paste the content as below:</p>
<pre>
# crontab -e 33 3 * * * /usr/local/etc/mlocate.cron
</pre>
<p>Rather than traipse through <em>/var/log/cron</em><em> </em>and its older, rotated, versions, you can quickly tell the last time your cron.daily jobs were fired, in the case of anacron at least, with:</p>
<pre>
# ls -hal /var/spool/anacron
</pre>
<p>Next time, we’ll look at more ways to use locate, updatedb, and other tools for finding files.</p>
<p><em>Learn more about essential sysadmin skills: Download the <a href="https://go.pardot.com/l/6342/2017-07-17/3vwshv?utm_source=linco&utm_medium=blog&utm_campaign=sysadmin&utm_content=promo">Future Proof Your SysAdmin Career</a> ebook now.</em></p>
<p><em>Chris Binnie’s latest book, Linux Server Security: Hack and Defend, shows how hackers launch sophisticated attacks to compromise servers, steal data, and crack complex passwords, so you can learn how to defend against these attacks. In the book, he also talks you through making your servers invisible, performing penetration testing, and mitigating unwelcome attacks. You can find out more about DevSecOps and Linux security via his website (<a href="http://www.devsecops.cc/">http://www.devsecops.cc</a>).</em></p>
</div>