05-01-2020, 11:37 PM
Using mergerfs to increase your virtual storage
<div><p>What happens if you have multiple disks or partitions that you’d like to use for a media project and you don’t want to lose any of your existing data, but you’d like to have everything located or mounted under one drive. That’s where mergerfs can come to your rescue!</p>
<p> <span id="more-29616"></span> </p>
<p><a href="https://github.com/trapexit/mergerfs">mergerfs</a> is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices.</p>
<p>You will need to grab the latest RPM from their github page <a href="https://github.com/trapexit/mergerfs/releases">here</a>. The releases for Fedora have <strong><em>fc</em></strong> and the version number in the name. For example here is the version for Fedora 31:</p>
<p><a href="https://github.com/trapexit/mergerfs/releases/download/2.29.0/mergerfs-2.29.0-1.fc31.x86_64.rpm">mergerfs-2.29.0-1.fc31.x86_64.rpm</a></p>
<h2>Installing and configuring mergerfs</h2>
<p>Install the mergerfs package that you’ve downloaded using sudo:</p>
<pre class="wp-block-preformatted">$ sudo dnf install mergerfs-2.29.0-1.fc31.x86_64.rpm</pre>
<p>You will now be able to mount multiple disks as one drive. This comes in handy if you have a media server and you’d like all of your media files to show up under one location. If you upload new files to your system, you can copy them to your mergerfs directory and mergerfs will automatically copy them to which ever drive has enough free space available.</p>
<p>Here is an example to make it easier to understand:</p>
<pre class="wp-block-preformatted">$ df -hT | grep disk
/dev/sdb1 ext4 23M 386K 21M 2% /disk1
/dev/sdc1 ext4 44M 1.1M 40M 3% /disk2 $ ls -l /disk1/Videos/
total 1
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv $ ls -l /disk2/Videos/
total 2
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv</pre>
<p>In this example there are two disks mounted as <em>disk1</em> and <em>disk2</em>. Both drives have a <em><strong>Videos</strong></em> directory with existing files.</p>
<p>Now we’re going to mount those drives using mergerfs to make them appear as one larger drive.</p>
<pre class="wp-block-preformatted">$ sudo mergerfs -o defaults,allow_other,use_ino,category.create=mfs,moveonenospc=true,minfreespace=1M /disk1:/disk2 /media</pre>
<p>The mergerfs man page is quite extensive and complex so we’ll break down the options that were specified.</p>
<ul>
<li><em>defaults</em>: This will use the default settings unless specified.</li>
<li><em>allow_other</em>: allows users besides sudo or root to see the filesystem. </li>
<li><em>use_ino</em>: Causes mergerfs to supply file/directory inodes rather than libfuse. While not a default it is recommended it be enabled so that linked files share the same inode value. </li>
<li><em>category.create=mfs</em>: Spreads files out across your drives based on available space. </li>
<li><em>moveonenospc=true</em>: If enabled, if writing fails, a scan will be done looking for the drive with the most free space. </li>
<li><em>minfreespace=1M</em>: The minimum space value used. </li>
<li><em>disk1</em>: First hard drive. </li>
<li><em>disk2</em>: Second hard drive. </li>
<li><em>/media</em>: The directory folder where the drives are mounted.</li>
</ul>
<p>Here is what it looks like:</p>
<pre class="wp-block-preformatted">$ df -hT | grep disk /dev/sdb1 ext4 23M 386K 21M 2% /disk1 /dev/sdc1 ext4 44M 1.1M 40M 3% /disk2 $ df -hT | grep media 1:2 fuse.mergerfs 66M 1.4M 60M 3% /media </pre>
<p>You can see that the mergerfs mount now shows a total capacity of 66M which is the combined total of the two hard drives. </p>
<p>Continuing with the example:</p>
<p>There is a 30Mb video called <em>Baby’s second Xmas.mkv</em>. Let’s copy it to the <em>/media</em> folder which is the mergerfs mount.</p>
<pre class="wp-block-preformatted">$ ls -lh "Baby's second Xmas.mkv"
-rw-rw-r--. 1 curt curt 30M Apr 20 08:45 Baby's second Xmas.mkv
$ cp "Baby's second Xmas.mkv" /media/Videos/</pre>
<p>Here is the end result:</p>
<pre class="wp-block-preformatted">$ df -hT | grep disk
/dev/sdb1 ext4 23M 386K 21M 2% /disk1
/dev/sdc1 ext4 44M 31M 9.8M 76% /disk2 $ df -hT | grep media
1:2 fuse.mergerfs 66M 31M 30M 51% /media</pre>
<p>You can see from the disk space utilization that mergerfs automatically copied the file to disk2 because disk1 did not have enough free space.</p>
<p>Here is a breakdown of all of the files:</p>
<pre class="wp-block-preformatted">$ ls -l /disk1/Videos/
total 1
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv $ ls -l /disk2/Videos/
total 30003
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 30720000 Apr 20 08:47 Baby's second Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv $ ls -l /media/Videos/
total 30004
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 30720000 Apr 20 08:47 Baby's second Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv</pre>
<p>When you copy files to your mergerfs mount, it will always copy the files to the hard disk that has enough free space. If none of the drives in the pool have enough free space, then you won’t be able to copy them.</p>
</div>
https://www.sickgaming.net/blog/2020/05/...l-storage/
<div><p>What happens if you have multiple disks or partitions that you’d like to use for a media project and you don’t want to lose any of your existing data, but you’d like to have everything located or mounted under one drive. That’s where mergerfs can come to your rescue!</p>
<p> <span id="more-29616"></span> </p>
<p><a href="https://github.com/trapexit/mergerfs">mergerfs</a> is a union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices.</p>
<p>You will need to grab the latest RPM from their github page <a href="https://github.com/trapexit/mergerfs/releases">here</a>. The releases for Fedora have <strong><em>fc</em></strong> and the version number in the name. For example here is the version for Fedora 31:</p>
<p><a href="https://github.com/trapexit/mergerfs/releases/download/2.29.0/mergerfs-2.29.0-1.fc31.x86_64.rpm">mergerfs-2.29.0-1.fc31.x86_64.rpm</a></p>
<h2>Installing and configuring mergerfs</h2>
<p>Install the mergerfs package that you’ve downloaded using sudo:</p>
<pre class="wp-block-preformatted">$ sudo dnf install mergerfs-2.29.0-1.fc31.x86_64.rpm</pre>
<p>You will now be able to mount multiple disks as one drive. This comes in handy if you have a media server and you’d like all of your media files to show up under one location. If you upload new files to your system, you can copy them to your mergerfs directory and mergerfs will automatically copy them to which ever drive has enough free space available.</p>
<p>Here is an example to make it easier to understand:</p>
<pre class="wp-block-preformatted">$ df -hT | grep disk
/dev/sdb1 ext4 23M 386K 21M 2% /disk1
/dev/sdc1 ext4 44M 1.1M 40M 3% /disk2 $ ls -l /disk1/Videos/
total 1
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv $ ls -l /disk2/Videos/
total 2
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv</pre>
<p>In this example there are two disks mounted as <em>disk1</em> and <em>disk2</em>. Both drives have a <em><strong>Videos</strong></em> directory with existing files.</p>
<p>Now we’re going to mount those drives using mergerfs to make them appear as one larger drive.</p>
<pre class="wp-block-preformatted">$ sudo mergerfs -o defaults,allow_other,use_ino,category.create=mfs,moveonenospc=true,minfreespace=1M /disk1:/disk2 /media</pre>
<p>The mergerfs man page is quite extensive and complex so we’ll break down the options that were specified.</p>
<ul>
<li><em>defaults</em>: This will use the default settings unless specified.</li>
<li><em>allow_other</em>: allows users besides sudo or root to see the filesystem. </li>
<li><em>use_ino</em>: Causes mergerfs to supply file/directory inodes rather than libfuse. While not a default it is recommended it be enabled so that linked files share the same inode value. </li>
<li><em>category.create=mfs</em>: Spreads files out across your drives based on available space. </li>
<li><em>moveonenospc=true</em>: If enabled, if writing fails, a scan will be done looking for the drive with the most free space. </li>
<li><em>minfreespace=1M</em>: The minimum space value used. </li>
<li><em>disk1</em>: First hard drive. </li>
<li><em>disk2</em>: Second hard drive. </li>
<li><em>/media</em>: The directory folder where the drives are mounted.</li>
</ul>
<p>Here is what it looks like:</p>
<pre class="wp-block-preformatted">$ df -hT | grep disk /dev/sdb1 ext4 23M 386K 21M 2% /disk1 /dev/sdc1 ext4 44M 1.1M 40M 3% /disk2 $ df -hT | grep media 1:2 fuse.mergerfs 66M 1.4M 60M 3% /media </pre>
<p>You can see that the mergerfs mount now shows a total capacity of 66M which is the combined total of the two hard drives. </p>
<p>Continuing with the example:</p>
<p>There is a 30Mb video called <em>Baby’s second Xmas.mkv</em>. Let’s copy it to the <em>/media</em> folder which is the mergerfs mount.</p>
<pre class="wp-block-preformatted">$ ls -lh "Baby's second Xmas.mkv"
-rw-rw-r--. 1 curt curt 30M Apr 20 08:45 Baby's second Xmas.mkv
$ cp "Baby's second Xmas.mkv" /media/Videos/</pre>
<p>Here is the end result:</p>
<pre class="wp-block-preformatted">$ df -hT | grep disk
/dev/sdb1 ext4 23M 386K 21M 2% /disk1
/dev/sdc1 ext4 44M 31M 9.8M 76% /disk2 $ df -hT | grep media
1:2 fuse.mergerfs 66M 31M 30M 51% /media</pre>
<p>You can see from the disk space utilization that mergerfs automatically copied the file to disk2 because disk1 did not have enough free space.</p>
<p>Here is a breakdown of all of the files:</p>
<pre class="wp-block-preformatted">$ ls -l /disk1/Videos/
total 1
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv $ ls -l /disk2/Videos/
total 30003
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 30720000 Apr 20 08:47 Baby's second Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv $ ls -l /media/Videos/
total 30004
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Baby's first Xmas.mkv
-rw-rw-r--. 1 curt curt 30720000 Apr 20 08:47 Baby's second Xmas.mkv
-rw-rw-r--. 1 curt curt 0 Mar 8 17:21 Halloween hijinks.mkv
-rw-r--r--. 1 curt curt 0 Mar 8 17:17 Our Wedding.mkv</pre>
<p>When you copy files to your mergerfs mount, it will always copy the files to the hard disk that has enough free space. If none of the drives in the pool have enough free space, then you won’t be able to copy them.</p>
</div>
https://www.sickgaming.net/blog/2020/05/...l-storage/