10-21-2020, 10:01 PM
Incremental backup with Butterfly Backup
<div><h2>Introduction</h2>
<p>This article explains how to make incremental or differential backups, with a catalog available to restore (or export) at the point you want, with <a href="https://matteoguadrini.github.io/Butterfly-Backup/">Butterfly Backup</a>.</p>
<p> <span id="more-31944"></span> </p>
<h3>Requirements</h3>
<p>Butterfly Backup is a simple wrapper of rsync written in python; the first requirement is <strong><a href="https://fedoramagazine.org/help-port-python-packages-python-3/">python3.3</a></strong> or higher (plus module <a href="https://cryptography.io/en/latest/">cryptography</a> for <em>init</em> action). Other requirements are <strong><a href="https://fedoramagazine.org/open-source-ssh-clients/">openssh</a></strong> and <strong><a href="https://fedoramagazine.org/scp-users-migration-guide-to-rsync/">rsync</a></strong> (version 2.5 or higher). Ok, let’s go! </p>
<p>[Editors note: rsync version 3.2.3 is already installed on Fedora 33 systems]</p>
<pre class="wp-block-preformatted">$ <strong>sudo dnf</strong> install python3 openssh rsync git
$ <strong>sudo pip3</strong> install cryptography</pre>
<h2>Installation</h2>
<p>After that, installing Butterfly Backup is very simple by using the following commands to clone the repository locally, and set up Butterfly Backup for use:</p>
<pre class="wp-block-preformatted">$ <strong>git</strong> clone https://github.com/MatteoGuadrini/Butterfly-Backup.git
$ <strong>cd</strong> Butterfly-Backup
$ <strong>sudo python3</strong> setup.py
$ <strong>bb</strong> --help
$ <strong>man</strong> bb</pre>
<p>To upgrade, you would use the same commands too.</p>
<h2>Example</h2>
<p>Butterfly Backup is a server to client tool and is installed on a server (or workstation). The restore process restores the files into the specified client. This process shares some of the options available to the backup process.</p>
<p>Backups are organized accord to precise catalog; this is an example:</p>
<pre class="wp-block-preformatted">$ <strong>tree</strong> destination/of/backup
.
├── destination
│ ├── hostname or ip of the PC under backup
│ │ ├── timestamp folder
│ │ │ ├── backup folders
│ │ │ ├── backup.log
│ │ │ └── restore.log
│ │ ├─── general.log
│ │ └─── symlink of last backup
│
├── export.log
├── backup.list
└── .catalog.cfg</pre>
<p>Butterfly Backup has six main operations, referred to as <strong>actions</strong>, you can get information about them with the <em>–help</em> command.</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> --help
usage: bb [-h] [--verbose] [--log] [--dry-run] [--version] {config,backup,restore,archive,list,export} ... Butterfly Backup optional arguments: -h, --help show this help message and exit --verbose, -v Enable verbosity --log, -l Create a log --dry-run, -N Dry run mode --version, -V Print version action: Valid action {config,backup,restore,archive,list,export} Available actions config Configuration options backup Backup options restore Restore options archive Archive options list List options export Export options</pre>
<h3>Configuration</h3>
<p>Configuration mode is straight forward; If you’re already familiar with the exchange keys and OpenSSH, you probably won’t need it. First, you must create a configuration (rsa keys), for instance:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> config --new
SUCCESS: New configuration successfully created!</pre>
<p>After creating the configuration, the keys will be installed (copied) on the hosts you want to backup:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> config --deploy host1
Copying configuration to host1; write the password:
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arthur/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
arthur@host1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'arthur@host1'"
and check to make sure that only the key(s) you wanted were added. SUCCESS: Configuration copied successfully on host1!</pre>
<h3>Backup</h3>
<p>There are two backup modes: <em>single</em> and <em>bulk</em>.<br />The most relevant features of the two backup modes are the parallelism and retention of old backups. See the two parameters <em>–parallel</em> and <em>–retention</em> in the <a href="https://butterfly-backup.readthedocs.io/en/latest/">documentation</a>.</p>
<h4>Single backup</h4>
<p>The backup of a single machine consists in taking the files and folders indicated in the command line, and putting them into the cataloging structure indicated above. In other words, copy all file and folders of a machine into a path.</p>
<p>This is an examples:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> backup --computer host1 --destination /mnt/backup --data User Config --type Unix
Start backup on host1
SUCCESS: Command rsync -ah --no-links arthur@host1:/home :/etc /mnt/backup/host1/2020_09_19__10_28</pre>
<h4>Bulk backup</h4>
<p>Above all, bulk mode backups share the same options as single mode, with the difference that they accept a file containing a list of hostnames or ips. In this mode backups will performed in parallel (by default 5 machines at a time). Above all, if you want to run fewer or more machines in parallel, specify the <em>–parallel</em> parameter.</p>
<p>Incremental of the previous backup, for instance:</p>
<pre class="wp-block-preformatted">$ <strong>cat</strong> /home/arthur/pclist.txt
host1
host2
host3
$ <strong>bb</strong> backup --list /home/arthur/pclist.txt --destination /mnt/backup --data User Config --type Unix
ERROR: The port 22 on host2 is closed!
ERROR: The port 22 on host3 is closed!
Start backup on host1
SUCCESS: Command rsync -ahu --no-links --link-dest=/mnt/backup/host1/2020_09_19__10_28 arthur@host1:/home :/etc /mnt/backup/host1/2020_09_19__10_50</pre>
<p>There are four backup modes, which you specify with the <em>–mode</em> flag: <strong>Full</strong> (backup all files) , <strong>Mirror</strong> (backup all files in mirror mode), <strong>Differential</strong> (is based on the latest <em>Full</em> backup) and <strong>Incremental</strong> (is based on the latest backup).<br />The default mode is <em>Incremental</em>; <em>Full</em> mode is set by default when the flag is not specified.</p>
<h3>Listing catalog</h3>
<p>The first time you run backup commands, the catalog is created. The catalog is used for future backups and all the restores that are made through Butterfly Backup. To query this catalog use the list command.<br />First, let’s query the catalog in our example:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> list --catalog /mnt/backup BUTTERFLY BACKUP CATALOG Backup id: aba860b0-9944-11e8-a93f-005056a664e0
Hostname or ip: host1
Timestamp: 2020-09-19 10:28:12 Backup id: dd6de2f2-9a1e-11e8-82b0-005056a664e0
Hostname or ip: host1
Timestamp: 2020-09-19 10:50:59</pre>
<p>Press <em>q</em> for exit and select a backup-id:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> list --catalog /mnt/backup --backup-id dd6de2f2-9a1e-11e8-82b0-005056a664e0
Backup id: dd6de2f2-9a1e-11e8-82b0-005056a664e0
Hostname or ip: host1
Type: Incremental
Timestamp: 2020-09-19 10:50:59
Start: 2020-09-19 10:50:59
Finish: 2020-09-19 11:43:51
OS: Unix
ExitCode: 0
Path: /mnt/backup/host1/2020_09_19__10_50
List: backup.log
etc
home</pre>
<p>To export the catalog list use it with an external tool like <em>cat</em>, include the <em>–</em>–<em>log</em> flag:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> list --catalog /mnt/backup --log
$ <strong>cat</strong> /mnt/backup/backup.list</pre>
<h3>Restore</h3>
<p>The restore process is the exact opposite of the backup process. It takes the files from a specific backup and push it to the destination computer.<br />This command perform a restore on the same machine of the backup, for instance:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> restore --catalog /mnt/backup --backup-id dd6de2f2-9a1e-11e8-82b0-005056a664e0 --computer host1 --log
Want to do restore path /mnt/backup/host1/2020_09_19__10_50/etc? To continue [Y/N]? y
Want to do restore path /mnt/backup/host1/2020_09_19__10_50/home? To continue [Y/N]? y
SUCCESS: Command rsync -ahu -vP --log-file=/mnt/backup/host1/2020_09_19__10_50/restore.log /mnt/backup/host1/2020_09_19__10_50/etc arthur@host1:/restore_2020_09_19__10_50
SUCCESS: Command rsync -ahu -vP --log-file=/mnt/backup/host1/2020_09_19__10_50/restore.log /mnt/backup/host1/2020_09_19__10_50/home/* arthur@host1:/home</pre>
<blockquote class="wp-block-quote">
<p>Without specifying the “<em>type</em>” flag that indicates the operating system on which the data are being retrieved, Butterfly Backup will select it directly from the catalog via the backup-id.</p>
</blockquote>
<h3>Archive old backup</h3>
<p>Archive operations are used to store backups by saving disk space.</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> archive --catalog /mnt/backup/ --days 1 --destination /mnt/archive/ --verbose --log
INFO: Check archive this backup f65e5afe-9734-11e8-b0bb-005056a664e0. Folder /mnt/backup/host1/2020_09_18__17_50
INFO: Check archive this backup 4f2b5f6e-9939-11e8-9ab6-005056a664e0. Folder /mnt/backup/host1/2020_09_15__07_26
SUCCESS: Delete /mnt/backup/host1/2020_09_15__07_26 successfully.
SUCCESS: Archive /mnt/backup/host1/2020_09_15__07_26 successfully.
$ <strong>ls</strong> /mnt/archive
host1
$ <strong>ls</strong> /mnt/archive/host1
2020_09_15__07_26.zip</pre>
<p>After that, look in the catalog and see that the backup was actually archived:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> list --catalog /mnt/backup/ -i 4f2b5f6e-9939-11e8-9ab6-005056a664e0
Backup id: 4f2b5f6e-9939-11e8-9ab6-005056a664e0
Hostname or ip: host1
Type: Incremental
Timestamp: 2020-09-15 07:26:46
Start: 2020-09-15 07:26:46
Finish: 2020-09-15 08:43:45
OS: Unix
ExitCode: 0
Path: /mnt/backup/host1/2020_09_15__07_26
<em>Archived: True</em></pre>
<h2>Conclusion</h2>
<p>Butterfly Backup was born from a very complex need; this tool provides superpowers to rsync, automates the backup and restore process. In addition, the catalog allows you to have a system similar to a “time machine”.</p>
<p>In conclusion, Butterfly Backup is a lightweight, versatile, simple and scriptable backup tool.</p>
<p>One more thing; Easter egg: <em><strong>bb</strong> -Vv</em></p>
<p>Thank you for reading my post.</p>
<p>Full documentation: <a href="https://butterfly-backup.readthedocs.io/en/latest/">https://butterfly-backup.readthedocs.io/</a><br />Github: <a href="https://github.com/MatteoGuadrini/Butterfly-Backup">https://github.com/MatteoGuadrini/Butterfly-Backup</a></p>
<hr class="wp-block-separator" />
<p><em>Photo by <a href="https://unsplash.com/@leocub?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Manu M</a> on <a href="https://unsplash.com/s/photos/butterfly?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a></em>.</p>
</div>
https://www.sickgaming.net/blog/2020/10/...ly-backup/
<div><h2>Introduction</h2>
<p>This article explains how to make incremental or differential backups, with a catalog available to restore (or export) at the point you want, with <a href="https://matteoguadrini.github.io/Butterfly-Backup/">Butterfly Backup</a>.</p>
<p> <span id="more-31944"></span> </p>
<h3>Requirements</h3>
<p>Butterfly Backup is a simple wrapper of rsync written in python; the first requirement is <strong><a href="https://fedoramagazine.org/help-port-python-packages-python-3/">python3.3</a></strong> or higher (plus module <a href="https://cryptography.io/en/latest/">cryptography</a> for <em>init</em> action). Other requirements are <strong><a href="https://fedoramagazine.org/open-source-ssh-clients/">openssh</a></strong> and <strong><a href="https://fedoramagazine.org/scp-users-migration-guide-to-rsync/">rsync</a></strong> (version 2.5 or higher). Ok, let’s go! </p>
<p>[Editors note: rsync version 3.2.3 is already installed on Fedora 33 systems]</p>
<pre class="wp-block-preformatted">$ <strong>sudo dnf</strong> install python3 openssh rsync git
$ <strong>sudo pip3</strong> install cryptography</pre>
<h2>Installation</h2>
<p>After that, installing Butterfly Backup is very simple by using the following commands to clone the repository locally, and set up Butterfly Backup for use:</p>
<pre class="wp-block-preformatted">$ <strong>git</strong> clone https://github.com/MatteoGuadrini/Butterfly-Backup.git
$ <strong>cd</strong> Butterfly-Backup
$ <strong>sudo python3</strong> setup.py
$ <strong>bb</strong> --help
$ <strong>man</strong> bb</pre>
<p>To upgrade, you would use the same commands too.</p>
<h2>Example</h2>
<p>Butterfly Backup is a server to client tool and is installed on a server (or workstation). The restore process restores the files into the specified client. This process shares some of the options available to the backup process.</p>
<p>Backups are organized accord to precise catalog; this is an example:</p>
<pre class="wp-block-preformatted">$ <strong>tree</strong> destination/of/backup
.
├── destination
│ ├── hostname or ip of the PC under backup
│ │ ├── timestamp folder
│ │ │ ├── backup folders
│ │ │ ├── backup.log
│ │ │ └── restore.log
│ │ ├─── general.log
│ │ └─── symlink of last backup
│
├── export.log
├── backup.list
└── .catalog.cfg</pre>
<p>Butterfly Backup has six main operations, referred to as <strong>actions</strong>, you can get information about them with the <em>–help</em> command.</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> --help
usage: bb [-h] [--verbose] [--log] [--dry-run] [--version] {config,backup,restore,archive,list,export} ... Butterfly Backup optional arguments: -h, --help show this help message and exit --verbose, -v Enable verbosity --log, -l Create a log --dry-run, -N Dry run mode --version, -V Print version action: Valid action {config,backup,restore,archive,list,export} Available actions config Configuration options backup Backup options restore Restore options archive Archive options list List options export Export options</pre>
<h3>Configuration</h3>
<p>Configuration mode is straight forward; If you’re already familiar with the exchange keys and OpenSSH, you probably won’t need it. First, you must create a configuration (rsa keys), for instance:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> config --new
SUCCESS: New configuration successfully created!</pre>
<p>After creating the configuration, the keys will be installed (copied) on the hosts you want to backup:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> config --deploy host1
Copying configuration to host1; write the password:
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/arthur/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
arthur@host1's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'arthur@host1'"
and check to make sure that only the key(s) you wanted were added. SUCCESS: Configuration copied successfully on host1!</pre>
<h3>Backup</h3>
<p>There are two backup modes: <em>single</em> and <em>bulk</em>.<br />The most relevant features of the two backup modes are the parallelism and retention of old backups. See the two parameters <em>–parallel</em> and <em>–retention</em> in the <a href="https://butterfly-backup.readthedocs.io/en/latest/">documentation</a>.</p>
<h4>Single backup</h4>
<p>The backup of a single machine consists in taking the files and folders indicated in the command line, and putting them into the cataloging structure indicated above. In other words, copy all file and folders of a machine into a path.</p>
<p>This is an examples:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> backup --computer host1 --destination /mnt/backup --data User Config --type Unix
Start backup on host1
SUCCESS: Command rsync -ah --no-links arthur@host1:/home :/etc /mnt/backup/host1/2020_09_19__10_28</pre>
<h4>Bulk backup</h4>
<p>Above all, bulk mode backups share the same options as single mode, with the difference that they accept a file containing a list of hostnames or ips. In this mode backups will performed in parallel (by default 5 machines at a time). Above all, if you want to run fewer or more machines in parallel, specify the <em>–parallel</em> parameter.</p>
<p>Incremental of the previous backup, for instance:</p>
<pre class="wp-block-preformatted">$ <strong>cat</strong> /home/arthur/pclist.txt
host1
host2
host3
$ <strong>bb</strong> backup --list /home/arthur/pclist.txt --destination /mnt/backup --data User Config --type Unix
ERROR: The port 22 on host2 is closed!
ERROR: The port 22 on host3 is closed!
Start backup on host1
SUCCESS: Command rsync -ahu --no-links --link-dest=/mnt/backup/host1/2020_09_19__10_28 arthur@host1:/home :/etc /mnt/backup/host1/2020_09_19__10_50</pre>
<p>There are four backup modes, which you specify with the <em>–mode</em> flag: <strong>Full</strong> (backup all files) , <strong>Mirror</strong> (backup all files in mirror mode), <strong>Differential</strong> (is based on the latest <em>Full</em> backup) and <strong>Incremental</strong> (is based on the latest backup).<br />The default mode is <em>Incremental</em>; <em>Full</em> mode is set by default when the flag is not specified.</p>
<h3>Listing catalog</h3>
<p>The first time you run backup commands, the catalog is created. The catalog is used for future backups and all the restores that are made through Butterfly Backup. To query this catalog use the list command.<br />First, let’s query the catalog in our example:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> list --catalog /mnt/backup BUTTERFLY BACKUP CATALOG Backup id: aba860b0-9944-11e8-a93f-005056a664e0
Hostname or ip: host1
Timestamp: 2020-09-19 10:28:12 Backup id: dd6de2f2-9a1e-11e8-82b0-005056a664e0
Hostname or ip: host1
Timestamp: 2020-09-19 10:50:59</pre>
<p>Press <em>q</em> for exit and select a backup-id:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> list --catalog /mnt/backup --backup-id dd6de2f2-9a1e-11e8-82b0-005056a664e0
Backup id: dd6de2f2-9a1e-11e8-82b0-005056a664e0
Hostname or ip: host1
Type: Incremental
Timestamp: 2020-09-19 10:50:59
Start: 2020-09-19 10:50:59
Finish: 2020-09-19 11:43:51
OS: Unix
ExitCode: 0
Path: /mnt/backup/host1/2020_09_19__10_50
List: backup.log
etc
home</pre>
<p>To export the catalog list use it with an external tool like <em>cat</em>, include the <em>–</em>–<em>log</em> flag:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> list --catalog /mnt/backup --log
$ <strong>cat</strong> /mnt/backup/backup.list</pre>
<h3>Restore</h3>
<p>The restore process is the exact opposite of the backup process. It takes the files from a specific backup and push it to the destination computer.<br />This command perform a restore on the same machine of the backup, for instance:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> restore --catalog /mnt/backup --backup-id dd6de2f2-9a1e-11e8-82b0-005056a664e0 --computer host1 --log
Want to do restore path /mnt/backup/host1/2020_09_19__10_50/etc? To continue [Y/N]? y
Want to do restore path /mnt/backup/host1/2020_09_19__10_50/home? To continue [Y/N]? y
SUCCESS: Command rsync -ahu -vP --log-file=/mnt/backup/host1/2020_09_19__10_50/restore.log /mnt/backup/host1/2020_09_19__10_50/etc arthur@host1:/restore_2020_09_19__10_50
SUCCESS: Command rsync -ahu -vP --log-file=/mnt/backup/host1/2020_09_19__10_50/restore.log /mnt/backup/host1/2020_09_19__10_50/home/* arthur@host1:/home</pre>
<blockquote class="wp-block-quote">
<p>Without specifying the “<em>type</em>” flag that indicates the operating system on which the data are being retrieved, Butterfly Backup will select it directly from the catalog via the backup-id.</p>
</blockquote>
<h3>Archive old backup</h3>
<p>Archive operations are used to store backups by saving disk space.</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> archive --catalog /mnt/backup/ --days 1 --destination /mnt/archive/ --verbose --log
INFO: Check archive this backup f65e5afe-9734-11e8-b0bb-005056a664e0. Folder /mnt/backup/host1/2020_09_18__17_50
INFO: Check archive this backup 4f2b5f6e-9939-11e8-9ab6-005056a664e0. Folder /mnt/backup/host1/2020_09_15__07_26
SUCCESS: Delete /mnt/backup/host1/2020_09_15__07_26 successfully.
SUCCESS: Archive /mnt/backup/host1/2020_09_15__07_26 successfully.
$ <strong>ls</strong> /mnt/archive
host1
$ <strong>ls</strong> /mnt/archive/host1
2020_09_15__07_26.zip</pre>
<p>After that, look in the catalog and see that the backup was actually archived:</p>
<pre class="wp-block-preformatted">$ <strong>bb</strong> list --catalog /mnt/backup/ -i 4f2b5f6e-9939-11e8-9ab6-005056a664e0
Backup id: 4f2b5f6e-9939-11e8-9ab6-005056a664e0
Hostname or ip: host1
Type: Incremental
Timestamp: 2020-09-15 07:26:46
Start: 2020-09-15 07:26:46
Finish: 2020-09-15 08:43:45
OS: Unix
ExitCode: 0
Path: /mnt/backup/host1/2020_09_15__07_26
<em>Archived: True</em></pre>
<h2>Conclusion</h2>
<p>Butterfly Backup was born from a very complex need; this tool provides superpowers to rsync, automates the backup and restore process. In addition, the catalog allows you to have a system similar to a “time machine”.</p>
<p>In conclusion, Butterfly Backup is a lightweight, versatile, simple and scriptable backup tool.</p>
<p>One more thing; Easter egg: <em><strong>bb</strong> -Vv</em></p>
<p>Thank you for reading my post.</p>
<p>Full documentation: <a href="https://butterfly-backup.readthedocs.io/en/latest/">https://butterfly-backup.readthedocs.io/</a><br />Github: <a href="https://github.com/MatteoGuadrini/Butterfly-Backup">https://github.com/MatteoGuadrini/Butterfly-Backup</a></p>
<hr class="wp-block-separator" />
<p><em>Photo by <a href="https://unsplash.com/@leocub?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Manu M</a> on <a href="https://unsplash.com/s/photos/butterfly?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a></em>.</p>
</div>
https://www.sickgaming.net/blog/2020/10/...ly-backup/