periodic "zfs send" on incremental snaphots to remote host

[linmag7]

Hi,

NAS4Free lets me automatically create periodic snapshots. I would like to send those to my backup-machine using zfs send | ssh. Whats the best way of doing this? create some scripts that figures out which snapshots to send and run it in cron? Using zfs -i and the two latest snapshots should work, but what if the previous zfs send command failed? then the backup-machine will not be able to receive the next incremental snapshot. I somehow need to check which snapshot is the most current on the backup-machine before I send anything more?

Regards

Magnus

[raulfg3]

have a look at this usefull scrips: viewtopic.php?f=70&t=2197

[fritz]

Hi

I am the author of the scripts mentioned by raulfg3.

The script "backupData.sh" does nearly what you want... (this script uses other files of the package)
Unfortunately, the current released version does not yet support REMOTE backup (source and destination need to be on the same machine)

I am currently modifying the script to support this feature (the experimental branch of the script that support this feature is already available here: https://github.com/fritz-hh/scripts_NAS ... kup_remote
It would be a great help if you could test it and give me your feedback.
PLEASE DO NOT USE IT IN A PRODUCTIVE ENVIRONMENT

Two more things:
- ensure you have ssh public key authentication setup on your backup machine
- change the content of the variable "SSH_BATCHMODE" to "yes"

fritz

[linmag7]

Thanks,

I'll give it a try!

currently it complains about both source and destination filesystems not having any snapshots
"Backup of "sun711/home": No snapfound found in destination filesystem"
"Backup of "sun711/home": No snapshot to be backed up found in source filesystem"

zfs list -t snapshot gives:

NAME USED AVAIL REFER MOUNTPOINT
sun711/home@2013-02-23-backup 45.3K - 2.69G -
sun711/home@2013-02-24-backup 34.6K - 2.87G -
sun711/home@2013-02-25-backup 75.5M - 3.04G -

[fritz]

Hi linmag7,

Thank you for your message.
The reason for this behaviour, is that the backupData.sh script was originally tailored to be used together with manageSnashots.sh (script that create snapshots and deletes old ones)
In fact the function sortSnapshots() from commonSnapFcts.sh (and used by both backupData.sh and manageSnapshots.sh) is looking for snapshots following the snapshots naming conventions used by manageSnapshots.sh
It is pretty easy to correct it, so that backupData.sh works whatever the naming conventions.
I will post an update once it is solved.

fritz

[fritz]

Hi linmag7,
This problem issue should be solved now
fritz

[linmag7]

Sweet! Now it works, its synchronizing my NAS4Free zfs snapshots with my FreeBSD backup server! Really nice!
The script complains about:
"cat: /config.sh: No such file or directory"
"cat: /common/commonSnapFcts.sh: No such file or directory/"

I'm running the scripts from a directory in the zfs pool, it seems to assume that the scripts are installet in "/"

I'll keep experimenting.

[fritz]

Thank you linmag7 for beta-testing
Your feedback is very valuable!
In fact, there was a bug with the path defnition in the fct "run_fct_ssh".
The issue should be solved now.
On my machine the remote backup now works fine. I test it with the script option "-r user@localhost" as I currently do not have a second nas box for the backup destination.
I hope it works too on your machine now.

fritz

[linmag7]

No problem, thank you for doing all the actual work!

I'm now doing backups from my NAS4Free to a kvm-virtualization of freebsd-9.1 and i seems to work fine. I should set up cron on my NAS to periodically run the backup script and let it run for a few days.

[linmag7]

I've set up a cron job now on my NAS4Free and it works fine so far. However, in the config.sh file I had to append a "-i /path/to/my/id_rsa" to the "BIN_SSH="/usr/bin/ssh" line. Otherwise ssh failed to log in to the remote backup server. This is not necessary if I just run the script from the command line since then it uses the default path to the id_rsa file in the users .ssh directory.

[fritz]

Hi linmag7,
I am happy it works for you.
I will add as soon as possible an option to set the path to the keys.
Thanks
fritz

[fritz]

Dear all,
v2.0-beta1 of the scripts (supporting remote ZFS replication) is available: viewtopic.php?f=70&t=2197
The option to set the path to the keys if available in this beta version
fritz

[linmag7]

Hi again, and thanks for some really useful scripts.

I'm trying to replicate the contents of one server onto a backup server. I want to replicate data using ZFS incremental backups. In your scripts you check for the case where source and destination pools are identical and treat is as an error if they are. This makes sense if both zpools are local but if the destination zpool is on a remote system this should not be a problem.

On my server I have zpool "storage_pool/home" and I want to replicate data using ZFS snapshots to by backup system which also has zpool "storage_pool/home". Should the checking for source/destination zpool names only be performed when both systems are local?

/Magnus