External Backup spanning multiple HDDs?

[sheldar]

I have setup my NAS containing multiple HDDs (5x 3TB setup as raidz2) Currently I have 4-5 TB of data. I am trying to create an offline backup using external HDDs which I will connect occasionally. I have not used RSYNC before but am familiar with scripting in general and have had a small amount of experience with bash (more comfortable with perl and sh.) I am wondering if it is possible to backup a single NAS instance (1 ZFS pool for the entire NAS (three datasets so far, but the one i want to backup is 4-5 TB.) I have a number of HDDs that I have been using for backups. Up till now I have been using a perl script and MD5 to ensure the backup copy is valid. I was hoping to use RSYNC now that my NAS is up and functional. However, everything I am reading seems to be setup for a backup NAS with the same or larger size than the primary NAS. I have a 4 TB drive and a number of smaller ones (500 GB) which together are sufficient for the data I have but none is large enough for the entire dataset.

Does RSYNC provide a way to span drives? I cannot connect all the needed drives at once (I do not have a second machine to act as a backup NAS. I have a dual bay external HDD unit that allow me to dock and undock HDDs easily.

Is what I am trying to do even a possibility. Or am I limited to using RSYNC on part of the dataset and manually ballancing how much data goes to each HDD?

[sheldar]

It would be even better to use a ZFS style RSYNC since that would only update the blocks (not files) when changes happen (changes to metadata on video files or renamed files for examples.)

If I the only option is to use RSYNC on a UFS style drive is there a way to get a list of the files sync'd so I can run a checksum on them and update a database to detect bit-rot (poor man's BIT rot detection)?

[sheldar]

I decided to try and do this manually (choose a specific rsync directory base for each HDD. However, I found that the UFS which is the preferred external HDD format for FreeBSD/NAS4Free, but it is not visible on my Mac (El Capitan.) I also cannot use FAT32 or ExFAT for some reason.

Oh well, I guess I am going back to connecting the USB drive to my media computer (OS X) and using rsync and/or MD5 to manage the backups. (I was hoping to move away from that once I built my NAS.)

[ernie]

You can activate the afp service of nas4free for Mac computer access.
The afp allows access to Mac computer.

If your nas is with zfs, you can use zrep for the backup:
http://www.bolthole.com/solaris/zrep/

And for installation

viewtopic.php?f=70&t=8166

The constraint is that the destination is also zfs.

[sheldar]

Am I correct in assuming that "the constraint that the destination is ZFS" translates to "the External HDDs used for the backup are formatted for ZFS Pools"? If this is so then I would not expect to be able to mount the drives on my desktop (Mac OS X El Capitan) and get the data back if something failed on the NAS. I would expect to be able to rebuild the contents of the NAS if enough drives/configuration fails to cause an error, but I was planning to use the backups independent of the NAS.

Right now my fall-back is to attached the external HDDs to my desktop and maintain a MD5 database on my desktop which occasionally scans the backups for bit-rot. I would then use rsync or some other copy mechanism to keep the backups up-to-date with the NAS. I was hoping for an more closely integrated solution now that my NAS is built. However, I am starting to think that that my desire to be able to read the backups from my desktop will kill the chances of an integrated solutions. (Which is too bad, since I have already figured out how to modify some example scripts to call a bash script each time a UFS HDD is connected to the NAS, which was my biggest hurdle to doing backups.)

[ernie]

Yes the destination support had to be on a computer with zfs.
So you have 2 Nas : one master nas + one backup nas.

In your case, the best solution is to write a script with rsync command.
Nas4free via cron job will launch this script (with the frequency that you need)

I think that you need a destination disk with fat32 but you cannot for your specific constraint. So you need to keep ufs for destination disk.
A solution is to find a software for Mac in order to read ufs support.

[sheldar]

Unfortunately I have only one NAS. I would love to have a second one, but that is not possible at this point. As I understand it I have two possibilities:
1) backup from NAS to a UFS drive and put UFS reading software on Mac
2) backup from Mac treating both the NAS and external HDDs as drives

Hmm. nether option is really ideal. Oh well, at least I know what my choices are.

Thanks.

[ernie]

For me yes, ans perhaps another person on the forum can have other ideas.

You can add also:
3) use of fat32 drive for backup from nas.
But not possible for you.