Backup zvol(s)/dataset(s) to another nas4free server?

[daffy]

Hello,
This is what I have:

Server 1:
---------
OS: Nas4free 9.2.0.1 (full)
Storage: ZFS (dataset(s) for desktop clients, volume(s) for vmware)

Server 2:
---------
OS: Vmware
Storage: local drive for os and iscsi zvol mounted from server 1 for virtual machines.

Server 3: (new)
---------
OS: Nas4free 10.2.0.2 (embedded)
Storage: ZFS

Desktops:
---------
Sharing data through CIFS/SMB on Server 1.
Some desktops will have iscsi mounted zvols on Server 1.


The goal is to backup zvols & datasets present on server 1 into the new server 3 for disaster recovery if Server 1 would "disappear".

I would like to do this without shutting down server 2, which is running the virtual machines in vmware.

Suggestions about how to do this?

If you need more information about the systems just tell me and I will post here asap =)

Thanks!

[mdsystems]

If you are looking for some kind of redundant storage setup, in that server 1 is your primary storage server and server 3 is your backup storage server, perhaps you should be looking at HAST and CARP if you do not wish for any downtime on server 2. For HAST to work correctly, you are going to need block devices on server 1 and server 3 that are of the same capacity. HAST appears to work best (from my reading) directly upon a native block device, however it is designed so that it can work within partitions as well. With this in mind, if server 1 and server 3 have different configurations (i.e. different sized HDDs) you could create partitions on the disks in server 1 and 3 that are identical and set up HAST to use these partitions. Your life is much easier if server 1 and server 3 have identical HDDs. Once HAST is set up, you can then create your ZFS file system on the HAST volumes and this provides you with the storage redundancy you need.

You then need to consider how the primary storage server (or the backup server if the primary has failed) provides its storage. Do this via a CARP IP address. Here, you set up a "shared" IP address that is used by both storage servers, though only one at a time. Initially, the primary storage server manages the CARP IP address. If it fails, the backup server takes over and to all intents and purposes, following a short delay, server 2 is communicating with the backup server. Due to most iSCSI clients providing for timeouts, retries etc, it is likely that they will transparently handle the switchover but this is something that you need to test.

The above is a brief overview of how you could achieve what you want. I can provide more detail based on my own set-up, if it would help.

[daffy]

Ok, I was reading a little about HAST before I posted my initial question and it looks interesting, but got the impression that it wasn't stable enough for production environments yet (guess I was wrong?).
(I also got the impression that it would be more complicated to set up than just "simple backup" to another nas4free box).

But yes, it would be even better with HAST than backup for the vmware storage.

Server 1 and 3 are quite different beasts though, with completely different hardware & drive configurations. (Server 3 got less ram, is not as fast but got much more storage space)

You inform me that HAST needs block devices to work correctly. Am I correct if I think that it shouldn't be a problem for the vmware zvol(s)? (If so I could at least use HAST for the vmware storage, and maybe some other backup for datasets).

I appreciate any information and help to get the backup and/or hast going in a good way.

[mdsystems]

I was kind of working backwards from your desire to run VMs using your setup and I had assumed you wanted as little (if no) downtime as possible. This is what CARP and HAST would provide. You are quite correct that it is a little harder to set up than a simple ZFS copy.

If server 1 dies, you really need server 3 to be in a state where it can take over. This would imply the services being provided by server 3 being exactly the same as server 1 so that the clients and the VMware server see little or no change. If you do not use HAST/CARP, handling the failure would be quite complex as I see it, with potentially significant manual intervention. I think you risk trading set-up complexity with HAST/CARP with recovery complexity of not using it. For example, if server 1 fails and server 3 is to provide services, you would need to change the IP address of server 3, or change the clients. This would require some thought and if to be automated could be difficult. Also, is it important to retain client data such that if server 1 fails, server 3 has it available up to the point of failure? If so, how often do you transfer data between server 1 and server 3?

I would set up some virtual machines to emulate what you have and experiment. Add the correct number of storage devices etc etc (no need to match drive sizes, keep them relatively small) and if you were able to provide me with exports of the VMs (no data, of course) I could take a look at your set-up and see if I can configure it the way I suggest and you can then have them back and play with them. I would be happy to document this and post it here for other people as an example of how to achieve what I have done. Alternatively, please provide details of each server (memory, attached disks) and I can have a play myself. I would only look at servers 1 and 3. You would need to concern yourself with server 2. I have no time to do this before the weekend, but would happily do this at the weekend.

With regard to production use, I am sure that HAST/CARP have been use in FreeBSD in a production environment for some time. I am not sure whether or not the same can be said for Nas4Free's scripts that are used to manage HAST/CARP (for example I have found that they only support one CARP address on the servers by default and need changing if you want to use two like I did). To determine if something is stable enough for production use requires extensive testing and just because it works for me, does not mean it works for your set-up. With that in mind, thorough testing would be recommended. I have been testing my set-up for nearly two months and only just started moving some low priority VMs over to my cloud platform. So far, so good. However, I am observing these closely before I move my more high priority systems over.

I use Xenserver so am not to familiar with VMware (except Workstation). If you plan to use iSCSI as the back-end for the VMware storage, my experience with Nas4Free and iSCSI with a HAST/CARP set-up has been positive so far.

[daffy]

(I will write a longer reply later)

My initial reason to have Server 3 at all was to have a disaster backup available, if something really fatal happens to Server 1. Server 1 is already quite "safe" regarding hardware & setup, but who knows when something unexpected happens..?

The idea I had was to replicate the zfs volumes / datasets from Server 1 into Server 3, so I have copies ready to restore when resinstalling Server 1 if it "disappears". (Downtime during reinstallation of Server 1 was expected and isn't critical)

One reason (but I guess this can be done internally with snapshots or something?) it seems good to me with copies instead of synchronized storage is that there is some "hard backup" to revert to if the volumes/datasets on Server 1 gets trashed, corrupted or ransom ware encrypted or something similar. Then I would have a separate backup to revert to instead of having the changes directly reflected into the online hast "backup". But as I said, I guess this could/should be handled with snapshots instead?

[mdsystems]

Ok. Will await your longer reply, but since you are not looking for server 3 to take over in the event of server 1 failing, you could well look at zfs snapshot imports and exports. I guess it depends on how important the VMS are and the client data and how much data you can afford to lose, if any.

It sounds like you really are looking to use server 3 as simply a backup server. Would that be the case? If so, ZFS replication using snapshots may work in your case.

[daffy]

Some information about the systems:

Server 1:
Dual Xeon
32Gb ram
25 drives: 8 drives in each vdev, raidz3 and 1 hot spare.
zfs list:
Pool1 670G 328G 69.6K /mnt/Pool1
Pool1/firmware 832M 328G 832M /mnt/Pool1/firmware
Pool1/test 10.3G 338G 656M -
Pool1/vmware 659G 328G 659G -

Server 2:
Core i3
16Gb ram
6 drives: 6 drives in a single vdev, raidz2 (no spare)
zfs list:
pool1 1.42M 10.5T 304K /mnt/pool1
pool1/backup 288K 10.5T 288K /mnt/pool1/backup

The only important data for the moment is stored in Server 1:
Pool1/firmware
Pool1/vmware

The Pool1/vmware zvol is in use via iscsi 24/7 by a third server.

Would this be possible:
Setup HAST for firmware/vmware (of course it would be better with no downtime than some, if getting a fatal error on server 1)
Setup local backup inside Server 3 (daily/weekly etc), to make sure user errors won't get mirrored into the hast "backup"?

[raulfg3]

please check backup script here: viewtopic.php?f=70&t=2197

perhaps is what you need.

[mdsystems]

Let me know if what raulfg3 has provided is enough. If not, I can continue assisting.

Happy to help you set something up in a virtual environment that you can test and then transfer to your live one after sufficient testing.

[daffy]

raulfg3: Great link! Thanks!
mdsystems: I will start experimenting with HAST/CARP and learn about it and am happy for all help I can get. (But I should ask further questions about that in another thread under the correct topic). Thanks!

To get back on topic in this thread:
Is there any problem with making a backup of a live zvol which is in use? (Thinking about if there would be some zfs "volume is in use" error or something like that?).
I expect the backed up data from the zvol to be in a state as if the power was cut (for the machine(s) using the zvol over iscsi), but I don't see a big problem with that since it should be easy to recover from.

[mdsystems]

If you take a snapshot of the volume and back that up, the backup will reflect the state of the volume when the snapshot was taken. Be aware of possible inconsistencies - any open files will reflect the state they were in when the snapshot is taken but that does not necessarily mean that the file contents are consistent. Databases in particular come to mind as a potential issue for you.

As you say, the snapshot state is likely to be similar to a power cut. Many systems can recover from this eventuality, so would be OK as you suggest. Checking the software that you are using would be wise so that you can determine for sure that it handles such eventualities.

If you do happen to start a HAST/CARP question thread, PM me as I don't always check all of the forums and would like to assist if I can.

[ernie]

Zrep could be also a solution for your need.

http://www.bolthole.com/solaris/zrep/

BR