*New 11.3 series Release:
2019-10-05: XigmaNAS 11.3.0.4.6928 - released, 11.2 series are soon unsupported!

*New 12.0 series Release:
2019-10-05: XigmaNAS 12.0.0.4.6928 - released!

*New 11.2 series Release:
2019-09-23: XigmaNAS 11.2.0.4.6881 - released!

We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

Orchestrating the execution of multiple RSYNC's

Synchronize files & directories to/from NAS4Free with minimal data transfer.
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
timski
NewUser
NewUser
Posts: 8
Joined: 22 Jun 2014 10:08
Status: Offline

Orchestrating the execution of multiple RSYNC's

#1

Post by timski » 12 Aug 2018 02:43

Hi, apologies if this has been asked/solved somewhere else.

Is there any way of "chaining" a set of RSYNC commands together in XigmaNAS? I have separate RSYNC's configured to replicate multiple folders between 2 NAS servers. I have configured each RSYNC to run at different times/days (my crude form of resource management). In XigmaNAS, is there any way to get my RSYNC's to automatically start on the completion of the previous one? I note I can do it from a script (by using &&, &, etc as delimeters between my RSYNC commands) .. however it would be cool if I could somehow, in the GUI, establish an RSYNC run order (with conditions). Or is this requirement superfluous due to something I am not aware of?
System-1: XigmaNAS 11.2.0.4 - Omnius (revision 5748), Gigabyte Z68M-D2H, i5-2400@3.10GHz, 16GB DDR3, 44TB Storage (4x8TB Seagate IronWolf, 4x3TB WDRed)
System-2: XigmaNAS 11.2.0.4 - Omnius (revision 5748), Asus B250MK, i3-7100@3.90GHz, 32GB DDR4, 44TB Storage (4x8TB Seagate IronWolf, 4x3TB WDRed)

ijeffsc
Starter
Starter
Posts: 23
Joined: 05 Dec 2015 13:11
Location: NY, ON, Krabi Thailand
Status: Offline

Re: Orchestrating the execution of multiple RSYNC's

#2

Post by ijeffsc » 12 Aug 2018 15:38

I'm kind of curious as to what the underlying requirement is.

I have a lot of somewhat dependant rsyncs going on on my little network. I generally time them so they usually complete in order approximately. For example backups run to local disk then get replicated nightly to my XigmaNAS (I need to get used to that name!). Sometimes though the EOM backups over run. Meaning the local backup doesn't complete before the network backup runs. I don't really worry about this because the next day rsync fixes up everything (I think :-).

Such scenarios are statistical bets that everything doesn't fail at once. By the time I have three copies (source, local aux disk and remote) my chances of losing stuff are fairly small.

Anyway I'm curious about such a scenario also. Chaining rsyncs might be very useful but I also wonder about complexity.

Could you describe the use case for us?
NAS4Free 10.3.0.3 (2617)
Intel Corporation D525MW 4GB
Promise SATA 300 TX4
HGST HTS721010A9E630 X 3 TOSHIBA MQ01ABD100 x2 + spare ZFS R5

timski
NewUser
NewUser
Posts: 8
Joined: 22 Jun 2014 10:08
Status: Offline

Re: Orchestrating the execution of multiple RSYNC's

#3

Post by timski » 13 Aug 2018 12:51

ijeffsc wrote:
12 Aug 2018 15:38
I'm kind of curious as to what the underlying requirement is.

I have a lot of somewhat dependant rsyncs going on on my little network. I generally time them so they usually complete in order approximately. For example backups run to local disk then get replicated nightly to my XigmaNAS (I need to get used to that name!). Sometimes though the EOM backups over run. Meaning the local backup doesn't complete before the network backup runs. I don't really worry about this because the next day rsync fixes up everything (I think :-).

Such scenarios are statistical bets that everything doesn't fail at once. By the time I have three copies (source, local aux disk and remote) my chances of losing stuff are fairly small.

Anyway I'm curious about such a scenario also. Chaining rsyncs might be very useful but I also wonder about complexity.

Could you describe the use case for us?
Yeah, no probs. My media library is pretty large, 5k movies, 7k TV series, 180k music files, etc. Given the size, I distribute the library across 8 physical drives in my NAS, which is backed up (RSYNC) to my 2nd NAS. My process is as follows .. once I have a bunch of files downloaded, I rename and file them in my first NAS, their physical location according to media type (movie, TV, etc). My Plex library runs off my second NAS so in order to update my Plex library for viewing, I need to execute 8 seperate RSYNC jobs to synch files to their Plex library locations. If done from the Xigma GUI, I need to execute them one at a time, often they run for a few hours each, and thus invariably I only get one or two done per evening .. thus many days before my Plex is up to date .. and in that time, more files to categorise and re-start the entire process. It would just be easier to orchestrate all my RSYNC jobs fire off one after the other, thus I am greeted by an up-to-date Plex the following morning ... hopefully that is not too confusing?
System-1: XigmaNAS 11.2.0.4 - Omnius (revision 5748), Gigabyte Z68M-D2H, i5-2400@3.10GHz, 16GB DDR3, 44TB Storage (4x8TB Seagate IronWolf, 4x3TB WDRed)
System-2: XigmaNAS 11.2.0.4 - Omnius (revision 5748), Asus B250MK, i3-7100@3.90GHz, 32GB DDR4, 44TB Storage (4x8TB Seagate IronWolf, 4x3TB WDRed)

ijeffsc
Starter
Starter
Posts: 23
Joined: 05 Dec 2015 13:11
Location: NY, ON, Krabi Thailand
Status: Offline

Re: Orchestrating the execution of multiple RSYNC's

#4

Post by ijeffsc » 13 Aug 2018 13:52

Hmm. My underlying question is why do you need to execute the rsync jobs one at a time? Why not just run the 8 rsync jobs either simultaneously or "continuously" via cron?

At least in my case one rsync job fires every 5 minutes on my NAS. The other bigger one fires once an hour although it rarely picks anything up when it does it often runs for more than an hour. The scripting function in the GUI blocks subsequent executions of that script so the jobs don't pile up. That latter job is the one consolidating backups and the consolidation of the backups is dependant on the backup rsyncs.

I'm not familiar with plex libraries but I'm guessing there is an implied dependancy in your 8 rsyncs otherwise why not just run them simultaneously via cron? You might be worried about network bandwidth however if you are on copper I doubt that will be an issue and you can moderate this by using the bandwidth limiting switch on rsync. 8 jobs running simultaneously with BW moderated might be much faster (more efficient) than consecutive execution. OTOH I have had a lot of perceived BW and latency issues over wireless and there I would be much more careful.

Another suggestion is that you can cobble together a script by just looking at the scripts generated by the GUI in the /var/run directory. I'm lazy so if I have a working script I like to re-use it. If you truly have dependancies then a script may be the only way to go as you can check the completion/error code of the previous rsync to verify it completed "happily" before going on to the next. (i.e. like you said && but I'd be inclined to send a notification email also).

Code: Select all

[ijeff@nas4free /var/run]$ pwd
/var/run
[ijeff@nas4free /var/run]$ ls rsync*
rsync_client_698e0055-22f8-4a3b-a2c8-eb476975de7b.sh
rsync_client_7e5ccef4-427f-4fda-8b70-284df8e05dc0.sh
Best Regards

IJ
NAS4Free 10.3.0.3 (2617)
Intel Corporation D525MW 4GB
Promise SATA 300 TX4
HGST HTS721010A9E630 X 3 TOSHIBA MQ01ABD100 x2 + spare ZFS R5

Post Reply

Return to “RSYNC”