*New 11.3 series Release:
2019-10-05: XigmaNAS 11.3.0.4.6928 - released, 11.2 series are soon unsupported!

*New 12.0 series Release:
2019-10-05: XigmaNAS 12.0.0.4.6928 - released!

*New 11.2 series Release:
2019-09-23: XigmaNAS 11.2.0.4.6881 - released!

We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

How to Configuring Rsync ?

Synchronize files & directories to/from NAS4Free with minimal data transfer.
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
User avatar
xr4t3d
Starter
Starter
Posts: 24
Joined: 26 Oct 2016 11:46
Location: Houston, TX, USA
Status: Offline

How to Configuring Rsync ?

#1

Post by xr4t3d » 19 Nov 2016 10:30

All,

Yes this question is very basic, i have performed a quick search and didn't really find what I was looking for... I was hoping a possible sticky thread on this question so here goes.

I was wondering how to exactly setup the RSYNC on the NAS4FREE? I'm trying to perform a Rsync from a WDMYCLOUD EX2100 to the N4F. Can someone either point me to somewhere on how to setup the RSYNC Service or assist me a bit?

### Currently Running RSYNC Manually on MYCLOUD ### this is a slow process as I have to keep the CLI window open...
rsync -avzhe ssh /shares/Media/Movies/ root@192.168.0.11:/mnt/Pool01/Files/Media/Movies

Thanks,

John
Thanks,

John

Linksys WRT 1900AC
DD-WRT v3.0-r28628 std (12/29/15)
Linux 3.18.25 #54 SMP Tue Dec 22 00:34:08 CET 2015 armv7l

10.3.0.3 - Pilingitam (revision 2987)
x64-embedded on Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
Dell Poweredge 2950 - RAM: 16 GB - 2 x 1 TB's (For now...)

ijeffsc
Starter
Starter
Posts: 23
Joined: 05 Dec 2015 13:11
Location: NY, ON, Krabi Thailand
Status: Offline

Re: How to Configuring Rsync ?

#2

Post by ijeffsc » 19 Nov 2016 13:02

Where are you running into issues? I'd guess with authentication.

The way your are running rsync is what I'd call a "push" from WDMYCLOUD to N4F.

I run the other way around and run rsync over ssh "pulling" from a remote server to N4F so I set up rsync as a cron job on N4F but using the rsync GUI page to do so. I started with a command line, then a cron job, found a couple of bugs in N4F rsync GUI, the Dev folks fixed those now I run GUI all the time. Twerks good.

I don't use root accounts if I can avoid it so I set up a trusted ssh certificate between the N4F system and the remote. In other words I can ssh from N4F to the remote systems without a password using a self signed SSH certificate.

There are some minor problems with this setup but I've worked around most of them.

If you are on a local LAN then there are a number of other options I'd guess - like you can rsync over mounted remote/local file systems using CIFS or NFS etc.
NAS4Free 10.3.0.3 (2617)
Intel Corporation D525MW 4GB
Promise SATA 300 TX4
HGST HTS721010A9E630 X 3 TOSHIBA MQ01ABD100 x2 + spare ZFS R5

User avatar
xr4t3d
Starter
Starter
Posts: 24
Joined: 26 Oct 2016 11:46
Location: Houston, TX, USA
Status: Offline

Re: How to Configuring Rsync ?

#3

Post by xr4t3d » 19 Nov 2016 13:58

Jeff,

The part that I am having issues with so far is that I have no idea how to even start the RSYNC service with the GUI on the N4F GUI. Once I click on the RSYNC section, there are several different links for the "Client", "Server", "Local", "Settings", and "Modules". I was able to activate the RSYNC services, just not sure how to set it up. I would like to set up the RSYNC services on the N4F so that it will pull the data from the MyCloud device across the LAN.

At this point in time, I'm trying to just first copy the the 2 TB of data from the MyCloud to the N4F device. Then I plan to make adjustments to the Raid Configuration on the MyCloud and will eventually have the MyCloud as a backup. Once this has been done, eventually I will end up wanting to have RSYNC perform a sync between the two devices either on a scheduled time or as files are changed.

Network Configuration
MyCloud = 192.168.10
N4F = 192.168.0.11

With regards to using "root", i dont mind using it... however i plan to use "sshd" for the RSYNC scheduled events.

So my question here is how to use / setup the RSYNC services on the WebGUI on the N4F Box? Also you mentioned, CIFS / NFS, I dont think I have enabled that service just yet...should I?
Thanks,

John

Linksys WRT 1900AC
DD-WRT v3.0-r28628 std (12/29/15)
Linux 3.18.25 #54 SMP Tue Dec 22 00:34:08 CET 2015 armv7l

10.3.0.3 - Pilingitam (revision 2987)
x64-embedded on Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
Dell Poweredge 2950 - RAM: 16 GB - 2 x 1 TB's (For now...)

User avatar
tony1
Moderator
Moderator
Posts: 172
Joined: 14 Jul 2016 19:04
Status: Offline

Re: How to Configuring Rsync ?

#4

Post by tony1 » 19 Nov 2016 20:14

are you referring WDMYCLOUD to as a western digital mycloud?
if so you can enable ssh on the MDYCLOUD device and gain ssh access. then use System|Advanced|Cron to run rsync over ssh.

try a google for "enable ssh mycloud"

I think once you log into the gui change the url to " http://mybooklive/UI/ssh" from there you can enable ssh.
be careful not to brick the device with your new found power.

I use a mybooklive as a backup device in this fasion.

you can use ssh keys to automate the backup process.

you can also lower the encryption level (arc4) if you are on a trusted lan to increase performance.

if you expose you device to the wan or are on a hostile lan keep security in mind as the default password is week.

once you enable ssh you can make allot of things work or break everything :)

if I remember correct firmware upgrades remove your ssh keys.

have fun!!

User avatar
xr4t3d
Starter
Starter
Posts: 24
Joined: 26 Oct 2016 11:46
Location: Houston, TX, USA
Status: Offline

Re: How to Configuring Rsync ?

#5

Post by xr4t3d » 20 Nov 2016 13:49

Tony,

Thanks for the info, I already had the SSH stuff enabled on both WDMyCloud and N4F. I just need to figure out how to configure the SSH keys if Im going to use the cron on the N4F. I know when using Rsync, it does prompt me for a password and that is where I suspect the SSH Keys you mention comes into play. I haven't had a chance to ask Dr. Google, but if you already know of a few good articles / links to review I'm all ears..


Thanks,

John
Thanks,

John

Linksys WRT 1900AC
DD-WRT v3.0-r28628 std (12/29/15)
Linux 3.18.25 #54 SMP Tue Dec 22 00:34:08 CET 2015 armv7l

10.3.0.3 - Pilingitam (revision 2987)
x64-embedded on Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
Dell Poweredge 2950 - RAM: 16 GB - 2 x 1 TB's (For now...)

ijeffsc
Starter
Starter
Posts: 23
Joined: 05 Dec 2015 13:11
Location: NY, ON, Krabi Thailand
Status: Offline

Re: How to Configuring Rsync ?

#6

Post by ijeffsc » 20 Nov 2016 15:56

Hi John,

Good tips from Tony who seems familiar with the WDMyCloud stuff. I am not and I'm really only a beginner with N4F although I have been running FreeBSD for about 20 years on a hobby basis. Bear this in mind with my answers YMMV :-).

My next step in your circumstances would be to verify you can ssh from N4F into the WDMyCloud box with a password. This remote SSH is where I avoid using root as creating a direct root login troubles me. If you system is on a secure LAN and WAN/Internet access is blocked then you may feel comfortable to enable remote SSH for root but I believe by default it is disabled. To work around this I create local accounts on the respective systems that have just the permissions they need to complete the rsync/ssh. This does cause other issues with N4F though in particular only root has rights to write lock files in /var/run. These lock files prevent long running scripts to have multiple instances running so they are important in situations like yours. I solve this problem for "ordinary users" by adding the local account in N4F to the wheel group then open up RW on /var/run for the group wheel. I also use this wheel group account as my mechanism to remote login then su to root when I need it.

Once you can ssh with a password then you can try the password-less approach. I used self signed certificates which you should regardless protect. I used a procedure like this:

http://www.thegeekstuff.com/2008/11/3-s ... sh-copy-id

Once you can ssh from N4F to your WDMyCloud then you can try your command line rsync and see how that goes.

Finally for your initial GUI debug you should be using the rsync configuration without a server (from memory) this is rsync over ssh. The GUI will create a script that you will find in /var/run (from memory I'm not near my N4F box) and you can manually verify the script matches your desires.

At least in my case the complicated part is getting the right level of trust going between the two hosts. Once that was established I never bothered much with the command line. The GUI is very pleasant to make small adjustments from there on in.

I hope this is helpful.

Ian
NAS4Free 10.3.0.3 (2617)
Intel Corporation D525MW 4GB
Promise SATA 300 TX4
HGST HTS721010A9E630 X 3 TOSHIBA MQ01ABD100 x2 + spare ZFS R5

User avatar
xr4t3d
Starter
Starter
Posts: 24
Joined: 26 Oct 2016 11:46
Location: Houston, TX, USA
Status: Offline

Re: How to Configuring Rsync ?

#7

Post by xr4t3d » 20 Nov 2016 16:06

Ian,

Thanks for the input, I just got home and starting to research a bit.. Ill post my updates when Im able to do something...
Thanks,

John

Linksys WRT 1900AC
DD-WRT v3.0-r28628 std (12/29/15)
Linux 3.18.25 #54 SMP Tue Dec 22 00:34:08 CET 2015 armv7l

10.3.0.3 - Pilingitam (revision 2987)
x64-embedded on Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
Dell Poweredge 2950 - RAM: 16 GB - 2 x 1 TB's (For now...)

User avatar
tony1
Moderator
Moderator
Posts: 172
Joined: 14 Jul 2016 19:04
Status: Offline

Re: How to Configuring Rsync ?

#8

Post by tony1 » 20 Nov 2016 18:34

you will need to do something like,
/usr/local/bin/rsync -a --rsync-path=/usr/bin/rsync /mnt/tank/??? root@WDMYCLOUD
the keys go in the users home directory ~/.ssh

by default root is enabled on the WDMYCLOUD. if it's just on a home LAN root is fine but you can secure it.

I don't expose mine to the WAN so I try to keep it as close as I can to the default installation as possible but if your LAN is hostile you can secure it.



have fun a don't brick it ;)

User avatar
xr4t3d
Starter
Starter
Posts: 24
Joined: 26 Oct 2016 11:46
Location: Houston, TX, USA
Status: Offline

Re: How to Configuring Rsync ?

#9

Post by xr4t3d » 21 Nov 2016 13:54

Just wanted to post a quick reply...

I found this article online, I was able to get this to work one way however the other way not so much.
I was able to get this to work from the WD to N4F, however from N4F to WD doesn't want to work they way I want it. This is the way I need it to work. Ill keep chipping at it to see where I get...

I wanted to put this link in here for future reference....
http://www.linuxproblem.org/art_9.html
Thanks,

John

Linksys WRT 1900AC
DD-WRT v3.0-r28628 std (12/29/15)
Linux 3.18.25 #54 SMP Tue Dec 22 00:34:08 CET 2015 armv7l

10.3.0.3 - Pilingitam (revision 2987)
x64-embedded on Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
Dell Poweredge 2950 - RAM: 16 GB - 2 x 1 TB's (For now...)

User avatar
tony1
Moderator
Moderator
Posts: 172
Joined: 14 Jul 2016 19:04
Status: Offline

Re: How to Configuring Rsync ?

#10

Post by tony1 » 21 Nov 2016 21:56

make sure the .ssh directory and the keys have the correct permissions on N4F and the other device.
you can use the same public/private keys on both servers but the correct way would be for each device to have there own public/private key.
then the N4F public key would go to the wdmc and the wdmc public key would go to N4F.
a good way to remember it is a private key is private and stays with the device. a public key is public and anyone can have that because it is public.
so the wdmc needs to have the N4F public key in order for N4F to make the connection.
if you don't want wdmc accessing N4F then don't install the wdmc's public key on N4F.

you will get it! there is allot of good guides out there but some abuse their use.

ijeffsc
Starter
Starter
Posts: 23
Joined: 05 Dec 2015 13:11
Location: NY, ON, Krabi Thailand
Status: Offline

Re: How to Configuring Rsync ?

#11

Post by ijeffsc » 22 Nov 2016 13:05

John,

Sorry for the simple clarification but can you ssh from N4F to WD *with* a password? If you can do that then it is absolutely going to work but I thought I'd check as it is possible to have the ssh client without having the server (sshd).

Ian
NAS4Free 10.3.0.3 (2617)
Intel Corporation D525MW 4GB
Promise SATA 300 TX4
HGST HTS721010A9E630 X 3 TOSHIBA MQ01ABD100 x2 + spare ZFS R5

User avatar
xr4t3d
Starter
Starter
Posts: 24
Joined: 26 Oct 2016 11:46
Location: Houston, TX, USA
Status: Offline

Re: How to Configuring Rsync ?

#12

Post by xr4t3d » 22 Nov 2016 15:12

Ian,

Yes, I am. However, when ssh'ing into the WDM from N4F, I have to specify the key algorithm.

I will be going on vacation, however, Ill try to post back when I can.


John
Thanks,

John

Linksys WRT 1900AC
DD-WRT v3.0-r28628 std (12/29/15)
Linux 3.18.25 #54 SMP Tue Dec 22 00:34:08 CET 2015 armv7l

10.3.0.3 - Pilingitam (revision 2987)
x64-embedded on Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
Dell Poweredge 2950 - RAM: 16 GB - 2 x 1 TB's (For now...)

User avatar
xr4t3d
Starter
Starter
Posts: 24
Joined: 26 Oct 2016 11:46
Location: Houston, TX, USA
Status: Offline

Re: How to Configuring Rsync ?

#13

Post by xr4t3d » 02 Dec 2016 07:25

Just a quick update.... so after thinking about this, I decided to take a different approach on this. I have somewhat figured out the SSH Keys (very handy by the way!). At this time, the WDC is currently acting as the primary and the N4F is the secondary. My future plans are to swap, N4F (primary and WDC (secondary). Currently, I'm in the process of just moving everything off of the WDC to the N4F. Once this has been completed, I will then figure out where I will run the rsync commands as a chron task.

I'll update more as I progress....


Thanks,

John
Thanks,

John

Linksys WRT 1900AC
DD-WRT v3.0-r28628 std (12/29/15)
Linux 3.18.25 #54 SMP Tue Dec 22 00:34:08 CET 2015 armv7l

10.3.0.3 - Pilingitam (revision 2987)
x64-embedded on Intel(R) Xeon(R) CPU E5410 @ 2.33GHz
Dell Poweredge 2950 - RAM: 16 GB - 2 x 1 TB's (For now...)

Post Reply

Return to “RSYNC”