*New 12.1 series Release:
2019-11-08: XigmaNAS 12.1.0.4.7091 - released!

*New 11.3 series Release:
2019-10-19: XigmaNAS 11.3.0.4.7014 - released


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPools...

XigmaNAS Scripts and shell tips
Forum rules
Set-Up GuideFAQsForum Rules
fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#31

Post by fritz » 22 Feb 2013 16:38

Hi macsimus,

I am currently working on a tutorial on hithub (it is not yet very detailed, but already more detailed than on my initial post): https://github.com/fritz-hh/scripts_NAS ... BHOW-TO%5D
1- tmp and log folders where should they be created exactly?
The easiest is to create 2 subfolders in folder where your scripts are located:
A sub folder called "tmp" and a subfolder called "log"
2- not sure how to update config.sh
How do you access your nas? (e.g. SSH using putty, CIFS...)
3. cron job script - where would that be located or needs to be created?
In your NAS4Free web interface, go to: System|Advanced|Cron

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#32

Post by Onichan » 23 Feb 2013 01:06

fritz: I tested the snapshot and scrub and they both worked for me, thanks for fixing them.

macsimus
NewUser
NewUser
Posts: 2
Joined: 22 Feb 2013 13:48
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#33

Post by macsimus » 23 Feb 2013 04:10

The guide seems easier to follow. I'll see what I can do. Thank you.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#34

Post by fritz » 23 Feb 2013 15:10

Hi macsimus,

I think you should have a look here, if you have trouble setting up sleep/wake-up:
viewtopic.php?f=70&t=2733

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#35

Post by Onichan » 01 Mar 2013 02:38

I seem to be getting email alerts for no reason. Here is the most recent one that I don't see any problem in the email

Code: Select all

20130228_200500	INFO	-------------------------------------
20130228_200500	INFO	Starting snapshot script for dataset "pool"
20130228_200500	INFO	Keeping up to 0 hourly / 90 daily / 0 weekly / 0 monthly snapshots (<0 = all)
20130228_200500	INFO	Currently, no need to create any snapshot for filesystem pool
20130228_200500	INFO	Currently, no need to create any snapshot for filesystem pool/anime
20130228_200500	INFO	Currently, no need to create any snapshot for filesystem pool/media
20130228_200500	INFO	Currently, no need to create any snapshot for filesystem pool/private
20130228_200500	INFO	Currently, no need to create any snapshot for filesystem pool/random
20130228_200500	INFO	Currently, no need to create any snapshot for filesystem pool/stuff
20130228_200500	INFO	Currently, no need to create any snapshot for filesystem pool/youtube
20130228_200500	INFO	Removing superfluous snapshots
20130228_200500	INFO	Analyzing snapshots with tag "type01"
20130228_200500	INFO	Analyzing snapshots with tag "type02"
20130228_200500	INFO	Analyzing snapshots with tag "type03"
20130228_200500	INFO	Analyzing snapshots with tag "type04"

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#36

Post by fritz » 01 Mar 2013 16:38

Hi Onichan,

Thank you for your message.
I was aware of the problem (I am experiencing the same one)
This is was last know issue I wanted to close for v1.0-stable (I already solved 3 other issues since v1.0-rc3)

As the problem occurs with all scripts, I am suspecting the issue in one of the common*.sh scripts
But until now I did not yet find the cause of this issue. I will analyze it next week.
If you have any idea, please do not hesitate to tell me.

Thanks,

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#37

Post by Onichan » 02 Mar 2013 00:10

Well I don't know bash, but are you sure line 212 in commonLockFcts.sh is correct? It seems like its saying execute main then set err_in_main=1 which you then later say if its not equal to 0 then return 2. Which then causes whatever script running to thing there was a problem.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#38

Post by fritz » 02 Mar 2013 15:12

Hi Onichan,

Yo probably mean this:

Code: Select all

! main && err_in_main=1
In fact this is the shortcut form of the conditional statement:

Code: Select all

if ! main; then
    err_in_main=1
fi
(see: http://www.pixelbeat.org/programming/sh ... takes.html )

It changes the value of err_in_main, if the return code of the main function is not "0" (i.e. if an error occured)
Therefore, this statement should be correct.

Regards

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#39

Post by Onichan » 02 Mar 2013 17:35

Ah that makes sense then, thanks. I am not sure whats wrong then as from the little I figured out about bash it looks fine. I do have 3 different snapshot jobs and only one is sending the error. The one for my main pool is the one emailing, but I have two for two different specific datasets on a different pool that isn't emailing. Also the scrub one isn't email either.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#40

Post by fritz » 04 Mar 2013 21:14

Hi Onichan,

The issue leading to spurious mails when running "checkSpace" (resp. "manageSnapshots") should now be solved in v1.0-rc4
This issue occured when starting two instances of the script at the same time (even for different ZFS filesystems)

The cause of issue was threefold:
  • The lock id was computed wrongly, so that it was the equal even for different filesystems
  • The lock acquisition was not atomic, this allowed sometimes 2 script instances to acquire the same lock if they were started simultaneously. This leaded to an issue when the script that took longer to run tried to release the lock (as the lock has already been released before).
  • Unfortunately the issue was not logged correctly due to another bug, so that the error mail that was sent because of the above mentionned issue did not contain the corresponding error message.
As the above mentionned issue required major changes in the script, I prefered to generate a new release candidate before I release the stable version.

It should now be possible to start several instances of checkSpace.sh / manageSnapshots.sh at the same time for different filesystems.
If several instances are started at the same time for the same filesystem, the second instance is now prevented from execution, and a dedicated error message is logged.

v1.0-rc4 solves also various other issues: see release notes here: https://github.com/fritz-hh/scripts_NAS ... E_NOTES.md
v1.0-rc4 can be downloaded here: https://github.com/fritz-hh/scripts_NAS4Free/tags

Regards,

fritz
Last edited by fritz on 05 Mar 2013 19:27, edited 1 time in total.
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#41

Post by Onichan » 05 Mar 2013 19:17

Thanks for always quickly fixing the issues, I'll update it tonight and let you know if I see any problems in a day or two.

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#42

Post by Onichan » 07 Mar 2013 16:27

So far they are working without any problems, thanks.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#43

Post by fritz » 12 Mar 2013 18:41

Dear all,

The first stable release of the scripts (v1.0) is available on github.
There is only 1 small bugfix since v1.0-rc4 (thank you hagai for the hint)
If you still use v1.0-rc3 or an older version, I strongly recommand you to update to v1.0


release notes: https://github.com/fritz-hh/scripts_NAS ... E_NOTES.md
v1.0 can be downloaded here: https://github.com/fritz-hh/scripts_NAS4Free/tags

Regards

fritz

PS: For the next stable release, it is planned to include zfs REMOTE replication over ssh in backupData.sh
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

dpynn
NewUser
NewUser
Posts: 11
Joined: 26 Feb 2013 18:11
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#44

Post by dpynn » 15 Mar 2013 13:25

Hi Fritz, great work on the scripts man. I really appreciate the time and effort you put into this. I was wondering if you had made any progress on getting the script to replicate snapshots to a remote nas4free server. I have 2 nas4free boxes and need to find a way to replicate snaps from our primary nas4free box to the secondary nas4free box. I don't think I am experienced enough to amend the scripts myself to do this. Anyhow just wondering... Thanks very much

dpatino1
NewUser
NewUser
Posts: 8
Joined: 04 Mar 2013 16:33
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#45

Post by dpatino1 » 15 Mar 2013 16:25

Hello,

Is there a way to get the backup script to backup from a ZFS pool to a mounted smb share?

willy.tenner
NewUser
NewUser
Posts: 1
Joined: 14 Mar 2013 13:59
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#46

Post by willy.tenner » 15 Mar 2013 17:10

Hi Fritz,
thank you for your wonderful script collection. I love it. Especially the script for shutting down the home NAS: manageAcpi.sh. A long time ago I wrote my own script for doing this job. In the past I often asked myself, if there are any real conditions that might prevent the NAS from shutting down, when no other parameters were specified. It is very difficult to answer this question because everybody has other contraints.

For my case at home I found some conditions. The NAS can be shutdown when:
1. no active TCP connections exists. This includes any active ssh, http(s), ftp, afpovertcp or netbios-ssn sessions. Furthermore no nfs or smb connections have to exist.
2. no special damons are running in more than once instance
3. no given programs are running
4. no zpool actions are running like resilvering or scrubbing.
5. no given client are reacheable. This function already exist in your implemention.

Last but not least the above conditions will be checked 10 times. If they always be true the NAS can shutdown.

I modified your manageAcpi.sh and config.sh a little bit (based on your v1.0 stable scripts). The following functions were added:
isBusy, isZpoolActive, isTCPconnections, isDamonActive, isRunning, trim
Some changes in the main.

And some addition in config.sh.

Have a look at it (see attachment) and let me know, if you are willing to add these feature into your source tree.

Kind regards,
willy
You do not have the required permissions to view the files attached to this post.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#47

Post by fritz » 15 Mar 2013 19:05

dpatino1 wrote:Is there a way to get the backup script to backup from a ZFS pool to a mounted smb share?
Hi dpatino1,

Unfortunatelly, the backupData.sh script does not support backing-up a ZFS pool to a mounted smb share.
The reason is that it relies on the "zfs send" and "zfs receive" commands.

Regards,
fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#48

Post by fritz » 15 Mar 2013 19:14

dpynn wrote:Hi Fritz, great work on the scripts man. I really appreciate the time and effort you put into this. I was wondering if you had made any progress on getting the script to replicate snapshots to a remote nas4free server. I have 2 nas4free boxes and need to find a way to replicate snaps from our primary nas4free box to the secondary nas4free box. I don't think I am experienced enough to amend the scripts myself to do this. Anyhow just wondering... Thanks very much
Hi dpynn,

Thank you for your feedback.
I plan to release a first beta version supporting remote replication in the next weeks.
(If you can't wait, you can already try the latest version of the 2.x branch on github, but it may be buggy. Please do not use it on a productive environment...
Do not hesitate to write problem reports on github)

Please note that you will need to have ssh public key authentication configured on the secondary box to run the script with the remote replication option enabled.

Regards,
fritz
Last edited by fritz on 16 Mar 2013 11:13, edited 1 time in total.
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#49

Post by fritz » 15 Mar 2013 20:08

Hi willy,
willy.tenner wrote:Hi Fritz,
thank you for your wonderful script collection. I love it. Especially the script for shutting down the home NAS: manageAcpi.sh.
Thank you for your feedback.
willy.tenner wrote:For my case at home I found some conditions. The NAS can be shutdown when:
1. no active TCP connections exists. This includes any active ssh, http(s), ftp, afpovertcp or netbios-ssn sessions. Furthermore no nfs or smb connections have to exist.
2. no special damons are running in more than once instance
3. no given programs are running
4. no zpool actions are running like resilvering or scrubbing.
5. no given client are reacheable. This function already exist in your implemention.

Last but not least the above conditions will be checked 10 times. If they always be true the NAS can shutdown.
It was clear to me that the script was not fulfillng everybodys' need as it is in v1.0, therefore propositions to enhance the script are always welcome.
(To ease collaboration, it would be nevertheless easier if you would make a fork in github, makethe required changes in v2.x and them make a pull request in github)

Few remarks / questions:
- It would be nice if the conditions could be activated or desactivated using arguments passed when calling the script (like I did it for the conditions that are already implemented)
- I am not sure to understand the second condition. Why "more than one instance"? Which use case do you have in mind?
- What is the difference between 3 and 4?
willy.tenner wrote:I modified your manageAcpi.sh and config.sh a little bit (based on your v1.0 stable scripts). The following functions were added:
isBusy, isZpoolActive, isTCPconnections, isDamonActive, isRunning, trim
Some changes in the main.

And some addition in config.sh.

Have a look at it (see attachment) and let me know, if you are willing to add these feature into your source tree.
I will have a look in the next days, and give you my feedback

Regards
fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#50

Post by fritz » 02 Apr 2013 19:57

dpynn wrote:Hi Fritz, great work on the scripts man. I really appreciate the time and effort you put into this. I was wondering if you had made any progress on getting the script to replicate snapshots to a remote nas4free server. I have 2 nas4free boxes and need to find a way to replicate snaps from our primary nas4free box to the secondary nas4free box. I don't think I am experienced enough to amend the scripts myself to do this. Anyhow just wondering... Thanks very much
Dear all,

A new version of the scripts is available on guthub: v2.0-beta1. The most notable change is that this beta version supports remote ZFS backup to another machine.
Please note that this new function requires SSH public key authentication to be setup on your server.
(Please do not use this beta version in a productive environment)
The remote backup was has not yet been tested extensively, I would be please to get feedback on this new function.

Release notes:

v2.0-beta1 (2013-03-25):
New features

backupData.sh: support of remote backup through ssh
backupData.sh: check that the pool of the destination filesystem is different from the pool of each destination filesystem
backupData.sh: (closes #19) new option -c to set compression for the destination filesystems
manageAcpi.sh: (closes #7) new switch -s to prevent shutdown in case of active ssh connection to the server
manageSnapshots.sh: new switch -r to process recursively the filesystems at and given depth

Changes

backupData.sh: (closes #8) allow any zfs fs as detination of the backup (not only a pool) (e.g. not only "backup_pool" but also "backup_pool/abc" are now supported)
manageAcpi.sh: signature of -n switch changed (before [-n beg,end,acpi,delay,ips], after [-n ips,delay,acpi])
reportFromLogs.sh: mention package version in report

Fixes

None
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

dpynn
NewUser
NewUser
Posts: 11
Joined: 26 Feb 2013 18:11
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#51

Post by dpynn » 04 Apr 2013 16:46

Hi Fritz,

Thanks for the updated scripts, I cant begin to tell you how much this will help. The snapshots are copying over to a remote system now. The scripts seem to work. There are a couple of things I am not sure about however.

When executing the script as follows:
./backupData.sh -s root@x.x.x.x nas1-pool/nas1-dataset nas2-pool/nas2-dataset

The snapshots copy over no problem, however the terminal outputs the following error:
“Pseudo-terminal will not be allocated because stdin is not a terminal.”

And

After the script is run and the snapshots are copied over, nas2 shows several new mounts ( from nas1) as mounted locally. I believe that the source file system(s) should be un-mounted after the copy is complete, but I am not certain about that. These mounts below in RED are new and show up after running the script. Is this normal behavior?

nas2:~# mount
/dev/ufsid/512b6bc6877c398e on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
procfs on /proc (procfs, local)
nas2-pool on /mnt/nas2-pool (zfs, local, read-only, nfsv4acls)
nas2-pool/nas2-dataset on /mnt/nas2-pool/nas2-dataset (zfs, local, read-only, nfsv4acls)
nas2-pool/nas2-dataset on /mnt/nas2-pool/nas2-dataset (zfs, local, nfsv4acls)
nas2-pool/nas2-dataset/nas2-dataset on /mnt/nas2-pool/nas2-dataset/nas2-dataset (zfs, local, nfsv4acls)
nas2-pool/nas2-pool on /mnt/nas2-pool/nas2-pool (zfs, local, read-only, nfsv4acls)
nas2-pool/nas2-pool/nas2-dataset on /mnt/nas2-pool/nas2-pool/nas2-dataset (zfs, local, read-only, nfsv4acls)
nas2-pool/nas2-dataset/nas2-pool on /mnt/nas2-pool/nas2-dataset/nas2-pool (zfs, local, nfsv4acls)
nas2-pool/nas2-dataset/nas2-pool/nas2-dataset on /mnt/nas2-pool/nas2-dataset/nas2-pool/nas2-dataset (zfs, local, nfsv4acls)


Thanks Fritz, great stuff!

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#52

Post by fritz » 04 Apr 2013 18:37

Hi dpynn,

Are you sure there aren't typos in the mounts mentioned in red?
I would expect to see "nas1-pool" and "nas1-dataset" at several places.

Fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

dpynn
NewUser
NewUser
Posts: 11
Joined: 26 Feb 2013 18:11
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#53

Post by dpynn » 04 Apr 2013 19:23

Sorry Fritz, I was screwing around with another config and copied the wrong output.

This is the correct mount output now:

bunas2:/# mount
/dev/ufsid/512b6bc6877c398e on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
procfs on /proc (procfs, local)
bunas2-pool on /mnt/bunas2-pool (zfs, local, read-only, nfsv4acls)
bunas2-pool/bunas1-dataset on /mnt/bunas2-pool/bunas1-dataset (zfs, local, read-only, nfsv4acls)
bunas2-pool/bunas2-dataset on /mnt/bunas2-pool/bunas2-dataset (zfs, local, nfsv4acls)
bunas2-pool/bunas2-dataset/bunas1-dataset on /mnt/bunas2-pool/bunas2-dataset/bunas1-dataset (zfs, local, nfsv4acls)
bunas2-pool/bunas1-pool on /mnt/bunas2-pool/bunas1-pool (zfs, local, read-only, nfsv4acls)
bunas2-pool/bunas1-pool/bunas1-dataset on /mnt/bunas2-pool/bunas1-pool/bunas1-dataset (zfs, local, read-only, nfsv4acls)
bunas2-pool/bunas2-dataset/bunas1-pool on /mnt/bunas2-pool/bunas2-dataset/bunas1-pool (zfs, local, nfsv4acls)
bunas2-pool/bunas2-dataset/bunas1-pool/bunas1-dataset on /mnt/bunas2-pool/bunas2-dataset/bunas1-pool/bunas1-dataset (zfs, local, nfsv4acls)
bunas2:/#

Thanks again

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#54

Post by fritz » 04 Apr 2013 20:12

Hi dpynn,

To make clear what the script does (the header of the script explains it too)

If you call:
./backupData.sh -s root@x.x.x.x "bunas1-pool/bunas1-dataset" "bunas2-pool/bunas2-dataset"

The scripts makes a remote replication of the zfs fs "bunas1-pool/bunas1-dataset" (available on your local nas) to the remote nas below the zfs fs "bunas2-pool/bunas2-dataset"
- 1st the script creates the zfs fs "bunas2-pool/bunas2-dataset/bunas1-pool" and "bunas2-pool/bunas2-dataset/bunas1-pool/bunas1-dataset" on your remote machine (if they do not yet exist) (if "bunas1-pool/bunas1-dataset", contains sub-filesystems on your local nas, thoses fs are also created below "bunas2-pool/bunas2-dataset/bunas1-pool/bunas1-dataset" on your remote nas)
- then it perform a remote replication of all the snapshots available in the filesystem "bunas1-pool/bunas1-dataset" (incl the snapshots available in the subfilesystems if any) to the corresponding filesystems in your remote nas. (if some of the snapshots are already available in the remote nas, it will only replicate the newer ones that are not yet available in the remote nas)

At the end of the script you should have the following zfs fs on your remote nas (Suposing that "bunas1-pool/bunas1-dataset" does not have any sub-filesystem):
- bunas2-pool
- bunas2-pool/bunas2-dataset
- bunas2-pool/bunas2-dataset/bunas1-pool
- bunas2-pool/bunas2-dataset/bunas1-pool/bunas1-dataset
This is why you see the corresponding mounts on your destination nas
I believe that the source file system(s) should be un-mounted after the copy is complete
There is a misunderstanding: These mount are not mounts of the source filesystem, but are zfs fs available on your destination nas and containing the replicated data.

I can see in your mount list, that there are more zfs created on the remote nas than the 4 zfs fs I mentionned above:

I assume that the following mounts are due to the fact that you also called at least once:

Code: Select all

./backupData.sh -s root@x.x.x.x "bunas1-pool/bunas1-dataset" "bunas2-pool"
bunas2-pool/bunas1-pool on /mnt/bunas2-pool/bunas1-pool (zfs, local, read-only, nfsv4acls)
bunas2-pool/bunas1-pool/bunas1-dataset on /mnt/bunas2-pool/bunas1-pool/bunas1-dataset (zfs, local, read-only, nfsv4acls)
It is not clear to me why you have also the following mount:
bunas2-pool/bunas1-dataset on /mnt/bunas2-pool/bunas1-dataset (zfs, local, read-only, nfsv4acls)
bunas2-pool/bunas2-dataset/bunas1-dataset on /mnt/bunas2-pool/bunas2-dataset/bunas1-dataset (zfs, local, nfsv4acls)
Is it possible that you have created them manually?
The snapshots copy over no problem, however the terminal outputs the following error:
“Pseudo-terminal will not be allocated because stdin is not a terminal.”
I do also experience this error message. It seems that it does not have any functional consequence.
I will analyze it until the final release. Any hint would be welcome.

Regards

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

dpynn
NewUser
NewUser
Posts: 11
Joined: 26 Feb 2013 18:11
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#55

Post by dpynn » 04 Apr 2013 20:42

Fritz you were correct. Seems like I had left-overs from other scripts I was trying. So, I completely blew away my vdev, pool and dataset on bunas2 - and then recreated them. Now after running the script from bunas1 the following mounts show:

bunas2:~# mount
/dev/ufsid/512b6bc6877c398e on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
procfs on /proc (procfs, local)
bunas2-pool on /mnt/bunas2-pool (zfs, local, read-only, nfsv4acls)
bunas2-pool/bunas2-dataset on /mnt/bunas2-pool/bunas2-dataset (zfs, local, nfsv4acls)
bunas2-pool/bunas2-dataset/bunas1-pool on /mnt/bunas2-pool/bunas2-dataset/bunas1-pool (zfs, local, nfsv4acls)
bunas2-pool/bunas2-dataset/bunas1-pool/bunas1-dataset on /mnt/bunas2-pool/bunas2-dataset/bunas1-pool/bunas1-dataset (zfs, local, nfsv4acls)

Is this what we would expect to see? I am guessing so as you indicated that above. Also, I agree that the “Pseudo-terminal will not be allocated because stdin is not a terminal.” message has no negative effects. We will continue to work it however and if we find anything I will post back to you.

Again, thanks Fritz we really appreciate this.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#56

Post by fritz » 04 Apr 2013 20:51

Is this what we would expect to see?
Yes

Please do not forget that this is a beta version, so I strongly recommand not to use it in a productive environment.

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

dpynn
NewUser
NewUser
Posts: 11
Joined: 26 Feb 2013 18:11
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#57

Post by dpynn » 04 Apr 2013 21:01

fritz wrote:
Is this what we would expect to see?
Yes

Please do not forget that this is a beta version, so I strongly recommand not to use it in a productive environment.

fritz

Understood. Thanks

xuesheng
Starter
Starter
Posts: 64
Joined: 23 Jun 2012 10:56
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#58

Post by xuesheng » 06 Apr 2013 17:13

Recently I added a second NAS4Free system (with more disk space than my original NAS) to my network and have started using it to provide extra storage for each user.

Yesterday I installed the v2-beta1 version of fritz's scripts and used backupData.sh to backup the original NAS to the new NAS.

The remote NAS only has one pool so I used the GUI to create a "backup" dataset (remote-pool/backup) on the remote NAS and used that as the destination when I ran the backupData.sh script.

The backup script seems to have worked without any errors, apart from some "Pseudo terminal" messages.

However the script has set the single pool on the remote NAS to readonly mode. This means I could no longer use that NAS for "ordinary" storage (it has some folders for each user).

I've used "zfs set readonly=off remote-pool" to enable write access to the only pool on the NAS (remote-pool) and "zfs set readonly=on remote-pool/backup" to make the dataset used by the backupData.sh script readonly again. Now I need to try modifying the script so that if the destination is a dataset then only that dataset is made readonly.

This is a very useful collection of scripts!

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#59

Post by fritz » 06 Apr 2013 22:31

Hi,

Thanks for bringing this to my notice (I did not consider this use case)
Do not hesitate to send me a patch (ideally on github)

Fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

User avatar
lindsay
Forum Moderator
Forum Moderator
Posts: 308
Joined: 23 Jun 2012 09:59
Location: Steinkjer,Norway
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#60

Post by lindsay » 07 Apr 2013 14:04

I find these scipts useful and added for me also a script wich is optimizing all tables in MySQL.
MySQL i use to have XBMC always synced to the clients
Protected by smoothiebox Red,Green,
Purple,Orange Zones/VLAN`s
Powered by AMD A10-6700T


XigmaNAS Box-1 11.2.0.4 - Omnius (revision 6625)
Platform : x64-embedded on 2X Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Motherboard: ASUS Z10PA-D8, 2xSocket-2011-3
SATA Controllers : 1X Avago Technologies (LSI) SAS2008 and 1x Avago Technologies (LSI) SAS2308
Pool 1 (Media-Pool) 8X4TB in raidz2
Pool 2 (Media-Pool-2) 4X2TB in raidz2 and 2X2TB in mirror mirror and 2X3TB in mirror
Pool 3 (Media-Pool-3) 2X2TB in mirror and 2X4TB in mirror and 2X1TB in mirror

Post Reply

Return to “Scripts and shell tips”