*New 11.3 series Release:
2019-10-19: XigmaNAS 11.3.0.4.7014 - released

*New 12.0 series Release:
2019-10-05: XigmaNAS 12.0.0.4.6928 - released!

*New 11.2 series Release:
2019-09-23: XigmaNAS 11.2.0.4.6881 - released!

We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPools...

XigmaNAS Scripts and shell tips
Forum rules
Set-Up GuideFAQsForum Rules
xuesheng
Starter
Starter
Posts: 64
Joined: 23 Jun 2012 10:56
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#61

Post by xuesheng » 08 Apr 2013 14:31

fritz wrote:Do not hesitate to send me a patch (ideally on github)
I am testing some changes I made to backupData.sh to fix the problem I reported.

I have never used github before so I am not sure how to send you a patch on github

rampie
NewUser
NewUser
Posts: 13
Joined: 12 Jun 2013 12:03
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#62

Post by rampie » 14 Jun 2013 15:25

Hey Fritz,

I checked your scripts, but i want to append the information on my daile e-mail report. All i need to know is the free space of the zfs pools (i got 2 of them) and the amount of available RAM.

I already looked into the checkspace.sh you created but it points back to $BIN_ZFS and then calls the getAvailable.

Is there a possibility the script runs through all ZFS-pools and replies me their disk space, AND everything in 1 script in a few lines and not 190 (i know this includes logging + mailing..)?

Any ideas for the RAM issue?

User avatar
kkd
Forum Moderator
Forum Moderator
Posts: 303
Joined: 08 May 2013 21:47
Location: Budapest, Hungary
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#63

Post by kkd » 14 Jun 2013 15:54

rampie wrote:Hey Fritz,

I checked your scripts, but i want to append the information on my daile e-mail report. All i need to know is the free space of the zfs pools (i got 2 of them) and the amount of available RAM.

I already looked into the checkspace.sh you created but it points back to $BIN_ZFS and then calls the getAvailable.

Is there a possibility the script runs through all ZFS-pools and replies me their disk space, AND everything in 1 script in a few lines and not 190 (i know this includes logging + mailing..)?

Any ideas for the RAM issue?
viewtopic.php?f=70&t=4142#p22233
XigmaNAS x64-embedded | INTEL Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz | 9216MiB RAM | X x YTB WD Red ZFS mirror compressed

neptunus
experienced User
experienced User
Posts: 86
Joined: 11 Jun 2013 08:50
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#64

Post by neptunus » 21 Jun 2013 15:44

I was reading the issue page on wiki fritz-hh. I came across:

https://github.com/fritz-hh/scripts_NAS ... state=open
Issue 2
Lock files not delete in case of abnormal termination
Some script create lock files when starting (e.g. backup.sh)
If the script stop abnormally, the lock file is not deleted. As a consequense, the script will fail to start the next time it is called.

Maybe the information below can be of help (It was of help to me):
http://stackoverflow.com/questions/1440 ... dy-running

Xell1984
NewUser
NewUser
Posts: 9
Joined: 10 Jul 2013 19:26
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#65

Post by Xell1984 » 12 Jul 2013 12:30

Hi there,

i am new to FreeBSD/Nas4free and have a question.

is it for the snapshot only the command: /mnt/Daten1/Apps/manageSnapshots.sh -k Daten1 or do i need something else?
Where does he save those snapshots?

My logfile say:

Code: Select all

20130711_190147	INFO	-------------------------------------
20130711_190147	INFO	Starting snapshot script for dataset "Daten1"
20130711_190147	INFO	Keeping up to -1 hourly / -1 daily / -1 weekly / -1 monthly snapshots (<0 = all)
20130711_190147	INFO	Creating new snapshot "Daten1@20130711_1901_autosnap_type04"
20130711_190148	INFO	Removing superfluous snapshots
20130711_190148	INFO	Analyzing snapshots with tag "type01"
20130711_190148	INFO	Daten1 (and sub-filesystems): All snapshots with tag "type01" kept
20130711_190148	INFO	Analyzing snapshots with tag "type02"
20130711_190148	INFO	Daten1 (and sub-filesystems): All snapshots with tag "type02" kept
20130711_190148	INFO	Analyzing snapshots with tag "type03"
20130711_190148	INFO	Daten1 (and sub-filesystems): All snapshots with tag "type03" kept
20130711_190148	INFO	Analyzing snapshots with tag "type04"
20130711_190148	INFO	Daten1 (and sub-filesystems): All snapshots with tag "type04" kept

User avatar
kkd
Forum Moderator
Forum Moderator
Posts: 303
Joined: 08 May 2013 21:47
Location: Budapest, Hungary
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#66

Post by kkd » 12 Jul 2013 12:42

snapshot is not saved. u can clone it via webgui.
snapshot is created to watch if files are modified or not, so its just a mark on a file.
XigmaNAS x64-embedded | INTEL Intel(R) Core(TM) i3-2100 CPU @ 3.10GHz | 9216MiB RAM | X x YTB WD Red ZFS mirror compressed

egrimisu
NewUser
NewUser
Posts: 1
Joined: 23 Jul 2013 11:43
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#67

Post by egrimisu » 23 Jul 2013 11:45

Great work to bring these functions, i was wondering if these are any good for ubuntu, i have zfs on ubuntu.

trendco
Starter
Starter
Posts: 70
Joined: 20 Jan 2013 18:59
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#68

Post by trendco » 14 Aug 2013 14:50

Hey Fritz,

very good work, thanks a lot for these great Scripts!
Will be further developed at the Scripts? I ask, because 2.x is in Beta since 5 Months.

It would be great if all these things could be managed via the Web Interface.


Alex

neptunus
experienced User
experienced User
Posts: 86
Joined: 11 Jun 2013 08:50
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#69

Post by neptunus » 23 Aug 2013 14:45

Hello Fritz and others,

I have two questions.

ONE
Is it possible to use manageSnapshots.sh only ones a day? I only need:
- 15 daily snapshots
- 8 weekly snapshots
- 12 monthly snapshots

TWO
I also question my self why not using: "Disks|ZFS|Snapshots|Auto Snapshot"

Onichan
Advanced User
Advanced User
Posts: 237
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#70

Post by Onichan » 24 Aug 2013 00:02

neptunus wrote:Hello Fritz and others,

I have two questions.

ONE
Is it possible to use manageSnapshots.sh only ones a day? I only need:
- 15 daily snapshots
- 8 weekly snapshots
- 12 monthly snapshots

TWO
I also question my self why not using: "Disks|ZFS|Snapshots|Auto Snapshot"
Yes you can just run the script once a day if you want and it will work. For example for your requirements it would be like "manageSnapshots.sh -h 0 -d 15 -w 8 -m 12 pool"

The purpose of this script is it has more advanced options like having weekly/monthly snapshots, the built in one is only daily. Plus it is far easier to configure more advanced snapshot setup like if you wanted multiple hourly and daily that would require many separately configured auto snapshots. Also it is able to detect if a required snapshot was unable to be taken when it was supposed to and take one. For example say you run the script hourly and have it configured to keep 7 dailys, but you lost power for the first 12 hours of the day. Well when it gets power again it will take the snapshot for the day. Whereas if you were just running the built in one and it was suppose to take one during the time you had no power then you would never get a snapshot for that day.

joopa
NewUser
NewUser
Posts: 3
Joined: 19 Jul 2013 08:42
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#71

Post by joopa » 22 Sep 2013 12:14

Hi Fritz,

Excellent work, thank you.

For now I do not use your scripts snapshots, but it will come, for sure.
I focused on the wakeup of the NAS and its extinction with manageAcpi.sh. I use scripts in beta 2.

Your scripts are very well structured, I think that after the implementation of the test ssh connection, additional functionality can be integrated without too much difficulty.
As dnar, I would implement :
- Test for sockets and therefore access. For example, samba shares, the UPnP
- Test for process (not my priority)

Where are you in your development ?
Have you ever considered these features ?
If so, should I wait or launch into these new developments ?
NAS4Free 9.1.0.1 (revision 847) x64-embedded
E45M1-M PRO - RAM 8Go - Network 1Gb/s - Nas4Free on USB 8Go - pool 4x3To SATA RaidZ1

joopa
NewUser
NewUser
Posts: 3
Joined: 19 Jul 2013 08:42
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#72

Post by joopa » 22 Sep 2013 12:29

sorry not dnar, this is chrikai's script for the features sockets and process
http://www.forums.nas4free.org/viewtopi ... =70&t=4830
NAS4Free 9.1.0.1 (revision 847) x64-embedded
E45M1-M PRO - RAM 8Go - Network 1Gb/s - Nas4Free on USB 8Go - pool 4x3To SATA RaidZ1

neptunus
experienced User
experienced User
Posts: 86
Joined: 11 Jun 2013 08:50
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#73

Post by neptunus » 27 Sep 2013 22:30

Is it possible to use manageSnapshots.sh and backupData.sh not on a ZFS Pools but on a ZFS Datasets?

I only need the Snapshots on userA and userB.

Code: Select all

NAME             USED    AVAIL  REFER  MOUNTPOINT
zssddata               61.9G   622M   304K   /zssddata
zstore                 6.39G   13.6T  420K   /zstore
zstore/userA           329K    13.6T  329K   /zstore/userA
zstore/userB           1003M   13.6T  1003M  /zstore/userB
zstore/series          329K    13.6T  329K   /zstore/series
zstore/movies          5.36G   13.6T  5.36G  /zstore/movies

Onichan
Advanced User
Advanced User
Posts: 237
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#74

Post by Onichan » 28 Sep 2013 02:34

I know it's possible with manageSnapshots.sh as I do it, here is what I run "manageSnapshots.sh -h 0 -d 14 -w 0 -m 0 temp/public" and it only takes one of the public dataset.

neptunus
experienced User
experienced User
Posts: 86
Joined: 11 Jun 2013 08:50
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#75

Post by neptunus » 28 Sep 2013 14:39

Onichan wrote:I know it's possible with manageSnapshots.sh as I do it, here is what I run "manageSnapshots.sh -h 0 -d 14 -w 0 -m 0 temp/public" and it only takes one of the public dataset.
Thanks

I run "manageSnapshots.sh -k zstore/roy " every hour.
Image

I get to following error:

Code: Select all

20130928_140000 INFO    -------------------------------------
20130928_140000 INFO    Starting snapshot script for dataset "zstore/roy" (depth: -1)
20130928_140000 INFO    Keeping up to -1 hourly / -1 daily / -1 weekly / -1 monthly snapshots (<0 = all)
20130928_140000 INFO    zstore/roy: Creating new snapshot "zstore/roy@20130928_1400_autosnap_type01"
20130928_140000 INFO    zstore/roy: All snapshots with tag "type01" kept
20130928_140000 INFO    zstore/roy: All snapshots with tag "type02" kept
20130928_140000 INFO    zstore/roy: All snapshots with tag "type03" kept
20130928_140000 INFO    zstore/roy: All snapshots with tag "type04" kept
20130928_140000 INFO    zstore/roy@20130928_0900_autosnap_type04: Creating new snapshot "zstore/roy@20130928_0900_autosnap_type04@20130928_1400_autosnap_type03"
20130928_140000 ERROR   zstore/roy@20130928_0900_autosnap_type04: Problem while creating snapshot zstore/roy@20130928_0900_autosnap_type04@20130928_1400_autosnap_type03 (A snapshot having the same name may already exist)
20130928_140000 INFO    zstore/roy@20130928_0900_autosnap_type04: All snapshots with tag "type01" kept
20130928_140000 INFO    zstore/roy@20130928_0900_autosnap_type04: All snapshots with tag "type02" kept
20130928_140000 INFO    zstore/roy@20130928_0900_autosnap_type04: All snapshots with tag "type03" kept
20130928_140000 INFO    zstore/roy@20130928_0900_autosnap_type04: All snapshots with tag "type04" kept
20130928_140000 INFO    zstore/roy@20130928_1000_autosnap_type03: Creating new snapshot "zstore/roy@20130928_1000_autosnap_type03@20130928_1400_autosnap_type04"
20130928_140000 ERROR   zstore/roy@20130928_1000_autosnap_type03: Problem while creating snapshot zstore/roy@20130928_1000_autosnap_type03@20130928_1400_autosnap_type04 (A snapshot having the same name may already exist)
20130928_140000 INFO    zstore/roy@20130928_1000_autosnap_type03: All snapshots with tag "type01" kept
20130928_140000 INFO    zstore/roy@20130928_1000_autosnap_type03: All snapshots with tag "type02" kept
20130928_140000 INFO    zstore/roy@20130928_1000_autosnap_type03: All snapshots with tag "type03" kept
20130928_140000 INFO    zstore/roy@20130928_1000_autosnap_type03: All snapshots with tag "type04" kept
20130928_140000 INFO    zstore/roy@20130928_1100_autosnap_type02: Creating new snapshot "zstore/roy@20130928_1100_autosnap_type02@20130928_1400_autosnap_type04"
20130928_140000 ERROR   zstore/roy@20130928_1100_autosnap_type02: Problem while creating snapshot zstore/roy@20130928_1100_autosnap_type02@20130928_1400_autosnap_type04 (A snapshot having the same name may already exist)
20130928_140000 INFO    zstore/roy@20130928_1100_autosnap_type02: All snapshots with tag "type01" kept
20130928_140000 INFO    zstore/roy@20130928_1100_autosnap_type02: All snapshots with tag "type02" kept
20130928_140000 INFO    zstore/roy@20130928_1100_autosnap_type02: All snapshots with tag "type03" kept
20130928_140000 INFO    zstore/roy@20130928_1100_autosnap_type02: All snapshots with tag "type04" kept
20130928_140000 INFO    zstore/roy@20130928_1200_autosnap_type01: Creating new snapshot "zstore/roy@20130928_1200_autosnap_type01@20130928_1400_autosnap_type04"
20130928_140000 ERROR   zstore/roy@20130928_1200_autosnap_type01: Problem while creating snapshot zstore/roy@20130928_1200_autosnap_type01@20130928_1400_autosnap_type04 (A snapshot having the same name may already exist)
20130928_140000 INFO    zstore/roy@20130928_1200_autosnap_type01: All snapshots with tag "type01" kept
20130928_140000 INFO    zstore/roy@20130928_1200_autosnap_type01: All snapshots with tag "type02" kept
20130928_140000 INFO    zstore/roy@20130928_1200_autosnap_type01: All snapshots with tag "type03" kept
20130928_140001 INFO    zstore/roy@20130928_1200_autosnap_type01: All snapshots with tag "type04" kept
20130928_140001 INFO    zstore/roy@20130928_1300_autosnap_type01: Creating new snapshot "zstore/roy@20130928_1300_autosnap_type01@20130928_1400_autosnap_type04"
20130928_140001 ERROR   zstore/roy@20130928_1300_autosnap_type01: Problem while creating snapshot zstore/roy@20130928_1300_autosnap_type01@20130928_1400_autosnap_type04 (A snapshot having the same name may already exist)
20130928_140001 INFO    zstore/roy@20130928_1300_autosnap_type01: All snapshots with tag "type01" kept
20130928_140001 INFO    zstore/roy@20130928_1300_autosnap_type01: All snapshots with tag "type02" kept
20130928_140001 INFO    zstore/roy@20130928_1300_autosnap_type01: All snapshots with tag "type03" kept
20130928_140001 INFO    zstore/roy@20130928_1300_autosnap_type01: All snapshots with tag "type04" kept
Is there somthing I do wrong?

Onichan
Advanced User
Advanced User
Posts: 237
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#76

Post by Onichan » 28 Sep 2013 20:19

Hmm I am not sure, maybe there is something wrong with the 2.0b1? I am using v1.0 so try using that and see if it works. Your log does seem off as the name seems quite long like it is trying to create one snapshot with the name of multiple ones. For example here is my log from a snapshot

Code: Select all

20130925_001000 INFO    -------------------------------------
20130925_001000 INFO    Starting snapshot script for dataset "temp/public"
20130925_001000 INFO    Keeping up to 0 hourly / 14 daily / 0 weekly / 0 monthly snapshots (<0 = all)
20130925_001000 INFO    Creating new snapshot "temp/public@20130925_0010_autosnap_type02"
20130925_001000 INFO    Removing superfluous snapshots
20130925_001000 INFO    Analyzing snapshots with tag "type01"
20130925_001001 INFO    Analyzing snapshots with tag "type02"
20130925_001001 INFO    Snapshot "temp/public@20130911_0010_autosnap_type02" DELETED
20130925_001001 INFO    Analyzing snapshots with tag "type03"
20130925_001001 INFO    Analyzing snapshots with tag "type04"
20130925_201000 INFO    -------------------------------------

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#77

Post by fritz » 29 Sep 2013 12:19

Hi Neptunus,

Indeed, it seems that there is a bug in the script (probably not only in v2.0-beta1), as the script tries to create the snapshot "zstore/roy@20130928_0900_autosnap_type04@20130928_1400_autosnap_type03" that contains 2 "@" characters.
The issue is probably in the "main" function. It seems that "for subfilesystem in `$BIN_ZFS list -H $depth_flag -o name $I_FILESYSTEM`; do" itterates not only the subfilesystems of the fs given as parameter, but also its snapshots in special cases

Could you please tell me which version of zfs / nas4free you are running? The content of the "Disks|ZFS|Datasets|Information" in the webinterface of nas4free would help too

Thanks

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

neptunus
experienced User
experienced User
Posts: 86
Joined: 11 Jun 2013 08:50
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#78

Post by neptunus » 29 Sep 2013 15:41

fritz wrote:Hi Neptunus,

Indeed, it seems that there is a bug in the script (probably not only in v2.0-beta1), as the script tries to create the snapshot "zstore/roy@20130928_0900_autosnap_type04@20130928_1400_autosnap_type03" that contains 2 "@" characters.
The issue is probably in the "main" function. It seems that "for subfilesystem in `$BIN_ZFS list -H $depth_flag -o name $I_FILESYSTEM`; do" itterates not only the subfilesystems of the fs given as parameter, but also its snapshots in special cases

Could you please tell me which version of zfs / nas4free you are running? The content of the "Disks|ZFS|Datasets|Information" in the webinterface of nas4free would help too

Thanks

fritz
Below the version information you asked. Let me know if you need a tester!

Nas4free: 9.1.0.1 - Sandstorm (revision 804)

ZFS dataset information and status

Code: Select all

NAME                    USED  AVAIL  REFER  MOUNTPOINT
zssddata               61.9G   622M   304K  /zssddata
zstore                 6.34G  13.6T   420K  /zstore
zstore/annelies         329K  13.6T   329K  /zstore/annelies
zstore/movies          5.36G  13.6T  5.36G  /zstore/movies
zstore/roy             1003M  13.6T  1003M  /zstore/roy
zstore/royandannelies   329K  13.6T   329K  /zstore/royandannelies
zstore/series           329K  13.6T   329K  /zstore/series

Code: Select all

ZFS dataset properties

NAME                   PROPERTY              VALUE                   SOURCE
zssddata               type                  filesystem              -
zssddata               creation              Mon Aug 12 21:18 2013   -
zssddata               used                  61.9G                   -
zssddata               available             622M                    -
zssddata               referenced            304K                    -
zssddata               compressratio         2.55x                   -
zssddata               mounted               yes                     -
zssddata               quota                 none                    default
zssddata               reservation           none                    default
zssddata               recordsize            128K                    default
zssddata               mountpoint            /zssddata               default
zssddata               sharenfs              off                     default
zssddata               checksum              fletcher4               local
zssddata               compression           lzjb                    local
zssddata               atime                 off                     local
zssddata               devices               on                      default
zssddata               exec                  on                      default
zssddata               setuid                on                      default
zssddata               readonly              off                     default
zssddata               jailed                off                     default
zssddata               snapdir               hidden                  default
zssddata               aclmode               discard                 default
zssddata               aclinherit            restricted              default
zssddata               canmount              on                      default
zssddata               xattr                 off                     temporary
zssddata               copies                1                       default
zssddata               version               5                       -
zssddata               utf8only              off                     -
zssddata               normalization         none                    -
zssddata               casesensitivity       sensitive               -
zssddata               vscan                 off                     default
zssddata               nbmand                off                     default
zssddata               sharesmb              off                     default
zssddata               refquota              none                    default
zssddata               refreservation        none                    default
zssddata               primarycache          all                     default
zssddata               secondarycache        all                     default
zssddata               usedbysnapshots       0                       -
zssddata               usedbydataset         304K                    -
zssddata               usedbychildren        61.9G                   -
zssddata               usedbyrefreservation  0                       -
zssddata               logbias               latency                 default
zssddata               dedup                 off                     default
zssddata               mlslabel                                      -
zssddata               sync                  standard                default
zssddata               refcompressratio      1.53x                   -
zssddata               written               304K                    -
zstore                 type                  filesystem              -
zstore                 creation              Mon Aug 12 21:18 2013   -
zstore                 used                  6.34G                   -
zstore                 available             13.6T                   -
zstore                 referenced            420K                    -
zstore                 compressratio         1.00x                   -
zstore                 mounted               yes                     -
zstore                 quota                 none                    default
zstore                 reservation           none                    default
zstore                 recordsize            128K                    default
zstore                 mountpoint            /zstore                 default
zstore                 sharenfs              off                     default
zstore                 checksum              fletcher4               local
zstore                 compression           lzjb                    local
zstore                 atime                 off                     local
zstore                 devices               on                      default
zstore                 exec                  on                      default
zstore                 setuid                on                      default
zstore                 readonly              off                     default
zstore                 jailed                off                     default
zstore                 snapdir               hidden                  default
zstore                 aclmode               discard                 default
zstore                 aclinherit            restricted              default
zstore                 canmount              on                      default
zstore                 xattr                 off                     temporary
zstore                 copies                1                       default
zstore                 version               5                       -
zstore                 utf8only              off                     -
zstore                 normalization         none                    -
zstore                 casesensitivity       sensitive               -
zstore                 vscan                 off                     default
zstore                 nbmand                off                     default
zstore                 sharesmb              off                     default
zstore                 refquota              none                    default
zstore                 refreservation        none                    default
zstore                 primarycache          all                     default
zstore                 secondarycache        all                     default
zstore                 usedbysnapshots       0                       -
zstore                 usedbydataset         420K                    -
zstore                 usedbychildren        6.34G                   -
zstore                 usedbyrefreservation  0                       -
zstore                 logbias               latency                 default
zstore                 dedup                 off                     default
zstore                 mlslabel                                      -
zstore                 sync                  standard                default
zstore                 refcompressratio      1.00x                   -
zstore                 written               420K                    -
zstore/annelies        type                  filesystem              -
zstore/annelies        creation              Thu Aug 22 20:14 2013   -
zstore/annelies        used                  329K                    -
zstore/annelies        available             13.6T                   -
zstore/annelies        referenced            329K                    -
zstore/annelies        compressratio         1.00x                   -
zstore/annelies        mounted               yes                     -
zstore/annelies        quota                 none                    default
zstore/annelies        reservation           none                    default
zstore/annelies        recordsize            128K                    default
zstore/annelies        mountpoint            /zstore/annelies        default
zstore/annelies        sharenfs              off                     default
zstore/annelies        checksum              fletcher4               inherited from zstore
zstore/annelies        compression           lzjb                    local
zstore/annelies        atime                 off                     local
zstore/annelies        devices               on                      default
zstore/annelies        exec                  on                      default
zstore/annelies        setuid                on                      default
zstore/annelies        readonly              off                     default
zstore/annelies        jailed                off                     default
zstore/annelies        snapdir               hidden                  default
zstore/annelies        aclmode               discard                 default
zstore/annelies        aclinherit            restricted              default
zstore/annelies        canmount              on                      default
zstore/annelies        xattr                 off                     temporary
zstore/annelies        copies                1                       default
zstore/annelies        version               5                       -
zstore/annelies        utf8only              off                     -
zstore/annelies        normalization         none                    -
zstore/annelies        casesensitivity       sensitive               -
zstore/annelies        vscan                 off                     default
zstore/annelies        nbmand                off                     default
zstore/annelies        sharesmb              off                     default
zstore/annelies        refquota              none                    default
zstore/annelies        refreservation        none                    default
zstore/annelies        primarycache          all                     default
zstore/annelies        secondarycache        all                     default
zstore/annelies        usedbysnapshots       0                       -
zstore/annelies        usedbydataset         329K                    -
zstore/annelies        usedbychildren        0                       -
zstore/annelies        usedbyrefreservation  0                       -
zstore/annelies        logbias               latency                 default
zstore/annelies        dedup                 off                     local
zstore/annelies        mlslabel                                      -
zstore/annelies        sync                  standard                local
zstore/annelies        refcompressratio      1.00x                   -
zstore/annelies        written               329K                    -
zstore/movies          type                  filesystem              -
zstore/movies          creation              Thu Aug 22 19:25 2013   -
zstore/movies          used                  5.36G                   -
zstore/movies          available             13.6T                   -
zstore/movies          referenced            5.36G                   -
zstore/movies          compressratio         1.00x                   -
zstore/movies          mounted               yes                     -
zstore/movies          quota                 none                    local
zstore/movies          reservation           none                    local
zstore/movies          recordsize            128K                    default
zstore/movies          mountpoint            /zstore/movies          default
zstore/movies          sharenfs              off                     default
zstore/movies          checksum              fletcher4               inherited from zstore
zstore/movies          compression           lzjb                    local
zstore/movies          atime                 off                     local
zstore/movies          devices               on                      default
zstore/movies          exec                  on                      default
zstore/movies          setuid                on                      default
zstore/movies          readonly              off                     local
zstore/movies          jailed                off                     default
zstore/movies          snapdir               hidden                  local
zstore/movies          aclmode               discard                 default
zstore/movies          aclinherit            restricted              default
zstore/movies          canmount              on                      local
zstore/movies          xattr                 off                     temporary
zstore/movies          copies                1                       default
zstore/movies          version               5                       -
zstore/movies          utf8only              off                     -
zstore/movies          normalization         none                    -
zstore/movies          casesensitivity       sensitive               -
zstore/movies          vscan                 off                     default
zstore/movies          nbmand                off                     default
zstore/movies          sharesmb              off                     default
zstore/movies          refquota              none                    default
zstore/movies          refreservation        none                    default
zstore/movies          primarycache          all                     default
zstore/movies          secondarycache        all                     default
zstore/movies          usedbysnapshots       0                       -
zstore/movies          usedbydataset         5.36G                   -
zstore/movies          usedbychildren        0                       -
zstore/movies          usedbyrefreservation  0                       -
zstore/movies          logbias               latency                 default
zstore/movies          dedup                 off                     local
zstore/movies          mlslabel                                      -
zstore/movies          sync                  standard                local
zstore/movies          refcompressratio      1.00x                   -
zstore/movies          written               5.36G                   -
zstore/roy             type                  filesystem              -
zstore/roy             creation              Thu Aug 22 20:13 2013   -
zstore/roy             used                  1003M                   -
zstore/roy             available             13.6T                   -
zstore/roy             referenced            1003M                   -
zstore/roy             compressratio         1.00x                   -
zstore/roy             mounted               yes                     -
zstore/roy             quota                 none                    local
zstore/roy             reservation           none                    local
zstore/roy             recordsize            128K                    default
zstore/roy             mountpoint            /zstore/roy             default
zstore/roy             sharenfs              off                     default
zstore/roy             checksum              fletcher4               inherited from zstore
zstore/roy             compression           lzjb                    local
zstore/roy             atime                 off                     local
zstore/roy             devices               on                      default
zstore/roy             exec                  on                      default
zstore/roy             setuid                on                      default
zstore/roy             readonly              off                     local
zstore/roy             jailed                off                     default
zstore/roy             snapdir               visible                 local
zstore/roy             aclmode               discard                 default
zstore/roy             aclinherit            restricted              default
zstore/roy             canmount              on                      local
zstore/roy             xattr                 off                     temporary
zstore/roy             copies                1                       default
zstore/roy             version               5                       -
zstore/roy             utf8only              off                     -
zstore/roy             normalization         none                    -
zstore/roy             casesensitivity       sensitive               -
zstore/roy             vscan                 off                     default
zstore/roy             nbmand                off                     default
zstore/roy             sharesmb              off                     default
zstore/roy             refquota              none                    default
zstore/roy             refreservation        none                    default
zstore/roy             primarycache          all                     default
zstore/roy             secondarycache        all                     default
zstore/roy             usedbysnapshots       0                       -
zstore/roy             usedbydataset         1003M                   -
zstore/roy             usedbychildren        0                       -
zstore/roy             usedbyrefreservation  0                       -
zstore/roy             logbias               latency                 default
zstore/roy             dedup                 off                     local
zstore/roy             mlslabel                                      -
zstore/roy             sync                  standard                local
zstore/roy             refcompressratio      1.00x                   -
zstore/roy             written               0                       -
zstore/royandannelies  type                  filesystem              -
zstore/royandannelies  creation              Thu Aug 22 19:25 2013   -
zstore/royandannelies  used                  329K                    -
zstore/royandannelies  available             13.6T                   -
zstore/royandannelies  referenced            329K                    -
zstore/royandannelies  compressratio         1.00x                   -
zstore/royandannelies  mounted               yes                     -
zstore/royandannelies  quota                 none                    default
zstore/royandannelies  reservation           none                    default
zstore/royandannelies  recordsize            128K                    default
zstore/royandannelies  mountpoint            /zstore/royandannelies  default
zstore/royandannelies  sharenfs              off                     default
zstore/royandannelies  checksum              fletcher4               inherited from zstore
zstore/royandannelies  compression           lzjb                    local
zstore/royandannelies  atime                 off                     local
zstore/royandannelies  devices               on                      default
zstore/royandannelies  exec                  on                      default
zstore/royandannelies  setuid                on                      default
zstore/royandannelies  readonly              off                     default
zstore/royandannelies  jailed                off                     default
zstore/royandannelies  snapdir               hidden                  default
zstore/royandannelies  aclmode               discard                 default
zstore/royandannelies  aclinherit            restricted              default
zstore/royandannelies  canmount              on                      default
zstore/royandannelies  xattr                 off                     temporary
zstore/royandannelies  copies                1                       default
zstore/royandannelies  version               5                       -
zstore/royandannelies  utf8only              off                     -
zstore/royandannelies  normalization         none                    -
zstore/royandannelies  casesensitivity       sensitive               -
zstore/royandannelies  vscan                 off                     default
zstore/royandannelies  nbmand                off                     default
zstore/royandannelies  sharesmb              off                     default
zstore/royandannelies  refquota              none                    default
zstore/royandannelies  refreservation        none                    default
zstore/royandannelies  primarycache          all                     default
zstore/royandannelies  secondarycache        all                     default
zstore/royandannelies  usedbysnapshots       0                       -
zstore/royandannelies  usedbydataset         329K                    -
zstore/royandannelies  usedbychildren        0                       -
zstore/royandannelies  usedbyrefreservation  0                       -
zstore/royandannelies  logbias               latency                 default
zstore/royandannelies  dedup                 off                     local
zstore/royandannelies  mlslabel                                      -
zstore/royandannelies  sync                  standard                local
zstore/royandannelies  refcompressratio      1.00x                   -
zstore/royandannelies  written               329K                    -
zstore/series          type                  filesystem              -
zstore/series          creation              Thu Aug 22 19:25 2013   -
zstore/series          used                  329K                    -
zstore/series          available             13.6T                   -
zstore/series          referenced            329K                    -
zstore/series          compressratio         1.00x                   -
zstore/series          mounted               yes                     -
zstore/series          quota                 none                    default
zstore/series          reservation           none                    default
zstore/series          recordsize            128K                    default
zstore/series          mountpoint            /zstore/series          default
zstore/series          sharenfs              off                     default
zstore/series          checksum              fletcher4               inherited from zstore
zstore/series          compression           lzjb                    local
zstore/series          atime                 off                     local
zstore/series          devices               on                      default
zstore/series          exec                  on                      default
zstore/series          setuid                on                      default
zstore/series          readonly              off                     default
zstore/series          jailed                off                     default
zstore/series          snapdir               hidden                  default
zstore/series          aclmode               discard                 default
zstore/series          aclinherit            restricted              default
zstore/series          canmount              on                      default
zstore/series          xattr                 off                     temporary
zstore/series          copies                1                       default
zstore/series          version               5                       -
zstore/series          utf8only              off                     -
zstore/series          normalization         none                    -
zstore/series          casesensitivity       sensitive               -
zstore/series          vscan                 off                     default
zstore/series          nbmand                off                     default
zstore/series          sharesmb              off                     default
zstore/series          refquota              none                    default
zstore/series          refreservation        none                    default
zstore/series          primarycache          all                     default
zstore/series          secondarycache        all                     default
zstore/series          usedbysnapshots       0                       -
zstore/series          usedbydataset         329K                    -
zstore/series          usedbychildren        0                       -
zstore/series          usedbyrefreservation  0                       -
zstore/series          logbias               latency                 default
zstore/series          dedup                 off                     local
zstore/series          mlslabel                                      -
zstore/series          sync                  standard                local
zstore/series          refcompressratio      1.00x                   -
zstore/series          written               329K                    -

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#79

Post by fritz » 29 Sep 2013 17:36

Hi Neptunus,

After searching the internet, I probably found the reason of the issue.

The command "subfilesystem in `$BIN_ZFS list -H $depth_flag -o name $I_FILESYSTEM`; do", should itterate filesystems for which a snapshot should be created.
This works fine in the normal case.
Nevertheless, if the "listsnaps" property is "on" (default if "off"), the command mentionned above also itterates snapshots.
As snapshots cannot be created for snapshots, the script fails...

The "listsnaps" property is a pool level property.
You can get the value of the property calling: "zpool get all <pool_name> | grep listsnaps".

You can either change the property value to "false", or replace:

Code: Select all

for subfilesystem in `$BIN_ZFS list -H $depth_flag -o name $I_FILESYSTEM`; do
by

Code: Select all

for subfilesystem in `$BIN_ZFS list -t filesystem,volume -H $depth_flag -o name $I_FILESYSTEM`; do
Please tell me if it solved your problem.
I will them update the scripts in github to make them more robust.

Thanks

fritz
Last edited by fritz on 29 Sep 2013 18:42, edited 2 times in total.
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#80

Post by fritz » 29 Sep 2013 17:42

neptunus wrote:Is it possible to use manageSnapshots.sh and backupData.sh not on a ZFS Pools but on a ZFS Datasets?

I only need the Snapshots on userA and userB.

Code: Select all

NAME             USED    AVAIL  REFER  MOUNTPOINT
zssddata               61.9G   622M   304K   /zssddata
zstore                 6.39G   13.6T  420K   /zstore
zstore/userA           329K    13.6T  329K   /zstore/userA
zstore/userB           1003M   13.6T  1003M  /zstore/userB
zstore/series          329K    13.6T  329K   /zstore/series
zstore/movies          5.36G   13.6T  5.36G  /zstore/movies
Hi Neptunus,

Yes, it is possible for both manageSnapshots.sh and backupData.sh.
(see usage in the header if the files)

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#81

Post by fritz » 29 Sep 2013 17:47

joopa wrote:Hi Fritz,

Excellent work, thank you.

For now I do not use your scripts snapshots, but it will come, for sure.
I focused on the wakeup of the NAS and its extinction with manageAcpi.sh. I use scripts in beta 2.

Your scripts are very well structured, I think that after the implementation of the test ssh connection, additional functionality can be integrated without too much difficulty.
As dnar, I would implement :
- Test for sockets and therefore access. For example, samba shares, the UPnP
- Test for process (not my priority)

Where are you in your development ?
Have you ever considered these features ?
If so, should I wait or launch into these new developments ?
Hi joopa,

Thanks for your feedback.

In fact I do not have much time currently. Therefore I do not plan to develop additional functions.
Nevertheless, in case any bug is detected, I will try to solve it.
Feel free to extend the scripts and to send me pull request on github, I will try to incorporate those improvements in the may branch.

Thanks

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

neptunus
experienced User
experienced User
Posts: 86
Joined: 11 Jun 2013 08:50
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#82

Post by neptunus » 29 Sep 2013 20:43

fritz wrote: The "listsnaps" property is a pool level property.
You can get the value of the property calling: "zpool get all <pool_name> | grep listsnaps".

You can either change the property value to "false", or replace:

Code: Select all

for subfilesystem in `$BIN_ZFS list -H $depth_flag -o name $I_FILESYSTEM`; do
by

Code: Select all

for subfilesystem in `$BIN_ZFS list -t filesystem,volume -H $depth_flag -o name $I_FILESYSTEM`; do
z
I set the listsnapshots to "on" and to be honest I don't remember why! But I changed the value to "off" with: zpool set listsnapshots=off zstore

Now testing again, will let you know what the outcome is.

THANKS FOR THE SUPPORT!!

PS if you need a tester, let me know!

neptunus
experienced User
experienced User
Posts: 86
Joined: 11 Jun 2013 08:50
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#83

Post by neptunus » 30 Sep 2013 07:50

Fritz it is working now! But I have some other issues. Something with "Could not start script (Another instance is running)" probably because I have not configured everything properly.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#84

Post by fritz » 30 Sep 2013 09:54

Hi Neptunus,

You get this message if either:
- Another instance is running, or
- An instance has stopped abnormally previously (e.g. crashed or stopped by the user by Ctrl-C)

In the latter case, please delete manually the corresponding lock file (you can fing the path to the lock folder in config.sh: CFG_LOCKS_FOLDER)

I have improved the log message in GitHub (trunk) to make that clear (as your are not the 1st one having this problem...)

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#85

Post by fritz » 30 Sep 2013 10:18

fritz wrote:Hi Neptunus,

After searching the internet, I probably found the reason of the issue.

The command "subfilesystem in `$BIN_ZFS list -H $depth_flag -o name $I_FILESYSTEM`; do", should itterate filesystems for which a snapshot should be created.
This works fine in the normal case.
Nevertheless, if the "listsnaps" property is "on" (default if "off"), the command mentionned above also itterates snapshots.
As snapshots cannot be created for snapshots, the script fails...

The "listsnaps" property is a pool level property.
You can get the value of the property calling: "zpool get all <pool_name> | grep listsnaps".

You can either change the property value to "false", or replace:

Code: Select all

for subfilesystem in `$BIN_ZFS list -H $depth_flag -o name $I_FILESYSTEM`; do
by

Code: Select all

for subfilesystem in `$BIN_ZFS list -t filesystem,volume -H $depth_flag -o name $I_FILESYSTEM`; do
Please tell me if it solved your problem.
I will them update the scripts in github to make them more robust.

Thanks

fritz
The robustness issue has been solved in github (trunk)
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

neptunus
experienced User
experienced User
Posts: 86
Joined: 11 Jun 2013 08:50
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#86

Post by neptunus » 01 Oct 2013 08:48

fritz wrote:
The robustness issue has been solved in github (trunk)
I tested the trunk version. It is working OK! I made a little mistake in configuring the cron jobs.

trendco
Starter
Starter
Posts: 70
Joined: 20 Jan 2013 18:59
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#87

Post by trendco » 03 Oct 2013 12:03

Hi fritz,

which Version should I use with the current nas4free (9.1.0.1.847) - V1.0 or V2-beta?


Alex

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#88

Post by fritz » 03 Oct 2013 14:00

Hi trendco,

I Am still using an older version of nas4free.
Nevertheless I am quite sure that both versions should work well with 847, as 847 is still based on the same freebsd version (9.1).
I would recommend 1.0 for productive use (though 2.0b1 should already be quite robust.

Fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

philb
NewUser
NewUser
Posts: 4
Joined: 30 Sep 2013 18:18
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#89

Post by philb » 03 Oct 2013 18:58

Fritz,

I've just started using your scripts, excellent work. All good so far. Many thanks.

Phil

kenZ71
Advanced User
Advanced User
Posts: 403
Joined: 27 Jun 2012 20:18
Location: Northeast, USA
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#90

Post by kenZ71 » 07 Oct 2013 04:49

Scripts are awesome!!! Kudos!

Now I need to figure out why they won't run from Cron. They run if I click "Run Now" but not as scheduled.
11.2-RELEASE-p3 | ZFS Mirror - 2 x 8TB WD Red | 28GB ECC Ram
HP ML10v2 x64-embedded on Intel(R) Core(TM) i3-4150 CPU @ 3.50GHz

Extra memory so I can host a couple VMs
1) Unifi Controller on Ubuntu
2) Librenms on Ubuntu

Post Reply

Return to “Scripts and shell tips”