*New 12.1 series Release:
2019-11-08: XigmaNAS 12.1.0.4.7091 - released!

*New 11.3 series Release:
2019-10-19: XigmaNAS 11.3.0.4.7014 - released


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPools...

XigmaNAS Scripts and shell tips
Forum rules
Set-Up GuideFAQsForum Rules
fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPools...

#1

Post by fritz » 02 Jan 2013 14:48

Hi all,

I am a happy Freenas7/Nas4Free user since more than 2 years.
Time to give something back to the community!!!

During the last 2 years I developped various scripts to complement missing functions of Freenas7/Nas4free (The scripts can probably be used on other FreeBSD distributions like FreeNAS8, but it was not tested)

These scripts will probably be usefull for many users having a similar usage profile as myself:
- NAS access from several computers in a home network
- NAS should only consume power when it is required (electricity is expensive here) (makes use of acpiconf)
- Prevent data loss (through regular monitoring)
- Be able to go back to a previous version of the data if needed
- NAS containing 2 ZFS pools (one for data shared with CIFS and one containing the backup of the data)

All scripts that I provide to you rely on a shell script library of common functions:
- To log what the script is doing in a file (resp. extract data from the log)
- To report issues per mail
- To prevent a script from be executed twice concurrently and prevent a script to start if the NAS if about to sleep

The scripts rely on a common configuration file (config.sh) in which the email addresses, path to log files and tmp files... can be set


DESCRIPTION
===========

In my opinion, the scripts are well documented enough to be understood by anybody having shell script knowledge.
The script usage (e.g. arguments that can be passed) as well as a detailed description of the functions is provided in the header of each script. PLEASE READ THE HEADER OF THE SCRIPT BEFORE USING IT!
I tried to develop the scripts to be as versatile as possible (the scripts accept various optional arguments), so that they should help many NAS4Free users

scrubPools.sh:
--------------

- Scrub all the ZPOOLS and report the results in a log file.
- Send a mail if an error is detected
- Typical scheduling: once a week

checkPools.sh:
--------------

- Checks the status of the ZPOOLS and report the results in a log file.
- Send a mail if an error is detected
- Typical scheduling: every hour

checkTemps.sh: (Thanks to miGi from NAS4Free forum)
--------------

- Checks the temperature of each respective CPU / CPU core and of each respective drive and report the results in a log file.
- Send a mail if the CPU (resp. HDD) temperature threshold is reached
- Typical scheduling: every hour or more often

checkSpace.sh:
--------------

- Check if there is enough space in the respective ZFS file systems (The filesystems to monitor as well as the threshold is configurable)
- Report the results in a log file.
- Send a mail if the space threshold is reached
- Typical scheduling: every day for the data pool
- Typical scheduling: every week for the backup pool

manageSnapshots.sh:
-------------------

- Create hourly/daily/weekly/monthly snapshots of a ZFS filesystem (and child filesystems) and keep a configurable number of them (i.e. delete the superfluous snapshots)
- Report the results in a log file.
- Send a mail if an error occurs
- This script was specially designed to work fine even if the NAS is NOT always ON. E.g. if the NAS is OFF when the next monthly snapshot should be created, the script will create it as soon as the NAS is back on.
- This script does only need to be scheduled every hour and it will take care of any type of snapshot (hourly/daily/...)

Note 1: In Services|CIFS/SMB|Share|Edit, set the parameter "Shadow Copy format" to "%Y%m%d_%H%M_autosnap_type%S" in order to make the snapshots visible in the windows shadow copy client

backupData.sh:
--------------

- Backup incrementally (using zfs send/receive) the content of a set of ZFS filesystems into another pool located either in the same NAS box, or in a remote machine
- Report the results in a log file.
- Send a mail if an error occurs
- This script was designed to able to cope with the deletion of superfluous snapshots in the data pool (This may be issued by the manageSnapshots.sh script)
- Typical scheduling: once a week

manageAcpi.sh:
--------------

- Controls shutdown (S5) or sleep (S3) of the NAS based on various parameters
- If no computer that may access the NAS is online during a certain time (based on the IP address)
- If the curfew is reached!
- (if requested) prevents a shutdown / sleep during a configurable timeslot
- (if requested) Prevents a shutdown / sleep if another machine is accessing to the NAS using SSH
- Prevents a shutdown / sleep to be issued when one of the above administrative scripts is running
- Reports the results in a log file.
- Script to be started at NAS startup. The script runs as an endless loop

Note 1: Depending on your motherboard and on your BIOS settings, the script may or may not work on your NAS.

Note 2: This script requires static IP addresses (of course static DHCP is OK too) on your NAS and other any other computer.

Note 3: I use Wake on LAN to wake the NAS. The WOL paket is sent by my DD-WRT router automatically when one of my computers is online (see script here: https://github.com/fritz-hh/autowake_NAS)

acpiStats.sh:
-------------

- Compute the percentage of time the NAS spent in each respective ACPI state (S0/S3/S5)
- Compute the average power usage in watt (based on typical power usage in S0/S3/S5)
- Report the results in a log file.
- Send a mail if an error occurs
- Typical scheduling: once a week

Note 1: This script requires manageAcpi.sh to be started when your NAS starts (indeed manageAcpi.sh generates a file (acpi.log) that is required by acpiStats.sh)

reportFromLogs.sh:
------------------

- Scripts aimed at generating a extract of all log files (the part that has been populated during the last week)
- Additionally it generates statistics on the number of errors and warnings that occured in each scripts
- This script can be sheduled in the "Status|Email Report" section of nas4free


DOWNLOAD:
=========

https://github.com/fritz-hh/scripts_NAS4Free/releases

INSTALLATION AND CONFIGURATION:
===============================

https://github.com/fritz-hh/scripts_NAS4Free/wiki

SUPPORT / QUESTIONS:
====================

viewtopic.php?f=70&t=2197

DISCLAIMER:
===========

Even though the scripts were developped with particular focus on robustness, I do not provide any warranty with the script !
(I am nor a freeBSD expert nor a SW developper, but a simple user)

Any contribution (problem reports, fixes, improvement of the wiki, comments, new functions) is welcome!

Feel free to use the scripts on your own NAS!

Kind Regards,

fritz
Last edited by fritz on 27 Jan 2015 17:37, edited 22 times in total.
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#2

Post by fritz » 06 Jan 2013 16:01

Hi all,

I have just updated my scripts in the initial post (from "scripts_20130102-3_for_NAS4Free_forum.tar" to "scripts_20130106-1_for_NAS4Free_forum.tar")

Main changes:

--> naming rules of the snapshots modifed in order to be compatible with the NAS4Free CIFS shadow copy function (e.g. provide all hourly, daily, weekly and monthly snapshots to the windows shadow copy client):
  • In Services|CIFS/SMB|Share|Edit, set the parameter "Shadow Copy format" to "%Y%m%d_%H%M_autosnap_type%S" in order to make the snapshots visible in the windows shadow copy client
  • If you already used the previous version of the package ("scripts_20130102-3_for_NAS4Free_forum.tar"), please run "updateSnapshotsNamingRules.sh" to rename the already generated snapshots
--> code quality improvements

Regards
fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

User avatar
raulfg3
Site Admin
Site Admin
Posts: 4949
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#3

Post by raulfg3 » 06 Jan 2013 22:33

thaks a lot, I don't use still, but perhaps in near future.

Is a good work and you have my gratitude.
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#4

Post by Onichan » 07 Jan 2013 23:24

I haven't looked at either source code, but does your SnapShot script work basically the same as https://github.com/graudeejs/zfSnap/wiki/zfSnap

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#5

Post by fritz » 08 Jan 2013 12:25

Hi Onichan,

Main conceptual differences:

After a quick analysis, I saw that there are major conceptual differences between my script (1) and the script you mentionned (2).
Thoses differences will mainly lead to different behaviours if the NAS is not always ON.
To make it simple (1) was developped for NAS that are always ON. (2) takes into account that the NAS is not always ON
Let me explain that a little more:

(2):

Snapshot creation:
Each time the script is called, a snapshot is created. The snapshot receives a time to live (TTL) (TTL is passed as argument to the script)
So to create hourly, daily, weekly and monthly snapshot, you will basically create 4 cron entries with different TTL.
Therefore is the NAS is not switched ON (for example) when a monthly snapshot should be created, there will be at least 1 more month delay until the next monthly snapshot is created. If this case occur several times in row, you will be missing several monthly snapshot. This is probably not what the user wants.

Snapshot deletion:
When TTL has elapsed

(1)

Snapshot creation:
The script shall be called hourly (only 1 cron job is needed). The script then takes care of all types of snapshots (hourly, daily, weekly, monthly).
Thereby it analyses the duration since the latest creation of hourly, daily, weekly and monthly snapshot.
Based on that information it decides which type of snapshot should be created
As a consequence, if your NAS was OFF when a monthly/weekly and daily snapshot had to be created, the creation of the missing snapshot will created latest 1 hour after the NAS is back ON.

Snapshot deletion:
The script keeps a configurable number of hourly/daily/weekly/monthly snapshots.
Here also there will be only little behavioral difference between both scripts if the NAS is always ON.
If the NAS is not always ON, snapshot deletion will be delayed.
The idea behind that is the following: Imagine the case of a home NAS user that has configured the NAS to keep the last 48 hourly snapshots. This user worked on a file during several hours before switching OFF the NAS and going on a holliday trip for 10 days. After coming back home he wants to finish his work on the file. If he wants to come back to the state of the file before the holliday it is stil possible. This would not possible with a TTL of 48 hours (To make it simple, one can say that (2) enables to define a TTL where the OFF time is not taken into account)

Other main differences:

I will only mention few other notable differences here. Of course, there are many more differences if you analyse both scripts in detail, but most of them should not be important for most of the users.
  • (2) Allows to define the periodicity of the snapshot creation in a more flexible ways than (1). (2) supports any periodicity, whereas (1) only hourly/daily/weekly/monthly
  • (1) Allows to show all type of snapshots in the NAS4Free CIFS shadow copy server, whereas (2) does not due to the way it names the snapshots
  • (1) Logs the actions performed in a file and sends a mail to the administrator in case an error occurs

Regards,
fritz
Last edited by fritz on 15 Jan 2013 11:37, edited 2 times in total.
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#6

Post by Onichan » 09 Jan 2013 02:10

Thank you fritz, that is what I was looking for. I did notice in your original post you mentioned yours supported the snapshot power state awareness which I thought was quite nice, but I wasn't sure if that was the only difference.

twoj
Starter
Starter
Posts: 26
Joined: 29 Dec 2012 00:06
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#7

Post by twoj » 12 Jan 2013 05:27

Thanks very much for the scripts - they are a great resource.

I'm having an issue with acpiStats.sh, it is always returning with an error;

20130111_232204 INFO Starting computation of ACPI statistics
20130111_232204 ERROR Could not start script (Another instance is running)

I copied your command as you had it;

/mnt/Data/scripts/fritz/acpiStats.sh -p 50000,2400,1000

Even after a restart it is giving this - any ideas?

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#8

Post by fritz » 12 Jan 2013 10:48

Hi twoj,
  • First of all, please note that it only makes sense to run acpiStats.sh if you run also manageAcpi.sh (see in my 1st post, what both scripts are doing). Indeed acpiStats.sh analyzes a log file (acpi.log) generated by manageAcpi.sh
    manageAcpi.sh should be called when your NAS starts (as a postInit command script) and runs in an endless loop. (You will find how I call that script in my first post in the extract of my nas4free config)
  • Please note also that acpiStats.sh and manageAcpi.sh rely on "config.sh", "commonLogFcts.sh", "commonMailFcts.sh" and "commonLockFcts.sh" that should be located in the same directory.
  • Please update the config variables in config.sh to your setup (location of the log files, lock files, email addresses...)
  • You need also to have a mechanism to wake up your NAS (e.g. the script I provided if you have a DD-WRT router)
In order to prevent acpiStats from being executed twice concurrently, the function "script_start" (located in file "commonLockFcts.sh") is called when acpiStats starts.
This function creates a lock file in the folder "CFG_RUNNING_SCRIPTS_FOLDER" (config param defined in "config.sh)" for the script that was just started.
If the lock file already exists, acpiStats returns with an error ("Could not start script (Another instance is running)")

If you stopped acpiStats (e.g. using Ctrl-C) before it was finished (or if it crashed for some reason), the lock file remains and you get the error you mentionned (I may catch the respective freebsd signal in a future version and delete the lock file automatically if the script stops anormally)
Therefore, if you get the above mentionned error mesage, you need to remove the lock manually before you restart acpiStats.sh

I am surprized that the lock was still available after restarting your nas.
manageAcpi.sh deletes automatically all remain locks. Did you run manageAcpi.sh after restarting? (you should add manageAcpi.sh as a postInit command script)

Hope that helped.

fritz
Last edited by fritz on 13 Jan 2013 19:35, edited 2 times in total.
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

twoj
Starter
Starter
Posts: 26
Joined: 29 Dec 2012 00:06
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#9

Post by twoj » 13 Jan 2013 19:03

Hi Fritz

ok - that makes more sense that it needs the log files from manageAcpi.sh, i was under the impression that it could generate them independently.
At this point i haven't installed manageAcpi.sh at all - thats why it suprises me that it is giving the error about another script running.
I guess also this is a moot point because I'm using a HP microserver and I'm waiting for the next release which should bring WoL support for the NIC. So at this time i have no way of remotely starting the machine. At the moment it is really being used as a once-a-month backup so i can just turn it on when its necessary.
When the WoL is working i will return to your script - i'm sure it just because i didn't have the manageAcpi.sh running.

Thanks for all you work!
cheers

PS - thanks for the PM

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#10

Post by fritz » 13 Jan 2013 19:47

Hi twoj,

I was not clear enough about this dependency in first post (I have updated it now)

Now, it is clear to me what happened:
  • You never executed manageAcpi.sh, so that the file acpi.log did not exist
  • When you executed acpiStats.sh the first time it crashed because the file acpi.log did not exist (I did not make the required check in the script...)
  • Because acpiStats.sh crashed, the lock file "acpiStats.sh.lock" has not been deleted (the script currently does not delete the lock when the scripts stops abnormally)
  • The next times you executed acpiStats you got an error message because the lock file was still existing ("Could not start script (Another instance is running)")
I will address these robustness issues in the next version of my scripts.

Cheers
fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

johnny.dr
NewUser
NewUser
Posts: 5
Joined: 15 Jan 2013 04:55
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#11

Post by johnny.dr » 15 Jan 2013 05:05

Hey fritz. Thanks for posting these great scripts and explanations. I'm super pumped about them. I'm new to N4F and new to Linux, and so I'm slowly learning them (I like to understand how things works). I was hoping you could help me w/ a question.

I'm using the embedded install on a USB disk. I'm learning I can't mount my OS disk (not completely understanding why @ this point). So as you stated in your original post, I'll have to copy the scripts to one of my ZFS pools. My question is this: is it really safe to keep the scripts on the disks that we intend to check for errors? If the disk fails, so do our scripts, right? If this is correct, you understand why I'd rather run the scripts from the USB flash disk and save the logs to a ZFS pool.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#12

Post by fritz » 15 Jan 2013 11:04

Hi johny.dr,
johnny.dr wrote:I'm using the embedded install on a USB disk.
I am using embedded too.
is it really safe to keep the scripts on the disks that we intend to check for errors? If the disk fails, so do our scripts, right?
I keep the scripts on one of the zpools that I intend to check for errors.
This is probably not ideal, but the zpool on which my scripts are located are redundant. Therefore, in case of a (single) disk failure, the script will continue to work normally. As a consequence the "checkPools.sh" will detect the error and send me error mail. As I run this script every hour it is highly improbable not to get an error mail before the 2nd disks fails (2 failures should occur within one hour)
(Note: It is nevertheless important to have a backup of your data in another pool)

To take into account the case where 2 failures occur within 1 hour, one could go one step further and modify the cron command that calls "checkPools.sh", so that it sends a mail if the script could not be started. With this change you would receive a error mail even if your zpools are not available anymore. The following cron command should work:

Code: Select all

/mnt/tank/nas_scripts/scripts_released/checkPools.sh ; [ $? -ne 0 ] && /usr/bin/printf "From: nas@isp.com\nTo: admin.of.nas@isp.com\nSubject: scrubPools.sh could not be started\n\nYour zpool may be unavailable" | /usr/local/bin/msmtp --file=/var/etc/msmtp.conf -t
Note: you need to configure mail in the NAS4Free GUI (System|Advanced|Email) before running the above mentionned command

Regards,

fritz
Last edited by fritz on 15 Jan 2013 15:39, edited 1 time in total.
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

RedAntz
experienced User
experienced User
Posts: 128
Joined: 11 Jul 2012 07:46
Location: Sydney, Australia
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#13

Post by RedAntz » 15 Jan 2013 12:50

This is very helpful. Thanks for sharing fritz !

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#14

Post by fritz » 26 Jan 2013 15:30

Hi all,

I have just updated my scripts in the initial post (see "scripts_20130125-1_for_NAS4Free_forum.zip")

Main changes:

--> Code quality and robustness improvements

--> In checkPools.sh: size written in MiB instead of byte in the log file

(Note: details of the changes can be found in the release notes of each respective script (at the end of the file))

Regards
fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

User avatar
raulfg3
Site Admin
Site Admin
Posts: 4949
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#15

Post by raulfg3 » 26 Jan 2013 18:39

thanks.
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

miGi
NewUser
NewUser
Posts: 3
Joined: 06 Feb 2013 12:14
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#16

Post by miGi » 06 Feb 2013 12:37

At first, thanks fritz for your wonderful scripts!

I want to help the community as well, so I decided to send the script I made for this script family.

checkTemp.sh:
Script checks CPU and HDD temperatures, and sends e-mail if desired limits are exceeded. Warning temperature thresholds are given as param1 (CPU) and param2 (HDD). Script works whit fritz's other scripts and needs them to run correctly.

-miGi
You do not have the required permissions to view the files attached to this post.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#17

Post by fritz » 06 Feb 2013 14:22

Thank you miGi.

Nice script. I will try it on my NAS in the next days.
Would you agree if I add your script in the package I provided in the 1st post?
I am also considering to put the whole package in github in order to ease collaboration (I may do it in the nex weeks, if I have enough time)

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

miGi
NewUser
NewUser
Posts: 3
Joined: 06 Feb 2013 12:14
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#18

Post by miGi » 07 Feb 2013 14:12

Of course, feel free to use the script any way you want. You can also edit it if necessary. It would be an honor if you put it in first post.

I made a few small updates to the script. Here is a new version that includes a changelog.

-miGi

Edit:
There is one know bug. Portable memory stick will cause "Bad number" error in console, because the temperature can not be read. It does not, however, affect the way scripts runs.
You do not have the required permissions to view the files attached to this post.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#19

Post by fritz » 08 Feb 2013 21:11

Hi miGi,

I have updated my first post and added your script.

I made some additional changes in the script:
- correction of "Bad number" error in case of portable memory stick
- verification of the input parameters added
- check temperature of all CPUs / CPU cores
- log device model correctly even if it contains spaces (which is the case for me)
- log device serial number correctly even if it contains spaces
- temperature of disks moved to the end of the line (readability)

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

User avatar
raulfg3
Site Admin
Site Admin
Posts: 4949
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#20

Post by raulfg3 » 08 Feb 2013 21:32

To avoid lost of this usefull piece of gold, I suggest to copy scripts to github, so updates in code are easy and more people can collaborate.
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#21

Post by fritz » 09 Feb 2013 15:55

Hi everybody,

The scripts are now on github: https://github.com/fritz-hh/scripts_NAS4Free
Do not hesitate to contribute with improvements, corrections, bug reports, ideas, new scripts...

I will contnue to update my 1st post when major updates of the scripts are available.

fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#22

Post by Onichan » 21 Feb 2013 01:17

I am having trouble getting these to work. Manually running it from ssh works, but when I create a job it doesn't work. When I try the run now I get "root: Failed to execute cron job '/apps/scripts/scripts_NAS4Free-1.0-rc2/scrubPools.sh'." and my cron job is currently "0 1 1 * * root /apps/scripts/scripts_NAS4Free-1.0-rc2/scrubPools.sh".

I am pretty sure it has to do with working directories. The scripts should use /apps/scripts/scripts_NAS4Free-1.0-rc2/ as the working directory, but I don't think they are as I found /var/log/tmp/running_scripts/manageSnapshots.sh..lock

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#23

Post by fritz » 21 Feb 2013 10:50

Hi Onichan,

Thank you for your message.
Yes, you are right with your assumption. This is an issue relating to working directories. This issue has an impact on the "tmp" and "log" folders (every thing else should be independent of the working directory)
I will try to implement a fix until the stable release.

There is a simple workaround:
In the config file: use an absolute path for "CFG_TMP_FOLDER" and for "CFG_LOG_FOLDER" instead of relative paths

(This is what I do, and cron jobs work fine)

fritz

(Do not hesitate to write problem reports in github if required)
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#24

Post by Onichan » 21 Feb 2013 21:24

I also have a question about the snapshots. I really don't want to use the default 24 hourly 15 daily and just want to keep the last 90 of daily's with no hourly or monthly's, for now. If I follow the guide and run it hourly as it is wont it create all those hourly snapshots and monthly ones? Should I modify the default values to what I want and then run it hourly? Or should I just leave the script as is and only run it once a day? If I only run it once a day then it seems to defeat the whole catching up snapshots if its turned off feature.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#25

Post by fritz » 21 Feb 2013 21:42

Hi onichan,
leave the script as is AND call it every hour (or it least more often then every day to ensure that it is allways called at least once a day even if your nas is not ON 24/7) with the following options:

Code: Select all

manageSnapshots.sh -h 0 -d 90 -w 0 -m 0 filesystem
The script should do what you want.
Fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#26

Post by Onichan » 21 Feb 2013 21:49

I actually tried that last night, but it gave me an error. Maybe I mistyped something. I'll try it again tonight, thanks.

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#27

Post by fritz » 21 Feb 2013 21:56

You are right. Thank you for bringing this bug/limitation to my notice.
I can see in the code that there is a mistake in the function that checks the options passed to the script (it does not accept "0"). I will correct it until the next release (v1.0-rc3 ). Should be available in a few days.
Fritz
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Onichan
Advanced User
Advanced User
Posts: 238
Joined: 04 Jul 2012 21:41
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#28

Post by Onichan » 21 Feb 2013 22:11

Sounds good, thanks.

macsimus
NewUser
NewUser
Posts: 2
Joined: 22 Feb 2013 13:48
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#29

Post by macsimus » 22 Feb 2013 14:38

How to install the scripts:

Copy the *.sh in a folder of your NAS(all scripts must be copied into the same folder)
Create a tmp folder
Create a log folder
Update the config.sh file according to your needs (paths, email addresses...)

Note 1: In case you use the embedded version, the folders mentionned above should be located on one of your data disks (you may create a dedicated ZFS filesystem for the scripts in your pool)
Note 2: There no dependencies to utilities that are not already included in NAS4Free (9.1.0.1 - Sandstorm (revision 531))

How I schedule the scripts:

Code: Select all

	<cron>
		<job>
			<enable/>
			<uuid>d230151f-0cd7-4718-b6c8-e762e3a149e3</uuid>
			<desc>Create snapshots in tank (superfluous snapshots not deleted to avoid interference with backup script)</desc>
			<minute>0</minute>
			<hour/>
			<day/>
			<month/>
			<weekday/>
			<all_mins>0</all_mins>
			<all_hours>1</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>1</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/manageSnapshots.sh -k tank</command>
		</job>
		<job>
			<enable/>
			<uuid>a41df59f-50bc-4a54-a69c-19c0c80daf5c</uuid>
			<desc>Scrub the ZFS pools to find inconsistencies</desc>
			<minute>5</minute>
			<hour>12</hour>
			<day/>
			<month/>
			<weekday>5</weekday>
			<all_mins>0</all_mins>
			<all_hours>0</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>0</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/scrubPools.sh</command>
		</job>
		<job>
			<enable/>
			<uuid>37f50d10-ffa5-4b24-b097-505a7953cc46</uuid>
			<desc>Check the health of the pools</desc>
			<minute>0</minute>
			<hour>
			<day/>
			<month/>
			<weekday/>
			<all_mins>0</all_mins>
			<all_hours>1</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>1</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/checkPools.sh</command>
		</job>
		<job>
			<enable/>
			<uuid>3c287d26-635b-4a21-8360-135bd90806ca</uuid>
			<desc>Check remaining disk space in pool tank</desc>
			<minute>0</minute>
			<hour>12</hour>
			<day/>
			<month/>
			<weekday/>
			<all_mins>0</all_mins>
			<all_hours>0</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>1</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/checkSpace.sh tank 80</command>
		</job>
		<job>
			<enable/>
			<uuid>f3ae2fb7-40c0-4a80-b484-3043d15e65ba</uuid>
			<desc>Backup everything except tank/pcs_images</desc>
			<minute>5</minute>
			<hour>12</hour>
			<day/>
			<month/>
			<weekday>2</weekday>
			<all_mins>0</all_mins>
			<all_hours>0</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>0</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/backupData.sh tank/data_common,tank/data_user1,tank/data_user2,tank/nas_scripts tank_backup 10</command>
		</job>
		<job>
			<enable/>
			<uuid>a8a01bfa-6680-491a-aa59-4ae82c6a48a4</uuid>
			<desc>Check remaining disk space in pool tank_backup</desc>
			<minute>0</minute>
			<hour>12</hour>
			<day/>
			<month/>
			<weekday>4</weekday>
			<all_mins>0</all_mins>
			<all_hours>0</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>0</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/checkSpace.sh tank_backup 80</command>
		</job>
		<job>
			<enable/>
			<uuid>7afbb7f2-cd9a-4454-971b-0a82ea9d91a3</uuid>
			<desc>Delete superfluous snapshot in tank_backup</desc>
			<minute>0</minute>
			<hour>12</hour>
			<day/>
			<month/>
			<weekday>3</weekday>
			<all_mins>0</all_mins>
			<all_hours>0</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>0</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/manageSnapshots.sh -n -w 26 -m -1 tank_backup</command>
		</job>
		<job>
			<enable/>
			<uuid>0964d819-ee08-46f4-92da-5a9edc965fbd</uuid>
			<desc>Delete superfluous snapshots in tank</desc>
			<minute>5</minute>
			<hour>12</hour>
			<day/>
			<month/>
			<weekday>1</weekday>
			<all_mins>0</all_mins>
			<all_hours>0</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>0</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/manageSnapshots.sh -n tank </command>
		</job>
		<job>
			<enable/>
			<uuid>65a41bb4-ef91-4d36-b765-23b008f940bc</uuid>
			<desc>Computes the ACPI statistics for the NAS</desc>
			<minute>0</minute>
			<hour>12</hour>
			<day/>
			<month/>
			<weekday>5</weekday>
			<all_mins>0</all_mins>
			<all_hours>0</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>0</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/acpiStats.sh -p 50000,2400,1000</command>
		</job>
		<job>
			<enable/>
			<uuid>07a763df-a686-49cc-9c39-ce2cbfcdb8b5</uuid>
			<desc>Monitor temperature of CPUs and HDDs</desc>
			<minute>0</minute>
			<hour/>
			<day/>
			<month/>
			<weekday/>
			<all_mins>0</all_mins>
			<all_hours>1</all_hours>
			<all_days>1</all_days>
			<all_months>1</all_months>
			<all_weekdays>1</all_weekdays>
			<who>root</who>
			<command>/mnt/tank/nas_scripts/scripts_released/checkTemp.sh 55 40</command>
		</job>
	</cron>
	<rc>
		<preinit/>
		<postinit>
			<cmd>/mnt/tank/nas_scripts/scripts_released/manageAcpi.sh -a 11:50,12:10 -c 22:00,7:00,5 -n 7:00,22:00,3,600,192.168.1.102+192.168.1.103+192.168.1.104</cmd>
		</postinit>
		<shutdown/>
	</rc>
Thanks fritz!!
I have a great respect for folks like yourself.
To whom it may concern; these features, especially auto suspend and wake on lan in true sense, would be a great addition in the web gui for average people like myself.
I'd like to use your scripts but I don't wanna break anything by doing the wrong thing.
Just wondering if anyone can simplify the above directions.
I believe I have the embedded install.
1- tmp and log folders where should they be created exactly?
2- not sure how to update config.sh
3. cron job script - where would that be located or needs to be created?
cheers

fritz
experienced User
experienced User
Posts: 87
Joined: 12 Dec 2012 16:40
Contact:
Status: Offline

Re: Helpful scripts: Backup,Snapshot,Standby,Scrub,CheckPool

#30

Post by fritz » 22 Feb 2013 14:48

Hi all,

A new version (v1.0-rc3) solving the issues mentionned by Onichan is available on GitHub (https://github.com/fritz-hh/scripts_NAS4Free/tags).
Unless any new major issue is discovered, this will be the last release candidate until v1.0 stable.

fritz
Last edited by fritz on 23 Feb 2013 15:13, edited 1 time in total.
O/S: NAS4Free 11.1.0.4 - Atomics (revision 5017) (Embedded 64bit), installed on 8GB USB flash drive
https://github.com/fritz-hh

Post Reply

Return to “Scripts and shell tips”