Page 1 of 1
Sudden ZFS stability problems
Posted: 19 Mar 2013 04:25
by JamesP
I've been using NSF4FREE for 3 months without any problems. My disk space usage has now reached 50% capacity and suddenly, ZFS performance (both read and write) is extremely bursty from 100MB/s down to 1MB/sec (mostly around 1MB/s). Any idea what might be causing this? Before this, I was able to sustain 80-100MB/s over gigE CIFS and much more running dd locally...
System configuration:
X9SCM-F motherboard
E3-1230v2 CPU
M1015 sata controller passed through to nas4free VM
ESXi v 5.1
6x 3TB WD red drives in RAID-Z2 config
NAS4FREE 9.1.0.1.636 installed as 'embedded'
Stumped...
James
Code: Select all
nas4free:/var/log# zpool status
pool: ZFS-pool
state: ONLINE
scan: scrub repaired 0 in 19h51m with 0 errors on Wed Mar 6 03:34:18 2013
config:
NAME STATE READ WRITE CKSUM
ZFS-pool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
da1 ONLINE 0 0 0
da2 ONLINE 0 0 0
da3 ONLINE 0 0 0
da4 ONLINE 0 0 0
da5 ONLINE 0 0 0
da6 ONLINE 0 0 0
errors: No known data errors
Code: Select all
nas4free:/var/log# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
ZFS-pool 16.2T 8.15T 8.10T 50% 1.00x ONLINE -
Re: Sudden ZFS stability problems
Posted: 20 Mar 2013 08:42
by raulfg3
check your Disk, normally is the answer. revise S.M.A.R.T.
Re: Sudden ZFS stability problems
Posted: 20 Mar 2013 18:51
by JamesP
raulfg3 wrote:check your Disk, normally is the answer. revise S.M.A.R.T.
Thanks! There were no S.M.A.R.T. errors reported but then I decided to run a self-test on the drives and sure enough, one was failing. RMA time.
JP
Re: Sudden ZFS stability problems
Posted: 24 May 2013 21:34
by curacaoflip
Hi,
got the same problem here, access time on my ZFS pool suddenly dropped. Restarted N4F and it took a loooooooong time to get the server back on line.
What do you use for a self test? Detach and hot swap on my win8 with a partition manager or is there a faster and more easy way.
Started scrub and it found a troubled device which it has started repairing (could take all night to finish). Within the first 30 gig already 2 Mb of repairs so I'm afraid the drive is a goner (WD Black 2Tb).
Downloaded the DOS edition of WD Data lifeguard diagnostic so I can start it on my server. If scrubs ends in time I'll run that, otherwise I'll have to abort scrub test and replace the drive anyway.
Any suggestions?
Thanks, Flip
Re: Sudden ZFS stability problems
Posted: 24 May 2013 22:28
by JamesP
curacaoflip wrote:
What do you use for a self test?
I just logged into the Nas4Free server via ssh and ran smartctl to launch a self-test on each of my drives.
BR,
James
Re: Sudden ZFS stability problems
Posted: 24 May 2013 22:56
by Lee Sharp
There is also a way to launch a self test in the GUI.
Re: Sudden ZFS stability problems
Posted: 24 May 2013 23:14
by curacaoflip
Browsed the GUI but can't find it, any ideas where to look?
Sorry, found it. Under SMART you can add a scheduled self test. I use the command line instead with smartctl which works fine.
Is already running 5 mins for the short test so I think I found the bad drive.
Regards, Flip
Re: Sudden ZFS stability problems
Posted: 25 May 2013 20:00
by Lee Sharp
The irony of "I found the answer under SMART" is still causing me to chuckle...
Glad you found it!
Re: Sudden ZFS stability problems
Posted: 25 May 2013 20:52
by curacaoflip
Right you are, 'SMART' move huh?
Fact is that I'm a bit cautious for the 'Try or Die' since this is the pool in my NAS with a huge amount of videos and personal data on it.
Now I identified the damaged drive I bought a new one and am ready to replace it.
I found some info on it but still have some questions.
I'll start a new thread instead of taking over this one.
Cheers, Flip