Sudden ZFS stability problems

[JamesP]

I've been using NSF4FREE for 3 months without any problems. My disk space usage has now reached 50% capacity and suddenly, ZFS performance (both read and write) is extremely bursty from 100MB/s down to 1MB/sec (mostly around 1MB/s). Any idea what might be causing this? Before this, I was able to sustain 80-100MB/s over gigE CIFS and much more running dd locally...

System configuration:
X9SCM-F motherboard
E3-1230v2 CPU
M1015 sata controller passed through to nas4free VM
ESXi v 5.1
6x 3TB WD red drives in RAID-Z2 config
NAS4FREE 9.1.0.1.636 installed as 'embedded'

Stumped...
James

Code: Select all

nas4free:/var/log# zpool status
  pool: ZFS-pool
 state: ONLINE
  scan: scrub repaired 0 in 19h51m with 0 errors on Wed Mar  6 03:34:18 2013
config:

        NAME        STATE     READ WRITE CKSUM
        ZFS-pool    ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0

errors: No known data errors

Code: Select all

nas4free:/var/log# zpool list
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
ZFS-pool  16.2T  8.15T  8.10T    50%  1.00x  ONLINE  -

[raulfg3]

check your Disk, normally is the answer. revise S.M.A.R.T.

[JamesP]

check your Disk, normally is the answer. revise S.M.A.R.T.

Thanks! There were no S.M.A.R.T. errors reported but then I decided to run a self-test on the drives and sure enough, one was failing. RMA time.

JP

[curacaoflip]

Hi,

got the same problem here, access time on my ZFS pool suddenly dropped. Restarted N4F and it took a loooooooong time to get the server back on line.
What do you use for a self test? Detach and hot swap on my win8 with a partition manager or is there a faster and more easy way.

Started scrub and it found a troubled device which it has started repairing (could take all night to finish). Within the first 30 gig already 2 Mb of repairs so I'm afraid the drive is a goner (WD Black 2Tb).

Downloaded the DOS edition of WD Data lifeguard diagnostic so I can start it on my server. If scrubs ends in time I'll run that, otherwise I'll have to abort scrub test and replace the drive anyway.

Any suggestions?

Thanks, Flip

[JamesP]

What do you use for a self test?

I just logged into the Nas4Free server via ssh and ran smartctl to launch a self-test on each of my drives.

BR,
James

[Lee Sharp]

There is also a way to launch a self test in the GUI.

[curacaoflip]

Browsed the GUI but can't find it, any ideas where to look?

Sorry, found it. Under SMART you can add a scheduled self test. I use the command line instead with smartctl which works fine.
Is already running 5 mins for the short test so I think I found the bad drive.

Regards, Flip

[Lee Sharp]

The irony of "I found the answer under SMART" is still causing me to chuckle... Glad you found it!

[curacaoflip]

Right you are, 'SMART' move huh?
Fact is that I'm a bit cautious for the 'Try or Die' since this is the pool in my NAS with a huge amount of videos and personal data on it.

Now I identified the damaged drive I bought a new one and am ready to replace it.
I found some info on it but still have some questions.
I'll start a new thread instead of taking over this one.

Cheers, Flip