Significant Performance Issues on ZFS Volume

[randomparity]

Testing NAS4Free 9.1.0.1 - Sandstorm (revision 573) and setup a ZFS pool/dataset on a Dell R210 system with an Areca ARC-1880 RAID controller (configured to pass-through the drives). After copying a sizeable amount of data to the CIFS shared ZFS dataset I've started actually using the NAS as a video file server. When running commands such as "ls" for the first time in a CIFS mounted directory the delay until I receive a directory listing is painfully slow, both from a Windows and Linux client. Actually playing a video file causes noticeable (5-10 second) pauses. Looking at the process information I see the following:

Code: Select all

last pid: 27551;  load averages:  0.29,  0.17,  0.07  up 5+18:30:17    10:05:11
24 processes:  1 running, 23 sleeping

Mem: 32M Active, 37M Inact, 3590M Wired, 33M Buf, 4008M Free
Swap:


  PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
45749 root            1  20    0 91964K 25896K zio->i  0  89:39  0.00% smbd
 2334 root            1  23    0 39304K  5396K kqread  3   2:12  0.00% lighttpd
 1965 root            1  20    0 12052K  1600K select  3   1:31  0.00% powerd
85446 root            1  20    0 65372K  7604K select  1   0:23  0.00% nmbd
85450 root            1  20    0 72028K 12288K select  5   0:01  0.00% smbd
 2147 root            1  20    0 25904K  3680K nanslp  2   0:01  0.00% smartd
47016 root            1  20    0  9912K  1864K select  6   0:01  0.00% mDNSResponderPosix
 1755 root            1  20    0 12080K  1708K select  5   0:01  0.00% syslogd
 2214 root            1  20    0 14128K  1808K nanslp  7   0:01  0.00% cron
85455 root            1  20    0 72028K 12384K select  3   0:00  0.00% smbd
24801 root            5  20    0 54244K  5252K uwait   4   0:00  0.00% sshd
46823 root            1  52    0 14132K  2024K rpcsvc  2   0:00  0.00% rpc.lockd
27550 root            1  44    0 77620K 16548K piperd  1   0:00  0.00% php-cgi
24803 root            1  21    0 14476K  3600K ttyin   3   0:00  0.00% csh
46805 root            1  20    0   268M  1992K select  7   0:00  0.00% rpc.statd
 1596 root            1  20    0  6280K   704K select  0   0:00  0.00% devd
27551 root            1  45    0 16560K  2244K CPU3    3   0:00  0.00% top
27481 root            1  21    0 72028K 12296K select  1   0:00  0.00% smbd

This seems to indicate some sort of ZFS I/O wait state but I'm unable to further troubleshoot the issue. No SMART errors on the drives which are all online and no running ZFS operations in progress such as a scrub. Where do I go next?

Dave

[raulfg3]

monitor your ZFS performance on Disks|ZFS|Pools|I/O statistics post after a BIG data copy.

[siftu]

You should updated too. If you had AIO enabled under CIFS on that build there were problems. It has been fixed in recent versions.

[randomparity]

Thanks for the suggestions. Turns out all of the Seagate drives on the controller were generating timeout messages when I examined the Areca's event log (the Hitachi drives were all running normally). I was surprised the driver didn't generate any messages in the system log otherwise I would have noticed the problem earlier.

Eventually the controller failed one of the Seagate drives so I swapped them all out and rebuilt the pool. No ZFS slowdowns or event log messages so far (fingers crossed).

Dave