What is my ZFS doing?

[sunshine]

I'm a relatively new ZFS user so forgive me if this is a dumb question..

I've recently built a NAS4Free system, and when I look at my ZFS IO Statistics, I see the details pasted below. If I'm reading this correctly, and it's been this way for several hours, it seems my system is quite active.

I just finished copying about 5TB of data to it about 4 hours ago, but the copy operation is complete and there are NO clients currently.

The design is pretty straight forward - a 15 disk raidz2 (in a Dell MD1000 expansion, 15x SATA enterprise 1TB, via Dell SAS 6gbps HBA (not raid, raw disk)), 2x 120gb SATA SSD on a Perc 6/i RAID card, both RAID 0, sliced into two 8gb virtual devices with 100% of the Perc 6/i's 512mb cache dedicated to write-back cache on them, which I use for my ZIL mirror, and two 16gb virtual devices for my L2ARC log. The rest of the SSD is unassigned. I might use it later for something, or expand my L2ARC.

What is the system up to? Am I right in saying that it's doing a lot of writes? Any idea as to why, what it's up to, or how I can determine what it's doing?

Code: Select all

                capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
space         5.19T  8.43T      2    727   304K  69.6M
  raidz2      5.19T  8.43T      2    413   304K  47.6M
    da0.nop       -      -      0     56  23.9K  3.88M
    da1.nop       -      -      0     56  23.7K  3.88M
    da2.nop       -      -      0     56  26.1K  3.88M
    da3.nop       -      -      0     56  23.9K  3.88M
    da4.nop       -      -      0     56  23.6K  3.88M
    da5.nop       -      -      0     56  26.1K  3.88M
    da6.nop       -      -      0     56  23.9K  3.88M
    da7.nop       -      -      0     41  23.9K  3.89M
    da8.nop       -      -      0     41  26.2K  3.89M
    da9.nop       -      -      0     41  24.1K  3.89M
    da10.nop      -      -      0     41  23.8K  3.89M
    da11.nop      -      -      0     41  26.2K  3.89M
    da12.nop      -      -      0     41  24.1K  3.89M
    da13.nop      -      -      0     41  23.8K  3.89M
    da14.nop      -      -      0     41  26.2K  3.89M
logs              -      -      -      -      -      -
  mirror       140K  7.94G      0    314      0  22.0M
    mfid2         -      -      0    314     15  22.0M
    mfid3         -      -      0    314     15  22.0M
cache             -      -      -      -      -      -
  mfid4       16.0G     8M      0     22  1.04K  2.74M
  mfid5       16.0G     8M      0     22  1.46K  2.76M
------------  -----  -----  -----  -----  -----  -----

[raulfg3]

Please repost your new ZFS IO Statistics you will see other values.

Values are not instantaneous, but I can't say how many time is involved in this average.

[sunshine]

Please repost your new ZFS IO Statistics you will see other values.

Values are not instantaneous, but I can't say how many time is involved in this average.

Thanks, that confirms my hunch. So am I correct in saying my disks are "up to something"? I think I'm right about this - the numbers refresh every few seconds and appear to be the direct output of zpool iostat, and indicate a decent rate of data transfer.

Is there any way to determine what it is the disks are so busy with? In case it matters, I did just load them with 5TB of data.

Latest Stats:

Code: Select all

                 capacity     operations    bandwidth
pool          alloc   free   read  write   read  write
------------  -----  -----  -----  -----  -----  -----
space         5.19T  8.43T      4    635   534K  60.8M
  raidz2      5.19T  8.43T      4    360   534K  41.6M
    da0.nop       -      -      1     49  43.4K  3.39M
    da1.nop       -      -      1     49  43.1K  3.39M
    da2.nop       -      -      1     49  44.0K  3.39M
    da3.nop       -      -      1     49  43.5K  3.39M
    da4.nop       -      -      1     49  43.0K  3.39M
    da5.nop       -      -      1     49  44.0K  3.39M
    da6.nop       -      -      1     49  43.4K  3.39M
    da7.nop       -      -      1     36  43.3K  3.39M
    da8.nop       -      -      1     36  44.2K  3.39M
    da9.nop       -      -      1     36  43.7K  3.39M
    da10.nop      -      -      1     36  43.3K  3.39M
    da11.nop      -      -      1     36  44.2K  3.39M
    da12.nop      -      -      1     36  43.7K  3.39M
    da13.nop      -      -      1     36  43.3K  3.39M
    da14.nop      -      -      1     36  44.2K  3.39M
logs              -      -      -      -      -      -
  mirror       140K  7.94G      0    274      0  19.2M
    mfid2         -      -      0    274     13  19.2M
    mfid3         -      -      0    274     13  19.2M
cache             -      -      -      -      -      -
  mfid4       16.0G     8M      0     19    934  2.40M
  mfid5       16.0G     8M      0     19  1.28K  2.41M
------------  -----  -----  -----  -----  -----  -----

[raulfg3]

please post zpool status and output of top to see what process are consuming CPU.

[00Roush]

If you are looking at webgui ZFS stats I have found they are not always up to date. In a console window you can run top and then press m. That will change view to show IO. To check disk load you can use zpool iostat -v 1 to show IO per second. Also gstat can show you a percentage of how busy disks are.

00Roush

[sunshine]

Thanks for the ideas everyone, I appreciate the comments. Since posting this, I had to blow away my configuration and start over. Same config, but this time without a silly and difficult to correct mistake I made with NFS.

When I originally configured NFS, I exported my pool directory "/mnt/space", enabled subdirectories in the NFS share config, and then accessed the datasets via NFS by CDing into them. I have datasets named Movies, Music, etc. and thought I would avoid configuring multiple NFS exports this way. Thought it was a nice shortcut.. I couldn't have been more wrong.

Turns out that exporting "/mnt/space/", mounting "nas4free:/mnt/space /nas" from a Linux client, then CDing into "/nas/Movies" DOES NOT access the Movies dataset. What it does is access the subdirectory mount point off the pool parent directory - IE: not the dataset, but it's mountpoint on the NAS4Free host. After uploading about 5TB, this was confirmed on the NAS4Free host by looking in the /mnt/space/Movies directory and finding it completely empty. Where did my data go, I wondered.

On the NAS4free host, I "umount /mnt/space/Movies" and "ls /mnt/space/Movies" and found all my data there in the mountpoint.. I have no idea where it was being stored, since that's outside any dataset and is just a subdirectory off the pool itself. It had to be going into the pool, since there was room for it on the root filesystem. I tried moving the data from there into the dataset, which I did by "umount /mnt/space/Movies && mv /mnt/space/Movies /mnt/space/THEYREHERE && zfs mount -a && mv /mnt/space/THEYREHERE/* /mnt/space/Movies". All the data was there, in the /mnt/space/THEYREHERE directory. Unfortunately, due to all the reads and writes, it was performing more slowly than my gige network. So, considering the config mess, the need to move all the data around, etc.. I decided to blow it all away, reinstall, and start fresh.

I wonder if pumping into a subdirectory off the pool, rather than into a dataset, somehow caused my issue?

I'll report back if I encounter the same issue. Currently, the data is copying still.. It'll take a good chunk of the day.

[substr]

try turning off atime for any datasets that don't need it. That can be a source of writes when there are no 'writes' happening.

[sunshine]

I've completed loading my data, and have determined that the IO Stats tab provides bad information in the web GUI.

Running "zpool iostat -v" showed me the ACTUAL values, and the disks are in fact IDLE.

Thanks everyone for your ideas.