*New 12.1 series Release:
2019-11-08: XigmaNAS 12.1.0.4.7091 - released!

*New 11.3 series Release:
2019-10-19: XigmaNAS 11.3.0.4.7014 - released


We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

procstat: 'pool read only' zpool: 'more writes than reads'

Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
harryc
Starter
Starter
Posts: 25
Joined: 08 Nov 2012 22:12
Status: Offline

procstat: 'pool read only' zpool: 'more writes than reads'

#1

Post by harryc » 29 Jan 2013 16:03

I need help tracking down why a zpool shows more writes than reads when the zpool is healthy and there is exactly one read-only process referencing the pool.

procstat -fa shows just 1 10g or so file open, for reading, on a zpool that has just one 3 tb disk. No other references in procstat to that pool. That file was written when the disk was nearly full. Now the disk is about half full. That file is being compared byte-for-byte with a file on another zpool using fdupes. But zpool iostat shows more writing going on to that drive than reading.

However, zpool iostat shows:

Code: Select all

nas4free:/mnt/pool1/wip_archive# zpool iostat -v
                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
pool1                                   3.02T  2.42T    115      1  14.2M  8.21K
  raidz1                                3.02T  2.42T    115      1  14.2M  8.21K
    gptid/650adf1b-bc5a-11e0-96d1-001372091de6      -      -     81      0  3.55M  2.98K
    gptid/68c7f7bd-bc5a-11e0-96d1-001372091de6      -      -     81      0  3.55M  2.89K
    gptid/6a0a404d-bc5a-11e0-96d1-001372091de6      -      -     81      0  3.55M  2.98K
    gptid/6b257bec-bc5a-11e0-96d1-001372091de6      -      -     80      0  3.55M  2.92K
--------------------------------------  -----  -----  -----  -----  -----  -----
pool2                                   2.01T   726G    101    104  12.4M  12.2M
  ada0.nop                              2.01T   726G    101    104  12.4M  12.2M
--------------------------------------  -----  -----  -----  -----  -----  -----

Code: Select all

nas4free:/mnt/pool1/wip_archive# zpool status pool2
  pool: pool2
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	pool2       ONLINE       0     0     0
	  ada0.nop  ONLINE       0     0     0
Something bad happens to files written when the disk is nearly full, something that stays bad even when the disk becomes less full. The system has 4G and 1.5G of it is free. 977M 'wired'. Environment is an unmodified nas4free livecd.

How to tell what's writing on a device when procstat says nothing is writing on the device?

harryc
Starter
Starter
Posts: 25
Joined: 08 Nov 2012 22:12
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#2

Post by harryc » 29 Jan 2013 16:17

Here's some detail from procstat:

Won't all fit in this, here's the whole of it:
http://pastebin.com/XeDqEJS3

As much as will fit here:

Code: Select all

nas4free:/mnt/pool1/wip_archive# procstat -f 7594
  PID COMM               FD T V FLAGS     REF  OFFSET PRO NAME        
 7594 fdupes           text v r r--------   -       - -   /usr/local/bin/fdupes
 7594 fdupes           ctty v c rw-------   -       - -   /dev/pts/0        
 7594 fdupes            cwd v d r--------   -       - -   /mnt/pool1/wip_archive
 7594 fdupes           root v d r--------   -       - -   /                 
 7594 fdupes              0 v c rw-------   7 6289204 -   /dev/pts/0        
 7594 fdupes              1 v r -w-------   1       0 -   /mnt/pool1/wip_archive/dupes
 7594 fdupes              2 v c rw-------   7 6289204 -   /dev/pts/0        
 7594 fdupes              3 v r r--------   1 311427072 -   /mnt/pool1/wip_archive/oldmediawip/LizzyProject/Scene 5.m2v
 7594 fdupes              4 v r r--------   1 311427072 -   /mnt/pool2/videos/hs/LizzyProject/Scene 5.m2v
nas4free:/mnt/pool1/wip_archive# procstat -kka
  PID    TID COMM             TDNAME           KSTACK                       
    0 100000 kernel           swapper          mi_switch+0x186 sleepq_timedwait+0x42 _sleep+0x1cf scheduler+0x34b mi_startup+0x77 btext+0x2c 
    0 100012 kernel           firmware taskq   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100016 kernel           thread taskq     mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100019 kernel           kqueue taskq     mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100021 kernel           acpi_task_0      mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 
    0 100022 kernel           acpi_task_1      mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 
    0 100023 kernel           acpi_task_2      mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 
    0 100024 kernel           ffs_trim taskq   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100058 kernel           mca taskq        mi_switch+0x186 sleepq_wait+0x42 msleep_spin+0x194 taskqueue_thread_loop+0x67 fork_exit+0x11f fork_trampoline+0xe 
    0 100059 kernel           system_taskq_0   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100060 kernel           system_taskq_1   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100400 kernel           zio_null_issue   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100401 kernel           zio_null_intr    mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100402 kernel           zio_read_issue_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100403 kernel           zio_read_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100404 kernel           zio_read_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100405 kernel           zio_read_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100406 kernel           zio_read_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100407 kernel           zio_read_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100408 kernel           zio_read_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100409 kernel           zio_read_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100410 kernel           zio_read_intr_0  mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100411 kernel           zio_read_intr_1  mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100412 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100413 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100414 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100415 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100416 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100417 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100418 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100419 kernel           zio_write_intr_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100420 kernel           zio_write_intr_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100421 kernel           zio_write_intr_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100422 kernel           zio_write_intr_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100423 kernel           zio_write_intr_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100424 kernel           zio_write_intr_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100425 kernel           zio_write_intr_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100426 kernel           zio_write_intr_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100427 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100428 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100429 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100430 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100431 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100432 kernel           zio_free_issue_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100433 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100434 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100435 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100436 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100437 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100438 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100439 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100440 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100441 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100442 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100443 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100444 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100445 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100446 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100447 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100448 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100449 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100450 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100451 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100452 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100453 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100454 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100455 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100456 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100457 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100458 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100459 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100460 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100461 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100462 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100463 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100464 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100465 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100466 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100467 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100468 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100469 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100470 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100471 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100472 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100473 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100474 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100475 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100476 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100477 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100478 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100479 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100480 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100481 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100482 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100483 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100484 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100485 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100486 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100487 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100488 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100489 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100490 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100491 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100492 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100493 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100494 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100495 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100496 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100497 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100498 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100499 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100500 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100501 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100502 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100503 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100504 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100505 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100506 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100507 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100508 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100509 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100510 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100511 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100512 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100513 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100514 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100515 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100516 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100517 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100518 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100519 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100520 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100521 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100522 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100523 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100524 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100525 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100526 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100527 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100528 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100529 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100530 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100531 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100532 kernel           zio_free_intr    mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100533 kernel           zio_claim_issue  mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100534 kernel           zio_claim_intr   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100535 kernel           zio_ioctl_issue  mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100536 kernel           zio_ioctl_intr   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100537 kernel           zfs_vn_rele_task mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100541 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100542 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100543 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100544 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100545 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100546 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100547 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100548 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100549 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100550 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100551 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100552 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100553 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100555 kernel           zio_null_issue   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100556 kernel           zio_null_intr    mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100557 kernel           zio_read_issue_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100558 kernel           zio_read_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100559 kernel           zio_read_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100560 kernel           zio_read_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100561 kernel           zio_read_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100562 kernel           zio_read_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100563 kernel           zio_read_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100564 kernel           zio_read_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100565 kernel           zio_read_intr_0  mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100566 kernel           zio_read_intr_1  mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100567 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100568 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100569 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100570 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100571 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100572 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100573 kernel           zio_write_issue_ mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100574 kernel           zio_write_intr_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100575 kernel           zio_write_intr_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100576 kernel           zio_write_intr_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100577 kernel           zio_write_intr_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100578 kernel           zio_write_intr_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100579 kernel           zio_write_intr_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100580 kernel           zio_write_intr_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100581 kernel           zio_write_intr_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100582 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100583 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100584 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100585 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100586 kernel           zio_write_intr_h mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100587 kernel           zio_free_issue_0 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100588 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100589 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100590 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100591 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100592 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100593 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100594 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100595 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100596 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100597 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100598 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100599 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100600 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100601 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100602 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100603 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100604 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100605 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100606 kernel           zio_free_issue_1 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100607 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100608 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100609 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100610 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100611 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100612 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100613 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100614 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100615 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100616 kernel           zio_free_issue_2 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100617 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100618 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100619 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100620 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100621 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100622 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100623 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100624 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100625 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100626 kernel           zio_free_issue_3 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100627 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100628 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100629 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100630 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100631 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100632 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100633 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100634 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100635 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100636 kernel           zio_free_issue_4 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100637 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100638 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100639 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100640 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100641 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100642 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100643 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100644 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100645 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100646 kernel           zio_free_issue_5 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100647 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100648 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100649 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100650 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100651 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100652 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100653 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100654 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100655 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100656 kernel           zio_free_issue_6 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100657 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100658 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100659 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100660 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100661 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100662 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100663 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100664 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100665 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100666 kernel           zio_free_issue_7 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100667 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100668 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100669 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100670 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100671 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100672 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100673 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100674 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100675 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100676 kernel           zio_free_issue_8 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100677 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100678 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100679 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100680 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100681 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100682 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100683 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100684 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100685 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100686 kernel           zio_free_issue_9 mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100687 kernel           zio_free_intr    mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100688 kernel           zio_claim_issue  mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100689 kernel           zio_claim_intr   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100690 kernel           zio_ioctl_issue  mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100691 kernel           zio_ioctl_intr   mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100692 kernel           zfs_vn_rele_task mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100695 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100696 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100721 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100722 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    0 100723 kernel           zil_clean        mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 taskqueue_thread_loop+0xbb fork_exit+0x11f fork_trampoline+0xe 
    1 100002 init             -                mi_switch+0x186 sleepq_catch_signals+0x2cc sleepq_wait_sig+0x16 _sleep+0x29d kern_wait+0x6e5 sys_wait4+0x35 amd64_syscall+0x546 Xfast_syscall+0xf7 
    2 100013 crypto           -                mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 crypto_proc+0x197 fork_exit+0x11f fork_trampoline+0xe 
    3 100014 crypto returns   -                mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 crypto_ret_proc+0x192 fork_exit+0x11f fork_trampoline+0xe 
    4 100057 ctl_thrd         -                mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 ctl_work_thread+0x300d fork_exit+0x11f fork_trampoline+0xe 
    5 100061 zfskern          arc_reclaim_thre mi_switch+0x186 sleepq_timedwait+0x42 _cv_timedwait+0x13c arc_reclaim_thread+0x29d fork_exit+0x11f fork_trampoline+0xe 
    5 100062 zfskern          l2arc_feed_threa mi_switch+0x186 sleepq_timedwait+0x42 _cv_timedwait+0x13c l2arc_feed_thread+0x1a8 fork_exit+0x11f fork_trampoline+0xe 
    5 100538 zfskern          txg_thread_enter mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x121 txg_thread_wait+0x79 txg_quiesce_thread+0xb5 fork_exit+0x11f fork_trampoline+0xe 
    5 100539 zfskern          txg_thread_enter mi_switch+0x186 sleepq_timedwait+0x42 _cv_timedwait+0x13c txg_thread_wait+0x3c txg_sync_thread+0x269 fork_exit+0x11f fork_trampoline+0xe 
    5 100693 zfskern          txg_thread_enter mi_switch+0x186 sleepq_wait+0x42 _cv_wait+0x121 txg_thread_wait+0x79 txg_quiesce_thread+0xb5 fork_exit+0x11f fork_trampoline+0xe 
    5 100694 zfskern          txg_thread_enter mi_switch+0x186 sleepq_timedwait+0x42 _cv_timedwait+0x13c txg_thread_wait+0x3c txg_sync_thread+0x269 fork_exit+0x11f fork_trampoline+0xe 
    6 100063 sctp_iterator    -                mi_switch+0x186 sleepq_wait+0x42 _sleep+0x376 sctp_iterator_thread+0x41 fork_exit+0x11f fork_trampoline+0xe 



harryc
Starter
Starter
Posts: 25
Joined: 08 Nov 2012 22:12
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#3

Post by harryc » 29 Jan 2013 16:32

Top shows the writer is 'zfskern{txg_thread_enter}' -- which begs the question-- what is being written? No thread is writing to that pool.

Only when top is asked to show threads of system processes does that line appear in the io.
ast pid: 7811; load averages: 0.84, 0.83, 0.82 up 2+17:21:56 09:31:28
400 processes: 5 running, 377 sleeping, 18 waiting
CPU: 38.7% user, 0.0% nice, 6.1% system, 0.0% interrupt, 55.2% idle
Mem: 286M Active, 128M Inact, 977M Wired, 295M Cache, 16M Buf, 1546M Free
Swap: 9277M Total, 7220K Used, 9270M Free
Displaying threads separately
PID USERNAME VCSW IVCSW READ WRITE FAULT TOTAL PERCENT COMMAND
5 root 131398 2327 15691 38354 0 54045 17547.08% zfskern{txg_thread_enter}
7594 root 326 82 326 0 0 326 105.84% fdupes
7500 root 0 0 0 0 0 0 0.00% csh
7498 root -79474 -3555 0 0 -16 -16 -5.19% sshd{sshd}
7498 root 260 467 0 0 0 0 0.00% sshd{sshd}
7498 root 1 0 0 0 0 0 0.00% sshd{sshd}
7498 root 1 0 0 0 0 0 0.00% sshd{sshd}
7498 root 79216 3088 0 0 16 16 5.19% sshd{sshd}
5103 root 0 0 0 0 0 0 0.00% csh
5101 root -600170 -227274 0 0 -19 -19 -6.17% sshd{sshd}
5101 root 590 1025 0 0 0 0 0.00% sshd{sshd}
5101 root 6 10 0 0 0 0 0.00% sshd{sshd}
5101 root 6 14 0 0 0 0 0.00% sshd{sshd}
5101 root 599568 226225 0 0 19 19 6.17% sshd{sshd}
5095 root 0 0 0 0 0 0 0.00% mDNSResponderPosix

fsbruva
Advanced User
Advanced User
Posts: 383
Joined: 21 Sep 2012 14:50
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#4

Post by fsbruva » 29 Jan 2013 17:10

More data (if that's possible) is needed.
Are you using NFS per chance? Or do you have Maildir on pool2? Are you doing any sort of automated snapshots? Is the underlying disk of ada0 capable of advanced formatting?

harryc
Starter
Starter
Posts: 25
Joined: 08 Nov 2012 22:12
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#5

Post by harryc » 29 Jan 2013 17:20

No network services in use other than sshd. No network shares of any sort. No snapshots. Disk underlying ada0 is a Seagate Barracuda 3tb. All atimes turned off, no change. Why is zfs writing to ada0 at all?

fsbruva
Advanced User
Advanced User
Posts: 383
Joined: 21 Sep 2012 14:50
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#6

Post by fsbruva » 29 Jan 2013 17:28

I have no earthly clue. Do you have a database? Also, what is the output from a couple seconds of gstat? Is it possible to reboot this server, or are you attempting to set an uptime record?

fsbruva
Advanced User
Advanced User
Posts: 383
Joined: 21 Sep 2012 14:50
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#7

Post by fsbruva » 29 Jan 2013 17:30

Do you have a jail running? Can you post the results of df -h?

fsbruva
Advanced User
Advanced User
Posts: 383
Joined: 21 Sep 2012 14:50
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#8

Post by fsbruva » 29 Jan 2013 17:31

Also, mount results.

harryc
Starter
Starter
Posts: 25
Joined: 08 Nov 2012 22:12
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#9

Post by harryc » 29 Jan 2013 18:14

Thanks for the questions. No database.

Code: Select all

dT: 10.002s  w: 10.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0      0      0      0    0.0      0      0    0.0    0.0| md0
    0      0      0      0    0.0      0      0    0.0    0.0| cd0
    0      0      0      0    0.0      0      0    0.0    0.0| md1
    0      0      0      0    0.0      0      0    0.0    0.0| iso9660/NAS4Free-x64-LiveCD-9.1.0.1
    0     58     58   7473    0.7      0      0    0.0    4.0| ada0
    0     44     44   1873    0.2      0      0    0.0    1.0| ada1
    0     44     44   1864    0.4      0      0    0.0    1.8| ada2
    0     44     44   1877    0.2      0      0    0.0    1.0| ada3
    0     44     44   1860    0.2      0      0    0.0    1.0| ada4
    0      0      0      0    0.0      0      0    0.0    0.0| ada1p1
    0     44     44   1873    0.2      0      0    0.0    1.0| ada1p2
    0      0      0      0    0.0      0      0    0.0    0.0| ada1p3
    0      0      0      0    0.0      0      0    0.0    0.0| ada2p1
    0     44     44   1864    0.4      0      0    0.0    1.9| ada2p2
    0      0      0      0    0.0      0      0    0.0    0.0| ada2p3
    0      0      0      0    0.0      0      0    0.0    0.0| ada3p1
    0     44     44   1877    0.2      0      0    0.0    1.1| ada3p2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3p3
    0      0      0      0    0.0      0      0    0.0    0.0| ada4p1
    0     44     44   1860    0.2      0      0    0.0    1.1| ada4p2
    0      0      0      0    0.0      0      0    0.0    0.0| ada4p3
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/e2f70ff7-bc58-11e0-96d1-001372091de6
    0     44     44   1873    0.2      0      0    0.0    1.1| gptid/650adf1b-bc5a-11e0-96d1-001372091de6
    0     58     58   7473    0.7      0      0    0.0    4.1| ada0.nop
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/e447ffab-bc58-11e0-96d1-001372091de6
    0     44     44   1864    0.4      0      0    0.0    1.9| gptid/68c7f7bd-bc5a-11e0-96d1-001372091de6
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/e571e5d6-bc58-11e0-96d1-001372091de6
    0     44     44   1877    0.3      0      0    0.0    1.1| gptid/6a0a404d-bc5a-11e0-96d1-001372091de6
    0      0      0      0    0.0      0      0    0.0    0.0| gptid/e696e41d-bc58-11e0-96d1-001372091de6
    0     44     44   1860    0.3      0      0    0.0    1.1| gptid/6b257bec-bc5a-11e0-96d1-001372091de6
No jails.

Code: Select all

nas4free:/boot#  df -h
Filesystem                                             Size    Used   Avail Capacity  Mounted on
/dev/md0                                               206M    200M    5.3M    97%    /
devfs                                                  1.0k    1.0k      0B   100%    /dev
procfs                                                 4.0k    4.0k      0B   100%    /proc
/dev/md1                                                30M    828k     28M     3%    /var
pool1                                                  1.8T     52k    1.8T     0%    /mnt/pool1
pool1/techlibrary                                      1.8T      5G    1.8T     0%    /mnt/pool1/techlibrary
pool1/videos                                           2.3T    533G    1.8T    23%    /mnt/pool1/videos
pool1/wip_archive                                      4.2T    2.5T    1.8T    59%    /mnt/pool1/wip_archive
pool2                                                  682G    176k    682G     0%    /mnt/pool2
pool2/videos                                           2.6T    1.9T    682G    74%    /mnt/pool2/videos
pool2/general                                          682G    144k    682G     0%    /mnt/pool2/general
pool2/music                                            687G    5.1G    682G     1%    /mnt/pool2/music
pool2/photos                                           793G    111G    682G    14%    /mnt/pool2/photos
nas4free:/boot# 

harryc
Starter
Starter
Posts: 25
Joined: 08 Nov 2012 22:12
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#10

Post by harryc » 29 Jan 2013 18:27

Do you see the really creepy part of this? gstat shows no writes. zpool iostat shows 12.3M writes/sec when there isn't even a file open for writing. Somehow it feels like once you access a file written at a time the pool was past a magic unknown amount of 'full', the thing just takes off writing at the kernel level, within zfs. What could it possibly be writing? How do you make it stop?

fsbruva
Advanced User
Advanced User
Posts: 383
Joined: 21 Sep 2012 14:50
Status: Offline

Re: procstat: 'pool read only' zpool: 'more writes than read

#11

Post by fsbruva » 30 Jan 2013 14:22

This seems to be a problem with ZFS as the pool approaches 80+% capacity. The issue is summarized here from the ZFS best practices guide:
Keep pool space under 80% utilization to maintain pool performance. Currently, pool performance can degrade when a pool is very full and file systems are updated frequently, such as on a busy mail server. Full pools might cause a performance penalty, but no other issues. If the primary workload is immutable files (write once, never remove), then you can keep a pool in the 95-98% utilization range. Keep in mind that even with mostly static content in the 95-98% range, write, read, and resilvering performance might suffer.
Some other explanation:
The fragmentation which
causes a slowdown when writing to an almost full pool is fragmentation
of the free-list/area (causing zfs to take longer to find free space
to write to) as opposed to fragmentation of the files themselves.
The files themselves will still not be fragmented any more severely
than the zfs blocksize. However, there are seeks and there are
*seeks* and some seeks take longer than others so some forms of
fragmentation are worse than others. When the free space is
fragmented into smaller blocks, there is necessarily more file
fragmentation then the file is written.
Zfs is obviously doing stuff, but I am unsure what. It would seem that one way to "make it stop" is to send the pool to another pool, then receive it back. This will cause all data to be written contiguously, and de-fragment the free space.

Also, what is the output of smartctl -a /dev/ada0?

Post Reply

Return to “ZFS (only!)”