Writing to zpool freezes intermittently with process state dp->dp

[krezek]

Hi,

I've been using NAS4Free for a few years now. It's been going well so far, but lately I've noticed slow performance during backups from a Windows PC writing to the N4F server using SMB. I then tried copying larger single files manually, or a series of large files, and the process freezes every few minutes for about 30-60 seconds, before continuing at full speed, only to freeze and unfreeze over and over again. This doesn't happen for every file, just every few 100s or 1000s MB in the overall file transfer. When the write process is frozen, the Web GUI and other SMB requests don't respond as well. Executing commands in an existing SSH session still works though.

Initially I thought it was an SMB issue, and tried different suggestions from other threads, until I tried using SCP, and had the exact same issue. Creating a dummy file using dd through ssh caused the same problem as well

Code: Select all

dd if=/dev/urandom of=testfile bs=8192k count=1000

There's nothing in the logs about any issues, other than the following:

ACL tag type ACL_EVERYONE. FreeBSD with ZFS? Use 'vfs objects = zfsacl'

One thing I've noticed though, is while reproducing the problem, looking at top shows the state of the process being "dp->dp", and as soon as that changes to "zio->i" and whatever else, the write action can continue.

I thought it was because less than 100M was free under Mem in top, but in other tests there was more than 1100M free Mem, so I don't think a memory issue is the cause.
I've tried turning off sync for a test dataset, and that didn't fix it either.

Basically my question is, what does dp->dp as a process state mean, and how can I prevent a process (smbd, dd, etc.) entering that state?

To my setup:
NAS4Free version: 10.3.0.3 - Pilingitam (revision 3955)
Platform: x64-embedded on Intel(R) Core(TM) i5 CPU 760 @ 2.80GHz
RAM: 12GB
zpool: 3x4TB in raidz1

Thanks for any help!

[sleid]

Make a smart report.
Check that a disk does not have writing problems.

[krezek]

Make a smart report.
Check that a disk does not have writing problems.

You're right. Out of the 3 drives, here's the info:

• Drive A is good
• Drive B has a Current_Pending_Sector of 8 and a Offline_Uncorrectable of 6
• Drive C has a Current_Pending_Sector of 440!

I'll see that I at least get a good replacement drive for drive C, probably both Drive B and C.

Thanks! Any idea what dp->dp means though?

[sleid]

dp data processus
dp->dp loop

And the health of the zfs pool?

[krezek]

The pool is healthy, it gets scrubbed once a month and maybe had an error once or twice. Scrub time is high because I shut it down twice during the scrub. Maybe I can use a drive as a write buffer until I can replace the drive(s)? Don't think a ZIL drive would help in this case.

Code: Select all

pool: zpool1
state: ONLINE
scan: scrub repaired 0 in 56h22m with 0 errors on Fri Dec 22 07:19:31 2017
config:
	NAME        STATE     READ WRITE CKSUM
	zpool1      ONLINE       0     0     0
	  raidz1-0  ONLINE       0     0     0
	    ada1    ONLINE       0     0     0
	    ada2    ONLINE       0     0     0
	    ada3    ONLINE       0     0     0

errors: No known data errors

[sleid]

The only solution is to replace the disk B then the disk C but the reconstruction operation is not without risks with 2 disks in bad condition (RaidZ1) .I hope you have a backup of your pool.

[krezek]

Thanks for your help and suggestions so far!

So you think the Offline_Uncorrectable count of 6 is worse than the Current_Pending_Sector of 440?

Also just out of curiosity, and I'm probably not going down this route, but is it possible to dd (or ddrescue) the contents of a bad drive to a new drive, and have the new drive be detected with the data? And then just do a scrub.

[sleid]

Personally I use in this case a replication box, but sometimes the discs to save have a hard time holding until the end of the copy.
But your solution is quite possible, you must make an exact copy of the disk with its structure.

[krezek]

Thanks, I'll have a look what I'll eventually end up doing. Was worried that maybe the changed UUID of the disk would be a problem, but I guess not. Thanks for all the help and info!