nfsv4 expired locks lost

[tuaris]

I'm getting these messages on the console for the NFS clients that use the NAS4Free NFSv4 server:

Code: Select all

nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfscl: never fnd open

The server does get busy, but not overloaded:


There are about 4 FreeBSD 9.3 boxes that connect to this NFSv4 server and the IO almost always at least 100 mbps (with some peaks to 600+mbits) 24/7.

The system is (I think) well capable of handling the load:

[daoyama]

I'm getting these messages on the console for the NFS clients that use the NAS4Free NFSv4 server:

Code: Select all

nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfscl: never fnd open
nfscl: never fnd open
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfsv4 expired locks lost
nfscl: never fnd open

I didn't see that message, but there is some tunable parameters.

The default value is:

Code: Select all

vfs.nfsd.v4statelimit: 500000
vfs.nfsd.sessionhashsize: 20
vfs.nfsd.fhhashsize: 20
vfs.nfsd.clienthashsize: 20
vfs.nfsd.statehashsize: 10

Try to increase from loader.conf.
I'm interesting about vfs.nfsd.v4statelimit.
Verify it by CLI:
# sysctl vfs.nfsd

[tuaris]

The current settings are:

Code: Select all

sysctl vfs.nfsd
vfs.nfsd.request_space_throttle_count: 0
vfs.nfsd.request_space_throttled: 0
vfs.nfsd.request_space_low: 31457280
vfs.nfsd.request_space_high: 47185920
vfs.nfsd.request_space_used_highest: 1657432
vfs.nfsd.request_space_used: 4344
vfs.nfsd.groups: 1
vfs.nfsd.threads: 32
vfs.nfsd.maxthreads: 32
vfs.nfsd.minthreads: 32
vfs.nfsd.fha.fhe_stats: No file handle entries.
vfs.nfsd.fha.max_reqs_per_nfsd: 0
vfs.nfsd.fha.max_nfsds_per_fh: 8
vfs.nfsd.fha.bin_shift: 22
vfs.nfsd.fha.enable: 1
vfs.nfsd.cachetcp: 1
vfs.nfsd.tcpcachetimeo: 43200
vfs.nfsd.udphighwater: 500
vfs.nfsd.tcphighwater: 0
vfs.nfsd.enable_stringtouid: 0
vfs.nfsd.debuglevel: 0
vfs.nfsd.enable_locallocks: 0
vfs.nfsd.issue_delegations: 0
vfs.nfsd.commit_miss: 0
vfs.nfsd.commit_blks: 0
vfs.nfsd.mirrormnt: 1
vfs.nfsd.async: 0
vfs.nfsd.server_max_nfsvers: 4
vfs.nfsd.server_min_nfsvers: 2
vfs.nfsd.nfs_privport: 0
vfs.nfsd.v4statelimit: 500000
vfs.nfsd.sessionhashsize: 20
vfs.nfsd.fhhashsize: 20
vfs.nfsd.clienthashsize: 20
vfs.nfsd.statehashsize: 10
vfs.nfsd.enable_nogroupcheck: 1
vfs.nfsd.enable_nobodycheck: 1
vfs.nfsd.disable_checkutf8: 0

Can I apply without rebooting? For example?

Code: Select all

sysctl vfs.nfsd.v4statelimit=800000
vfs.nfsd.v4statelimit: 500000 -> 800000

[daoyama]

Can I apply without rebooting? For example?

Code: Select all

sysctl vfs.nfsd.v4statelimit=800000
vfs.nfsd.v4statelimit: 500000 -> 800000

I don't know. At least, hash size must set before loading kernel.

[tuaris]

What value do you recommend I set for the hash size and state limit?
Is their an NFSv4 tuning guide I can read over?

[Parkcomm]

Dayoma - could you explain what is causing the problem the changing the vfs.nfsd.v4statelimit solves and how it might come about?

(FYI - I've seen this problem before and ended up "solving" it by falling back to NFSV3 - not ideal)

[tuaris]

I've experienced some data loss as a result of this problem. An application running on one of the NFS clients continued to write to a file and the data was never 'saved'.

I've switched back to NFSv3. Looks like NFSv4 isn't ready.

According to this page, this appears to be what's happening.
https://access.redhat.com/solutions/1179643