Problem with VMs on network share

[sfranzis]

Hi *,

the files of a few virtual machines are stored on my NAS - 10.2.0.2 Revision 1778. I connect to the NAS from an iMac using Parallels Desktop for my VMs. When I connect over AFP there is no problem. When using SMB/CIFS the following happens: the VM starts up. Memory usage of the smbd process goes up to 300MB and CPU usage to 100%. Then the connection drops. The smbd process stays at 100% CPU usage and hast to be terminated with 'kill -9'

Any ideas?

Stefan

Here my smb4.conf:
[global]
server role = standalone
encrypt passwords = yes
netbios name = infinity
workgroup = MATRIX
server string = Storage Array
security = user
max protocol = SMB3
dns proxy = no
# Settings to enhance performance:
strict locking = no
read raw = yes
write raw = yes
oplocks = yes
max xmit = 65535
deadtime = 15
getwd cache = yes
socket options = TCP_NODELAY SO_SNDBUF=1048576 SO_RCVBUF=1048576
# End of performance section
unix charset = UTF-8
local master = yes
domain master = yes
preferred master = yes
os level = 35
time server = no
guest account = ftp
map to guest = Never
max log size = 100
syslog only = yes
syslog = 1
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
log level = 1
dos charset = CP437
smb passwd file = /var/etc/private/smbpasswd
private dir = /var/etc/private
passdb backend = tdbsam
idmap config * : backend = tdb
idmap config * : range = 10000-39999

[Archiv]
comment = Archiv Videocam, Software, Bilder
path = /mnt/pool1/archiv/
writeable = yes
printable = no
veto files = /.snap/.sujournal/
hide dot files = yes
guest ok = no
inherit permissions = yes
vfs objects = shadow_copy2 zfsacl aio_pthread
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = yes
shadow:format = auto-%Y%m%d-%H%M%S
shadow:snapdir = .zfs/snapshot
shadow:sort = desc
shadow:localtime = yes
veto files = /.zfs/

[tonyd]

Hi,

I am having problems as well with VMs on a network share, but they are not quite the same. I know, this is not the most optimal way of using VMs, but I sacrifice absolute speed for ZFS integrity. These VMs are ones I run on my Windows 7 workstation. They aren't meant to be on the NAS server. I use phpvirtualbox on the NAS for other VMs and it works great!

For me, I also get the sudden disconnect/hang, but Windows appears to recover after about a minute. However, because VMware hypervisor lost contact with the VMs I have to stop them and relaunch them. Sometimes if I'm lucky, I can click "retry" for it to reestablish the connection and it is able to continue. I have not seen a case where it has corrupted data yet, but maybe I've been lucky.

Anyway, this same configuration worked perfectly when I was using the last build of the 9.2 release of NAS4Free. I jumped from there to the 10.1 release last month, which went pretty smooth. I have not seen any other issues with Samba in 10.1 except for this situation.

My server set up is:
- 10.1.0.2 - Prescience (revision 1761)
- ZFS dataset where VMs are located
- CIFS/SMB enabled (configuration below)

With this server configuration on a decent AMD box, I easily saturate the gigabit interface to all of my Windows clients.

My client set up is:
- Windows 7 SP1 64-bit
- VMware Workstation 11 (latest update)
- The VMs stored on the Samba share

Whenever I give the VMs a bit of load (like loading two of them at the same time), or even when I try to do a clean install of Windows 10, quite randomly the connection would just hang. If I try to view the shares from Windows Explorer, it hangs there as well. If I try to access the shares from any other client while this is happening, the shares are fully accessible and work fine on the other clients.

One of my problems in diagnosing this after turning the log level to full, I get no logs when the problem happens. The only logs I get are from VMware and Windows Event Logs. The file it complains about in the logs is not always the same, so it isn't a specific file that it has a problem with. Here is the message from Windows Logs:
"{Delayed Write Failed} Windows was unable to save all the data for the file \\server\path\Windows 10 x64.nvram; the data has been lost. This error may be caused by network connectivity issues. Please try to save this file elsewhere."

I tried setting the CIFS/SMB log level to debug, but there is just too much noise.

I thought it might have to do with oplocks, so I disabled them all for that share, but the problem still occurred.

Stefan do you see any errors in the NAS system log?

I will try to watch the smbd processes next time this happens to see if the memory/CPU increase dramatically. But I would have expected something in the logs if that was the problem.

UPDATE: I watched the smbd process while the hang occurred. The CPU usage seemed reasonable, but the state was in kqread. I think this means it was waiting for the kernel for something. Any easy way to tell what it's waiting for? I repeated this multiple times, and every time the process was in kqread state. When the client recovered I noticed that smbd process no longer existed.

Code: Select all

PID USERNAME      THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 51884 root            2  20    0   352M 53912K kqread  3   0:37   4.98% smbd

Any suggestions are appreciated!

Here is my samba config:

Code: Select all

[global]
server role = standalone
encrypt passwords = yes
netbios name = blah
workgroup = blahblah
server string = NAS4Free Server
security = user
max protocol = SMB3
dns proxy = no
# Settings to enhance performance:
strict locking = no
read raw = yes
write raw = yes
oplocks = yes
max xmit = 65535
deadtime = 15
getwd cache = yes
socket options = TCP_NODELAY SO_SNDBUF=128480 SO_RCVBUF=128480 
# End of performance section
unix charset = UTF-8
local master = yes
domain master = yes
preferred master = yes
os level = 35
time server = yes
guest account = ftp
map to guest = Never
max log size = 100
syslog only = yes
syslog = 1
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes
log level = 1
dos charset = CP437
smb passwd file = /var/etc/private/smbpasswd
private dir = /var/etc/private
passdb backend = tdbsam
idmap config * : backend = tdb
idmap config * : range = 10000-39999
aio read size = 1024
aio write size = 1024

[public]
comment = Public share
path = /mnt/pool1/dsets/public/
writeable = yes
printable = no
veto files = /.snap/.sujournal/
hide dot files = yes
guest ok = no
vfs objects = shadow_copy2 zfsacl aio_pthread 
nfs4:mode = special
nfs4:acedup = merge
nfs4:chown = yes
shadow:format = auto-%Y%m%d-%H%M%S
shadow:snapdir = .zfs/snapshot
shadow:sort = desc
shadow:localtime = yes
veto files = /.zfs/