Server becomes unresponsive

[bgold]

I am running 10.3.0.3.2898 with SMB shares on ZFS. It seems that the server locks up after there is a lot of data copied to the shares. I have 256gb of ram on HP server class hardward and it is at 100%. Can the issue be with the ZFS tunning or is this just Samba?

Can someone recommend the optimal settings for 256gb of ram for tuning ZFS?

Thanks

[crowi]

Please provide more information: embedded or full install / with/without swap? What settings did you use for ZFS tuning?

[Jack Foobar]

Wait, did you say you have 256 GB of RAM? What does 'top -d1', 'swapinfo', 'vmstat', & 'sysctl hw.model hw.machine hw.ncpu' say? I gotta see this beast, unless it's a typo and you're running 256 MB's of RAM.

[bgold]

Please provide more information: embedded or full install / with/without swap? What settings did you use for ZFS tuning?

It is Full install with swap. Yes it is an HP ML150 with 256gb ram and over 8tb of zfs storage.

I used the zfs tuning plugin to set the parameters for the amount of memory.

What other info from what files would you like to know?

This issues seems to have started with 10.3.0.3.

I did make one change that I have not been able to test yet and that is I lowered the amount of memory available for ZFS. I won't know if this helped or not until I can test it again in about a week.

Thanks

[Jack Foobar]

Just ssh into the box, and copy and paste those commands, and post output so we can get a better idea of what's doing what. Also posting errors from your system log(via the GUI) would be useful. I can't say samba has been good to me this release either. Something is amiss. But my server never locked up. That sounds a bit more low level. Unless you mean to say samba shit the bed, and your server kept puttering along. Either way, need to look at system logs.

[bgold]

Jack,

Looks like it was actually running out of memory. I lowered the amount of memory for ZFS to 192gb and now the memory sits at 77% and its digging into the swap by 1%. I only have a 8gb swap partition setup, this was what I had setup the system for years ago when I was running only 16gb of ram. Don't think I need to increase it with 256gb of ram. This is a dual processor HP ML150 Gen9 with about 15tb of storage, ssd cache drives and boot drive, has 10gb fiber to the switch with 1gb ethernet ports.

I do agree samba just doesn't seem to perform as well as you would expect it. On a 750 mbps wireless connection with gig port ap's to the switch and 10gb to the server I'm seeing about 14 MB a sec. I haven't checked my speed lately on a wired gig connection.

Thanks

UPDATE: well for its and giggles I tried 1gb wired and it started at about 116 MB/sec then about 25% in it drops to about 16 MB/sec. That was copying 10gb of large files.

[Jack Foobar]

On a 750 mbps wireless connection with gig port ap's to the switch and 10gb to the server I'm seeing about 14 MB a sec.
UPDATE: well for its and giggles I tried 1gb wired and it started at about 116 MB/sec then about 25% in it drops to about 16 MB/sec. That was copying 10gb of large files.

I'm seeing the exact same thing. Went from 114 MB/s to 14 MB/s on Gig-E Cat 7 between SSD arrays and intel server NIC's. Testing with tcp/ip via iperf3 pegs the needle. Testing the array with:

dd if=/dev/zero of=/mnt/storage/test/t.mkv count=17000000
dd if=/dev/zero of=/mnt/storage/test/t.mkv bs=262144 count=32424

Pegs the needle. My array is fast. My network stack is fast. It's just Samba/CIFS that isn't. The System log for my box looks like a Samba mosh pit. This reminds me of what happens when a network is flapping on bonded ethernet. It goes high throughput and then drops down to nothing.

As far as your memory issue, which is likely created by what we were just talking about. You can add additional swap mounts if you have any space to do so, and it'll use them concurrently. But it shouldn't be touching swap in the first place. Not with your rig. I'm at a bit of a loss as well. I've resigned myself to disappointment until samba gets upgraded to 4.4.5 stable. This version isn't working for me.

If you want to really upset yourself, type iperf3 -s on your server, and iperf3 -c IP.of.Server on the client, and see how fast you really are. And then switch it up. That'll give you an accurate reflection of what your network should be doing, yet isn't, via samba. It really takes the NA out of NAS.

[Lord Crc]

I'm seeing the exact same thing. Went from 114 MB/s to 14 MB/s on Gig-E Cat 7 between SSD arrays and intel server NIC's. Testing with tcp/ip via iperf3 pegs the needle.

Very weird that some have slow SMB speeds. Could it be some variables that are set to suboptimal performance? I'm getting 1200/1000 MB/s sequential read/write speed over my samba share, using CrystalDiskMark (details here and the post below).

[Jack Foobar]

This is Duke Nukem IPX/Token Ring level transfer rates. The only indicator of what's different, is that samba is crapping itself, running out of memory, bitching about printers, and all kinds of shit I've never seen before. I think it's just that this version of samba wasn't configured properly or is just plain dogshit. Do you see no errors in your system logs? Or did we just get the lucky image. I'm hoping they'll upgrade the thing, because a NAS I can't write to before I die of old age is a problem.

Update: I think I found the issue. It's the security revision in samba 4.3.11 that doesn't play well with lower life forms(lower revisions). It down protocols lower level clients.

[Lord Crc]

Update: I think I found the issue. It's the security revision in samba 4.3.11 that doesn't play well with lower life forms(lower revisions). It down protocols lower level clients.

Now that you mention it, I remember I had issues with SMB3 and forced it to SMB2.

And yes, I do get the printer errors.

For reference, I've attached my samba settings. Client is Windows 8.1 x64.

[Jack Foobar]

Your config looks identical to mine, except I didn't add those auxillary options.

My aux options are only:

bind interfaces only = YES
interfaces = x.x.x.x
smb ports = 139

To keep it from being so chatty, although it isn't working much on this build.