Poor performance after migrating from 9.3 to 10.2

[nef]

Hi all!

I have been happily using N4F 9.3 for about a year - no problems at all, performance was completely fine.
However, after migrating to 10.2, the NAS is struggling with more difficult tasks.
In example copying a ~5gb file - it will be hanging after 1gb - no matter is it SMB share or WinSCP.
During this copying, the webGUI becomes unrefreshable and movie from SMB stops playing.

Here are the specs:

AMD 5350
Asrock AM1H-ITX
4gb non-ecc ddr3
3TB WD Red UFS storage
32Gb Sandisk usb3 dongle
Embedded install

Usage:
-Samba
-Owncloud
-Transmission

During N4F 10.2 install there were some problems with reading/writing, they were gone after turning off USB3 controller from BIOS.

I would stay with N4F 9.3, but I want to use bullit-in VirtualBox.

[Lee Sharp]

Try to ssh in beofre you start the copy and see what is happening from the cli. Loot at top and memory use... I suspect that you are hitting some memory limitations. Setting the max memory that ZFS can grab could help.

[Knappe]

I have exact the same problems (version NAS4Free 10.2.0.2.2268): poor performance with smb (max. 1MB/s), nfs and (write IO´s) afp.

Syslog contains these entries every 10 sec:

Feb 5 23:23:13 my server winbindd[3539]: sid : S-0-0
Feb 5 23:23:13 myserver winbindd[3539]: result : NT_STATUS_NONE_MAPPED
Feb 5 23:23:13 myserver winbindd[2535]: [2016/02/05 23:23:13.056021, 1, pid=2535, effective(0, 0), real(0, 0)] ../librpc/ndr/ndr.c:439(ndr_print_function_debug)
Feb 5 23:23:13 myserver winbindd[2535]: wbint_Uid2Sid: struct wbint_Uid2Sid
Feb 5 23:23:13 myserver winbindd[2535]: out: struct wbint_Uid2Sid
Feb 5 23:23:13 my server winbindd[2535]: sid : *

In smb4.conf:

...
syslog = 1
log level = 1
...

I didn´t find any debug-level entries elsewhere.

ssh/top:

last pid: 5168; load averages: 5.42, 5.14, 4.94 up 0+03:56:43 23:29:26
45 processes: 4 running, 41 sleeping
CPU: 29.7% user, 0.0% nice, 57.5% system, 12.7% interrupt, 0.0% idle
Mem: 418M Active, 13G Inact, 2082M Wired, 66M Cache, 1685M Buf, 85M Free
Swap: 8096M Total, 8096M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1742 root 1 82 0 14548K 1620K CPU0 0 87:09 35.35% syslogd
4831 root 2 31 0 525M 217M uwait 1 25:54 32.23% smbd
4705 root 1 52 0 131M 10188K select 1 31:51 23.00% afpd
3539 root 1 33 0 1617M 1333M select 1 38:44 19.78% winbindd
2535 root 1 33 0 298M 13432K select 1 31:51 15.97% winbindd
1825 root 4 52 0 12360K 1532K RUN 1 17:06 15.23% nfsd
4706 1028 1 25 0 86720K 17760K RUN 1 18:39 9.57% cnid_dbd

Only 512 Kb/s !!!

No timing difference between Win7 and OSX shares and access.

I reduced AIO read and write buffer to 64 --> read access much more better, no changes for writing.
In version 9.3 I got 25-30 MB/s (from Win7- and OSX clients).

ssh/dd:

# time dd if=/dev/zero of=/mnt/backupdir/10,4g.img bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes transferred in 643.520914 secs (16294358 bytes/sec)
0.750u 118.749s 10:43.55 18.5% 24+168k 78+81188io 0pf+0w

not great !

Any help is appreciated !

Specs:
Asus B150ßM-C D3 (Skylake)
16 GB RAM
3 TB Seagate ST3000VX000 UFS storage
8 TB Seagate ST8000AS000 UFS storage
embedded install
No ZFS activated

Usage:
- Samba
- AFP
- NFS

[Knappe]

most solved:

In settings
1) System Advanced
- Tuning 'on' (not knowing which hidden 'features' behind)
- Power Mode to 'maximum'
Disk access performance increased up to 176 MB/s (see note 1)

2) Services CIFS/SMB under the 'Auxiliary Parameters' added
- TCP_NODELAY
- IPTOS_LOWDELAY.
see also thread viewtopic.php?t=206; both not enabled by default.
Performance increased to 30-35 MB/s
Changing AIO (Asynchronous I/O) had no measurable effect.

3) Services CIFS/SMB
- Send Buffer Size = 131070
- Receive Buffer Size = 131070
4) System Advanced/sysctl.conf
- net.inet.tcp.recvbuf_inc = 524288
- net.inet.tcp.recvbuf_max = 4194304
- net.inet.tcp.recvspace = 131070
- net.inet.tcp.sendbuf_inc = 131070
- net.inet.tcp.sendbuf_max = 4194304
- net.inet.tcp.sendspace = 131070
Performance increased up to 69 MB/s (see note 2).
(if Send Buffer Size & Receive Buffer Size = 128480 (default): 28-30 MB/s)

In Win7 clients (VM´s in Parallels, NIC Intel PRO/1000 MT)) performance only increased to 30-35 MB/s.
That's why set in device manager
5) network adapter
- receive buffer = 2048 (= maximum)
- send buffer = 2048 (maximum)
Performance increased up to 43-45 MB/s.

Are 69 MB/s (45 MB/s Win7) in fact the maximum performance ?



Note 1
measured in N4F console:
Disk 1 - Seagate SV35 3 TB

Code: Select all

# dd if=/dev/zero of=/mnt/disk1/test1.dd bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 6.355522 secs (164986601 bytes/sec)

--> ~ 157 MB/s.
Disk 2 - Seagate Archive disk ST8000AS0002 8 TB

Code: Select all

# dd if=/dev/zero of=/mnt/disk2/test1.dd bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 5.675190 secs (184764916 bytes/sec)

--> ~ 176 MB/s.

Note 2
measured ESXi server 6.0.0 (console) --> N4F-Server via nfs-share:

Code: Select all

time /sbin/vmkfstools -c 1G /vmfs/volumes/BackupServer-nfs-share/ESXserver/testdisk1.vmdk -d eagerzeroedthick
Creating disk '/vmfs/volumes/BackupServer-nfs-share/ESXserver/testdisk1.vmdk' and zeroing it out...
Create: 100% done.
real	0m 15.93s
user	0m 1.18s
sys	0m 0.00s

--> 1 (GB) x 1024 ( to get MB) : 15 (sec) = 68,26 MB/s

Note 3
Nic: Intel I210T1BLK PCIe x1 LAN Adapter

Note 4
many winbindd entries in syslog resulting in incorrect ldap parameters (Access/LDAP) after the restore of the config file.

Note 5
before and after the optimization, the results were almost identical with iperf3 (measured without any special parameters).[/size]