*New 11.3 series Release:
2019-10-05: XigmaNAS 11.3.0.4.6928 - released, 11.2 series are soon unsupported!

*New 12.0 series Release:
2019-10-05: XigmaNAS 12.0.0.4.6928 - released!

*New 11.2 series Release:
2019-09-23: XigmaNAS 11.2.0.4.6881 - released!

We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

system bottleneck - detect leaked resources?

XigmaNAS Kernel Tune-up
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
kfnas
Starter
Starter
Posts: 67
Joined: 06 Mar 2014 18:41
Status: Offline

system bottleneck - detect leaked resources?

#1

Post by kfnas » 13 May 2014 07:33

Dear all,

I'm currently testing my home setup of C2750D4I MB with 32MB ECC and 8 NAS seagate 4T HDDs + latest VirtualBOX extension running latest 9.2.0.1 - Shigawire (revision 972)

I was able to tune the networking stack (dual LACP link), SMB cifs with W7 client (both sides running RAMDISK) with amazing 115MB/s ; 105MB/s file share including overhead.. so around 90% of gig link...
I have seen, that DD under the kernel is very slow comparing the CP, so I used to measure the connections or local disk 2 disk copy (I'm testing standalone UFS, standalone ZFS and ZFS mirror and RAM disk) with "time cp /mnt/source /mnt dst" to see the differences...

powerd is disabled running 2400MHZ, 8 cores:

When running only one session of local CP or CIFS towards cifs, it is OK... but with one virtualbox (windows xp guest) where only once core was assigned and the guest do have the virtualboxadditions installed doing some 5MB/s network/localfolder stuff, I'm not able paralell to copy from CIFS/RAM disk or vice versa to achieve more then 3MB/s (instead of at least 105MB/s)

here is what top says:

last pid: 9833; load averages: 1.73, 1.43, 0.96 up 0+20:12:26 05:11:45
41 processes: 3 running, 37 sleeping, 1 zombie
CPU: 2.8% user, 0.0% nice, 14.1% system, 1.3% interrupt, 81.8% idle
Mem: 109M Active, 10G Inact, 19G Wired, 1165M Cache, 1671M Buf, 281M Free
ARC: 4233M Total, 4199M MFU, 17M MRU, 16K Anon, 9723K Header, 7834K Other
Swap:

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
31394 vboxuser 21 21 0 3364M 3160M select 0 509:07 116.31% VBoxHeadless
40057 root 1 4 0 86836K 18200K select 0 0:42 0.49% smbd
5572 root 1 20 0 16596K 2048K CPU7 7 0:00 0.10% top
2935 root 1 52 0 39416K 4428K kqread 5 1:09 0.00% lighttpd
2949 vboxuser 12 20 0 144M 13892K uwait 6 1:04 0.00% VBoxSVC
2947 vboxuser 1 20 0 65824K 7392K select 3 0:22 0.00% VBoxXPCOMIPCD
2461 root 1 20 0 54868K 4048K select 1 0:11 0.00% snmp-ups
2963 vboxuser 12 20 0 152M 17164K uwait 5 0:05 0.00% vboxwebsrv
2463 root 1 20 0 20392K 2804K select 2 0:03 0.00% upsd
2496 root 1 20 0 20404K 2852K nanslp 4 0:02 0.00% upsmon
2519 root 1 20 0 29760K 3616K select 6 0:01 0.00% proftpd
87918 root 1 20 0 76212K 7612K select 4 0:01 0.00% smbd
6519 root 1 20 0 56272K 3836K select 4 0:01 0.00% sshd
87914 root 1 20 0 67596K 5112K select 1 0:01 0.00% nmbd
3055 root 1 52 0 14536K 1632K wait 2 0:01 0.00% sh
87935 root 1 20 0 78644K 7864K select 2 0:00 0.00% smbd
2473 root 1 20 0 20400K 2852K nanslp 6 0:00 0.00% upslog
2325 root 1 20 0 12112K 1404K select 5 0:00 0.00% syslogd
87926 root 1 20 0 76224K 7680K select 0 0:00 0.00% smbd
2677 root 1 20 0 14176K 1516K nanslp 1 0:00 0.00% cron
60288 root 1 20 0 56272K 3872K select 5 0:00 0.00% sshd
9776 root 1 37 0 91660K 13240K piperd 3 0:00 0.00% php-cgi
9811 root 1 43 0 20364K 1596K CPU4 4 0:00 0.00% netstat
2119 root 1 20 0 6280K 560K select 2 0:00 0.00% devd
60607 root 1 20 0 14508K 2976K pause 5 0:00 0.00% csh
9777 root 1 52 0 91660K 11236K lockf 1 0:00 0.00% php-cgi
6521 root 1 20 0 14508K 2676K ttyin 5 0:00 0.00% csh
3163 root 1 52 0 14508K 2640K pause 1 0:00 0.00% csh
3160 root 1 52 0 45332K 1816K wait 4 0:00 0.00% login
9831 root 1 72 0 14164K 1968K CPU3 3 0:00 0.00% uptime
3171 root 1 52 0 14536K 2208K ttyin 6 0:00 0.00% sh
9822 root 1 52 0 14536K 1764K wait 4 0:00 0.00% sh
2631 root 1 20 0 28908K 3356K select 5 0:00 0.00% sshd
3161 root 1 52 0 12084K 1356K ttyin 5 0:00 0.00% getty
3162 root 1 52 0 12084K 1356K ttyin 2 0:00 0.00% getty
9833 root 1 52 0 16312K 1604K piperd 2 0:00 0.00% grep
9832 root 1 52 0 12136K 1556K piperd 1 0:00 0.00% awk
2494 root 1 52 0 20404K 2740K piperd 4 0:00 0.00% upsmon
8804 root 1 20 0 76212K 7620K select 2 0:00 0.00% smbd
2962 root 1 52 0 12084K 1308K sigwai 2 0:00 0.00% daemon

the question is,

where is the bottleneck then, when the system resources looks free?

also local copy of the 8Gig test binary file from unused UFS disk to local RAMDISK take instead of 17 seconds - 50 seconds... so it does not matter wheter SMB/network is in use or not.... just the virtualbox and one guest... (if the guest is idle, then the local disk2ramdisk copy takes around 32 sec instead of usual17...) (VT-x is enabled)

please help how to detect the bottleneck or leaked resources in freebsd nas4free

thankyou!

Onichan
Advanced User
Advanced User
Posts: 237
Joined: 04 Jul 2012 21:41
Status: Offline

Re: system bottleneck - detect leaked resources?

#2

Post by Onichan » 13 May 2014 17:12

Well you do know those Avoton cores aren't that particularly fast so my first guess would be that's your bottle neck since you only gave it one core. If I remember each core is about half as powerful as a ~3GHz i3 core. What CPU usage does your virtual show within windows?

Is there a reason you are using XP? It has been discontinued and you are forced to use the inferior version of cifs with it as well. Now I know that is not the bottle neck in this case, but it is a bottle neck in general.

Also what speeds does XP show the NIC connected as? Plus is the NIC NAT'ing or bridging, maybe bridging the adapter would be better? It could also be XP VBox NIC adapter drivers just suck. Again I have only ever tested Windows7 in VBox and I get good performance.

kfnas
Starter
Starter
Posts: 67
Joined: 06 Mar 2014 18:41
Status: Offline

Re: system bottleneck - detect leaked resources?

#3

Post by kfnas » 14 May 2014 08:30

Onichan wrote:Well you do know those Avoton cores aren't that particularly fast so my first guess would be that's your bottle neck since you only gave it one core. If I remember each core is about half as powerful as a ~3GHz i3 core. What CPU usage does your virtual show within windows?

Is there a reason you are using XP? It has been discontinued and you are forced to use the inferior version of cifs with it as well. Now I know that is not the bottle neck in this case, but it is a bottle neck in general.

Also what speeds does XP show the NIC connected as? Plus is the NIC NAT'ing or bridging, maybe bridging the adapter would be better? It could also be XP VBox NIC adapter drivers just suck. Again I have only ever tested Windows7 in VBox and I get good performance.
I have recently tested w7 64bit, it is the same... the cpu can be idle, it does not help much. I'm using only bridging. The performance under VB is very good, what is not good is, when VB runs (as mentioned before also at idle) then the samba of N4F is very very slow (around 3MB/s sometimes 15MB/s)... without VB, 115MB/s so around 920Mbit/s per interface...

I would not say the avoton is the bottleneck, the memory copy from RAM disk 2 null is around 1870MB/s that is fair RAM speed.. I have to detect the bottleneck in the system.. either it is a scheduler (but the cpu is average at 15-20% and only on some cores) or some system priority? Architecture of FreeBSD is not clear to me..

to summarise my issue:

when running VB (w7 64bit guest) on N4F, the guest runs natively good.. but even when idling, the N4F samba or system I/O processes are slow (I have 32GB of ram)

User avatar
b0ssman
Forum Moderator
Forum Moderator
Posts: 2458
Joined: 14 Feb 2013 08:34
Location: Munich, Germany
Status: Offline

Re: system bottleneck - detect leaked resources?

#4

Post by b0ssman » 14 May 2014 08:41

have you tried setting the entire thing up in a esxi server? to see if its a virtualbox problem and not a general one.
Nas4Free 11.1.0.4.4517. Supermicro X10SLL-F, 16gb ECC, i3 4130, IBM M1015 with IT firmware. 4x 3tb WD Red, 4x 2TB Samsung F4, both GEOM AES 256 encrypted.

kfnas
Starter
Starter
Posts: 67
Joined: 06 Mar 2014 18:41
Status: Offline

Re: system bottleneck - detect leaked resources?

#5

Post by kfnas » 14 May 2014 15:32

b0ssman wrote:have you tried setting the entire thing up in a esxi server? to see if its a virtualbox problem and not a general one.
no, not yet, but this avoton does not support VT-d (direct IO), so it could be useless to setup storage for the N4F and then paralell run the second guest...

the thing is, that even if I run IPERF then on the N4F with nice -20 priority, I get very bad throughput = 48Mbit for instance... when the Virtual box guest is paused/stopped but the kernel modules are installed, I can get back in about 30-60 seconds after the pause of the guest the original throughput = 920Mbit over Iperf, 110-115MB/s over SMB or local copy from ram disk to null ~ 1800MB/s

I have tried to set up more then 1 core for the guest, to change the execution cap from 100% to 40% but only the performance of the guest W7 64 was affected... not the system IO..

so... what I'm getting here is clear IO stuck of the FreeBSD system... when running VB... even with idling or full performing guest...

who can help here? I mean how to debug the bottleneck? How to isolate cores for SMBD or VB?

Post Reply

Return to “XigmaNAS Kernel Tune-up”