10Gbe crossover and 1Gbe regular network help

[ScottMN]

So I'm a bit of a noob to N4F but generally pretty good at configuring hardware and software. I'm struggling setting up a second NIC with a 10Gbe crossover cable directly to my N4F box and could use some help. I'm not trying to do LAGG, just a direct link at 10GBe between one machine and N4F as a high speed connection between the two boxes, while still keeping the 1Gbe infrastructure in place for all other network comms. Both boxes would have two NIC's - one at 10Gbe and one at 1Gbe. The reason is I move large 100GB+ backup files and virtual HDD's back and forth to the N4F box and it's saturating the 1Gbe connection. Needz more speed

Here's what I'm trying to do:

PC --> 1Gbe NIC#1 --> 1Gbe local network 192.168.1.X/16 with 1Gbe switches --> 192.168.1.1/16 router --> Internet (works as expected)
|
+--> 10Gbe NIC#2 --> 10Gbe crossover 192.168.10.X/16 directly to to NAS4Free 10Gbe NIC (struggling here)


and the NAS4Free box:

NAS4Free --->1Gbe NIC#1 --> 1Gbe local network 192.168.1.X/16 with 1Gbe switches --> 192.168.1.1/16 router --> Internet (works as expected)
|
+--> 10Gbe NIC#2 --> 10Gbe crossover 192.168.10.X/16 directly to 10Gbe PC NIC (struggling here)


If I use the 10Gbe NIC alone it's bottleneck by the lack of 10Gbe switches on my network. Yes a simple fix is buy a $800+ 10Gbe switch, but for now I'm not spending money to upgrade a switch when a direct connect should work. Can one of you wizards out there point me in the right direction? Or link to an install guide? My Google Fu is coming up short. Most of the hits are dual NIC LAGG stuff, not what I want.

Thanks!

[ScottMN]

Well it's probably user error on my part. I just realized the network masks are set to /16 so the 192.168.1.X and 192.168.10.X networks are probably colliding. I'll test it out sometime this weekend. Would love to have confirmation from an expert though...

[ScottMN]

Still struggling. Fixing the net masks to /24 was a good start. I can now ping both NIC's on the N4F server. LAN is set to the 1Gbps NIC. OPT1 is the 10Gbps NIC. I have full access to LAN (CIFS, web management, ssh, etc.). But there is nothing going on with OPT1.

To prove the hardware works I did two separate apples to apples test. First the 1gbps NIC only setup as LAN and directly connected to my PC. CIFS transfer speeds were approx 110MBs write/ 92MBs read. Switching to the 10gbps NIC and repeating showed 324MBs write/ 240MBs read. So the hardware works in single NIC mode.

I'm stuck - what's the proper way to setup my config?

[lindsay]

You still need to set a dhcp server,name server,gateway on that network if you use crossover cables.
I guess the client is set up that way? or use the extetention dhcp server in nas4free.

To make a network to work it needs dhcp server, nameserver and gateway.
If you use static ip`s you do not need dhcp server, but nameserver and gateway.

Do you then use a windows client you can use another card to do the internet (shared internet) on that subnet.

[ScottMN]

Thanks for the pointers but I was using static IP's on both ends of the 10Gbps crossover so the setups were simplified.

OK, problem solved. There is a setting under "Services->CIFS/SMB->Interface Selection". It was originally set to "LAN Only" so the file shares were only being seen on the 1Gbps NIC. Changed that to "ALL Interfaces" and boom, everything is working as expected on both NIC's.

Now when I connect to the N4F server via 192.168.1.X/24 (standard home LAN with all comps, NIC's and switches at 1Gbps) I'm seeing 115 MB/sec writes and 93 MB/sec reads which saturates the 1Gbps network. Connecting via 192.168.10.X/24 (10Gbps NIC dedicated crossover to my PC) the speeds jump up to 332 MB/sec write and 239 MB/sec reads as expected, well below the approximate 1250 MB/sec threshold for saturating the 10Gbps network.

Next up, performance tuning, probably starting with 4K blocks so rebuilding the RAIDZ2.

[ChriZathens]

Before messing with the vdevs try running iperf to see how much network throughput you can get.