Infiniband

[jasch]

At the moment(when i have time) iam experimenting with infiniband in Nas4free, since its not very well documented or supported in Freebsd, it is giving me a hard time.
So far what is done.
2 voltaire HCA 400ex, direct connected via CX4 cable.
For function testing i used Win7 and Ubuntu server (Openfabrics driver), UD mode ca. 5Gb/s, CM Mode 9Gb/s, wich is quite nice.
Now im compiled a new Nas4free with ported Driver.
https://wiki.freebsd.org/InfiniBand
CM Mode (connected mode) is not working so far, when i activating it on widows side i can copy to NAS but as soon as traffic want to go to Win i get
Packetdropps with message that those ore bigger as MTU 2044(wich is size for UD Mode).
Ifconfig mtu for infiniband is showing 65522.
UD mode is working almost fine, but not with the speed of Real Linux Host, getting arround 330MB/s testet with iperf und Data copy over CIFS/SMB.
Now i will try this with a normal Freebsd first, to get arround some of the Nas4free "Edges".
Maybe i just compile with UD Support, to see if there is any increase in Performance.

In Linux you switch beetween CM and UD mode

Code: Select all

echo datagram > /sys/class/net/ibX/mode 
echo connected > /sys/class/net/ibX/mode

since there is no such folders in Freebsd i dont now how to do it.


If someone is interrested i will post Results here.

[jasch]

Ok first Impression.
Hardware is running connection is stable using Datagram Mode (mtu 2044 max).

Iperf Nas4free Server/Win7 Client



Iperf Win7 Server/Nas4free Client



Samba Transfer Win7 Nas4free 3,8GB Testfile(Ramdisk)



I could not get the connected mode working, Win7 Ubuntu i was getting there over 700MB/s over Samba




When i activating connected mode on windows Seite then i get

Code: Select all

kernel: ib1: packet len 4203 (> 2044) too long to send, dropping

on Nas4free and no Traffic is going through.

So far that means Datagram is ok, Connected not under Freebsd, wich means arround 300MB/s is working(for 100 bucks(2 cards and a cable) this is ok).

MFG Alex

[rkagerer]

Hi jasch / Alex,

I too recently compiled a NAS4Free kernel using the instructions you referenced for Infiniband support. I'm using MHGH28-XTC cards, directly attached (no switch), over a 50ft CX4 active copper cable. On my Windows box I'm using the 3.2 drivers from Mellanox instead of the OFED ones. Getting roughly 7.5 Gbit/sec in iPerf. I put some additional quick & dirty benchmarks up here: http://goo.gl/TVmZG

Did you ever find a solution for the issue where packets > 2044 bytes were being dropped? I had to set the MTU for my IB interface in NAS4Free to 2044 to work around. Would like to try higher MTU's to see if they impact CIFS/iSCSI performance.

Also did you ever get the management utilities (specifically OpenSM) working on NAS4Free? I can't find the binaries, and am having trouble trying to build them on my vanilla FreeBSD 9.1 build. Would be grateful for some help here.

[jasch]

Hi, no i dont really try´d (lack of time)further.
I dont really find any helpfull things searching the web about this, seems Freebsd community ist not very adicted to IB.
Maybe i mail the guy who porteted the Driver, and he can bring light in this.
Since CM Mode not working just outcomment or dont put at all.

Code: Select all

options        IPOIB_CM 

Then Mtu is stock 2044.
The difference in Datagram mode speed is, because you adapters are 20gbs (i have only 10GBs Adpaters).
The tool are not compiled in Nas4free, ithink the must manually added to ports config.(have not try´d)
When you install a plain Freebsd 9 then compile a new Kernel with IB options then the tools are present(habe try´d myself).
there is a good description http://forums.freebsd.org/showthread.php?t=28792.
Since tools are only ibstat opensm...and not really nesessary(opensm are included in win Openfabrics) i dont have tested to put ot in Nas4free.


MFG Alex

[rkagerer]

Hi jash, thanks for getting back to me. I was finally able to do a vanilla FreeBSD 9.1 makeworld using WITH_OFED='yes' to build the userspace programs. I haven't yet figured out how to coerce NAS4Free's make.sh to include them in the images it outputs, but I was able to copy the opensm libs/binary onto the NAS4Free box as follows and it works:
1) Create /var/cache/
2) Copy these files in /usr/lib/:
- libibcommon.so.1
- libibmad.so.1
- libibumad.so.1
- libopensm.so.1
- libosmcomp.so.1
- libosmvendor.so.1
3) Copy opensm in /usr/bin/
4) Mark copied files as executable

Renewing my testing efforts now...

[MACscr]

what kind of progress have you guys made? I am desperately seeking a good distro with zfs and IB support. One of my arrays is going to have upwards of 28 drives, so I definitely need 10Gb IB working to get the most of out of my array. I plan to do iSCSI over it. Its just going to be an uplink to my switch where normal gb nics will be connected to it.

[jasch]

Its working except Connected Mode.
So from 10GBe you get arround 400-450 MB/s
With CM Mode you could do 700+ MB/s.(tested with Linux)
Iscsi should be already a bit faster, my Speeds are with Samba.
But you could also get 20Gbe Adapters so this should not the Needlepass, if you need more then 400 MB/s.

[MACscr]

Thanks for the reply. Ive actually already moved on to Omnios and 10Gb Ethernet instead. Found out my switch is actually 10GbE and Omnios has way better zfs performance than nas4free or freebsd in general. Plus recognized my IB card (when i thought I needed) right ouf of the gate.