esxi best multiple nic config

[justin]

We use a NAS4Free with some esxi in iscsi.

We have 2 nic for iscsi on each esxi.

We have 4 nic for iscsi on NAS4free server.

Config 1:
- each nic on NAS4free is on a separete /24
- we configurate each iscsi nic with 3 IPs

exemple:
NAS4Free : 10.1.0.1/24, 10.1.1.1/24, 10.1.2.1/24, 10.1.3.1/24
esxi1 : nic1 : 10.1.0.2/24, 10.1.1.2/24 ; nic2 : 10.1.2.2/24, 10.1.3.2/24
esxi2 : nic1 : 10.1.0.3/24, 10.1.1.3/24 ; nic2 : 10.1.2.3/24, 10.1.3.3/24

This generate errors and VMware Support ask to change our config. That we do yesterday at night.

Now we have :
Config 2:
- each nic on NAS4free is on a same /24
- we configurate each iscsi nic 1 IP on the /24

exemple:
NAS4Free : 10.20.0.1/24, 10.20.0.2/24, 10.20.0.3/24, 10.20.0.4/24
esxi1 : nic1 : 10.20.0.5/24 ; nic2 : 10.20.0.6/24
esxi1 : nic1 : 10.20.0.7/24 ; nic2 : 10.20.0.8/24

There is 2 group of portals. one portal1 with 10.20.0.1/24 and 10.20.0.3/24 ; one portal2 with 10.20.0.2/24 and 10.20.0.4/24.

We don't have errors but:

Code: Select all

            em3  in      0.058 KB/s          6.384 MB/s            2.763 GB
                 out     0.000 KB/s          0.000 KB/s            4.774 KB

            em2  in      4.134 KB/s          7.548 MB/s            2.141 GB
                 out     0.000 KB/s          0.162 KB/s           12.896 KB

            em1  in     13.505 MB/s         18.193 MB/s           30.927 GB
                 out     0.000 KB/s          0.081 KB/s           16.997 KB

            em0  in     54.929 KB/s          9.930 MB/s            6.187 GB
                 out   686.133 KB/s         53.675 MB/s          116.314 GB

In out : all pass by em0 !!!!

Event the lun on portal2 use em0 for output.

Do you have a idea ?

Best regards,
Thanks

[justin]

We could use LACP but this append a level that risk to slower iSCSI access. (there are cisco 3500 switch that could support that).
And we had to found a solution to create multiple interfaces for each IP that can't be create with webgui.

Any idea?

[justin]

I found this:
http://forums.freenas.org/threads/multi ... sues.8557/

last post:
Dalibor Andzakovic, Aug 21, 2013

For a single subnet MPIO setup you need working pf which implies 9.1. You can then use the reply-to rule to make sure that the reply packet leaves via the same interface it came on.

/etc/pf.conf
Code (text):

GW="172.16.241.2"
pass in quick on em0 reply-to (em0 $GW)
pass in quick on em1 reply-to (em1 $GW)
pass in quick on em2 reply-to (em2 $GW)
pass in quick on em3 reply-to (em3 $GW)
pass out quick

does anybody try this?

[00Roush]

I think maybe this could help...

http://n4f.siftusystems.com/index.php/2 ... hing-mpio/
http://n4f.siftusystems.com/index.php/2 ... e-esxi-51/

Let us know how it goes.
00Roush

[justin]

Thank you for your answer. So the solution will be more in the first config.

I continue to search.

[alexey123]

I found this:
http://forums.freenas.org/threads/multi ... sues.8557/
does anybody try this?

Good idea, I'll try it with ipfw!

[justin]

So, i tried with:
ipfw : i can't found same pf options. can you?
pf : i can't install it. modules are here but not binaries.
setfib : not present and can't install it.
I don't found packages for pf or setfib.

I think (sometimes) that my best solution is a mix between config 1 and 2.

NAS4Free : nic1 : 10.1.0.1/24; nic2 : 10.1.1.1/24; nic3 : 10.1.2.1/24; nic4 : 10.1.3.1/24
portals : 1 portal for all luns : 10.1.0.1, 10.1.1.1, 10.1.2.1, 10.1.3.1
esxi1 : nic1 : 10.1.0.2/24 ; nic2 : 10.1.2.2/24
esxi2 : nic1 : 10.1.1.3/24 ; nic2 : 10.1.3.3/24

with this solution, i risk to refound my first problem.
http://kb.vmware.com/selfservice/micros ... Id=2017084
and for information :
http://kb.vmware.com/selfservice/micros ... Id=2012171

if i do this, does esxi1 (or esxi2 vis versa), will auto discover lun over 10.1.0.1, 10.1.1.1, 10.1.2.1, 10.1.3.1 and so create problem and slow discovery?

[alexey123]

Ipfw will make "lan-balance" work with natd. But by default natd not include into nas4free. I want try load nat kernel module and link binaries from jail to main server and wrote firewall rules for lan-balance.
BUT I NOT HAVE TIME!!!!

[justin]

Hello alexey123,

but if we add level with ipfw + natd , does'nt we have lower performances?

I tried in lab my solution:
the problem is that esxi1 show also 10.1.2.1, 10.1.3.1 paths. and this create slower actions.

Code: Select all

2014-01-09T15:55:06Z iscsid: session login failed with error 4,retryCount=0
2014-01-09T15:55:06Z iscsid: Login Target Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk2 addr=10.20.3.1:3260 (TPGT:1 ISID:0x8) err=4
2014-01-09T15:55:06Z iscsid: Login Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk2 addr=10.20.3.1:3260 (TPGT:1 ISID:0x8) Reason: 00040000 (Initiator Connection Failure)
2014-01-09T15:55:06Z iscsid: Login Target: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk1 addr=10.20.4.1:3260 (TPGT:1 ISID:0x9)
2014-01-09T15:55:06Z iscsid: Notice: Assigned (H35 T2 C2 session=113, target=3/19)
2014-01-09T15:55:06Z iscsid: cannot make a connection to 10.20.4.1:3260 (101,Network is unreachable)
2014-01-09T15:55:06Z iscsid: Notice: Reclaimed Channel (H35 T2 C2 oid=3)
2014-01-09T15:55:06Z iscsid: session login failed with error 4,retryCount=0
2014-01-09T15:55:06Z iscsid: Login Target Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk1 addr=10.20.4.1:3260 (TPGT:1 ISID:0x9) err=4
2014-01-09T15:55:06Z iscsid: Login Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk1 addr=10.20.4.1:3260 (TPGT:1 ISID:0x9) Reason: 00040000 (Initiator Connection Failure)
2014-01-09T15:55:06Z iscsid: Login Target: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk2 addr=10.20.4.1:3260 (TPGT:1 ISID:0xa)
2014-01-09T15:55:06Z iscsid: Notice: Assigned (H35 T2 C2 session=114, target=3/19)
2014-01-09T15:55:06Z iscsid: cannot make a connection to 10.20.4.1:3260 (101,Network is unreachable)
2014-01-09T15:55:06Z iscsid: Notice: Reclaimed Channel (H35 T2 C2 oid=3)
2014-01-09T15:55:06Z iscsid: session login failed with error 4,retryCount=0
2014-01-09T15:55:06Z iscsid: Login Target Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk2 addr=10.20.4.1:3260 (TPGT:1 ISID:0xa) err=4
2014-01-09T15:55:06Z iscsid: Login Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk2 addr=10.20.4.1:3260 (TPGT:1 ISID:0xa) Reason: 00040000 (Initiator Connection Failure)
2014-01-09T15:55:06Z iscsid: Login Target: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk1 addr=10.20.5.1:3260 (TPGT:1 ISID:0xb)
2014-01-09T15:55:06Z iscsid: Notice: Assigned (H35 T2 C2 session=115, target=3/1a)
2014-01-09T15:55:06Z iscsid: cannot make a connection to 10.20.5.1:3260 (101,Network is unreachable)
2014-01-09T15:55:06Z iscsid: Notice: Reclaimed Channel (H35 T2 C2 oid=3)
2014-01-09T15:55:06Z iscsid: session login failed with error 4,retryCount=0
2014-01-09T15:55:06Z iscsid: Login Target Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk1 addr=10.20.5.1:3260 (TPGT:1 ISID:0xb) err=4
2014-01-09T15:55:06Z iscsid: Login Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk1 addr=10.20.5.1:3260 (TPGT:1 ISID:0xb) Reason: 00040000 (Initiator Connection Failure)
2014-01-09T15:55:06Z iscsid: Login Target: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk2 addr=10.20.5.1:3260 (TPGT:1 ISID:0xc)
2014-01-09T15:55:06Z iscsid: Notice: Assigned (H35 T2 C2 session=116, target=3/1a)
2014-01-09T15:55:06Z iscsid: cannot make a connection to 10.20.5.1:3260 (101,Network is unreachable)
2014-01-09T15:55:06Z iscsid: Notice: Reclaimed Channel (H35 T2 C2 oid=3)
2014-01-09T15:55:06Z iscsid: session login failed with error 4,retryCount=0
2014-01-09T15:55:06Z iscsid: Login Target Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk2 addr=10.20.5.1:3260 (TPGT:1 ISID:0xc) err=4
2014-01-09T15:55:06Z iscsid: Login Failed: iqn.2007-09.jp.ne.peach.istgt:disk2 if=iscsi_vmk@vmk2 addr=10.20.5.1:3260 (TPGT:1 ISID:0xc) Reason: 00040000 (Initiator Connection Failure)
2014-01-09T15:55:06Z iscsid: DISCOVERY: transport_name=iscsi_vmk Pending=0 Failed=30

best regards,

[justin]

This config is an alternative. But the best solution should be the round robin over nic on a same broadcast domain.

I'm going to try to find yet.

[alexey123]

Hello alexey123,

but if we add level with ipfw + natd , does'nt we have lower performances?

Yes, we have lower performance, but if you use fast CPU, not as my LAB server, you will not notice a decrease in performance.
For example, I was check My Home server as iSCSI target and my LAB server as diskless client. iSCSI speed was ~90 MB/s, when I was check ATOM D525 as client - ISCSI speed was 108 MB/s.
It is very high speed, some as IDE hard disk.

I tried in lab my solution:
the problem is that esxi1 show also 10.1.2.1, 10.1.3.1 paths. and this create slower actions.
best regards,

I cannot say anythig about it - you must have any bridge or router on network...

Now I explain why your topic interest for me.
~ last year I was create 2 jails and I want define lan card for each jail.
My building:
vr0 - onboard lan 100mb/s
re0 - additional lan card 1000 mb/s
re1 - second additional lan 1000 md/s
In all cases, if I define vr0 as LAN re<n> as OPT<n> for jails, or re0 - as LAN and other to jails or re1 as LAN others to jails, output traffic go over re0 only if re0 busy little traffic was over re1 . I don't understand this issue , I see only alphabetically dependence.
Then my old server was broke, and I replace it on current motherboard ( it have 1000 mbit/s on-board LAN, it OK for me )
Now you find solution from freenas forum - how to make balance and I want check it

[Grast5150]

Justin,

I had the same issues.

1. The issue may not be esxi or iscsi but rather the quality of nics on either NAS4FREE box or the esxi hosts. What NICS are you using? Do they have good driver support for each system. Resolution: I was using desktop NICS for esxi hosts and NAS4FREE. I replaced with server quality NIC from HP which has official support from vmware and good support for FREEBSD.

2. I would have all nics on the same broadcast domain as your iscsi intiators and enable multi-path IO.

http://www.vmware.com/files/pdf/techpap ... inding.pdf

For me the issue was not with esxi or nas4free but rather low quality Broadcom NICS. Once I replaced with respective OS supports NIC, I had now issues.

Thanks.

[justin]

Hello alexey123,

but if we add level with ipfw + natd , does'nt we have lower performances?

Yes, we have lower performance, but if you use fast CPU, not as my LAB server, you will not notice a decrease in performance.
For example, I was check My Home server as iSCSI target and my LAB server as diskless client. iSCSI speed was ~90 MB/s, when I was check ATOM D525 as client - ISCSI speed was 108 MB/s.
It is very high speed, some as IDE hard disk.

I tried in lab my solution:
the problem is that esxi1 show also 10.1.2.1, 10.1.3.1 paths. and this create slower actions.
best regards,

I cannot say anythig about it - you must have any bridge or router on network...

Now I explain why your topic interest for me.
~ last year I was create 2 jails and I want define lan card for each jail.
My building:
vr0 - onboard lan 100mb/s
re0 - additional lan card 1000 mb/s
re1 - second additional lan 1000 md/s
In all cases, if I define vr0 as LAN re<n> as OPT<n> for jails, or re0 - as LAN and other to jails or re1 as LAN others to jails, output traffic go over re0 only if re0 busy little traffic was over re1 . I don't understand this issue , I see only alphabetically dependence.
Then my old server was broke, and I replace it on current motherboard ( it have 1000 mbit/s on-board LAN, it OK for me )
Now you find solution from freenas forum - how to make balance and I want check it

I understand that you want to try it!

Let me know your results

[justin]

Hello Grast5150,

1. network cards are 4 ports :
<Intel(R) PRO/1000 Network Connection 7.3.2> port 0xec00-0xec1f mem 0xf8ee0000-0xf8efffff,0xf8ec0000-0xf8edffff irq 62 at device 0.0 on pci138
I don't think it s a driver problem. (may be wrong)

My errors message on ESXi :
2014-01-09T15:55:06.820Z cpu1:2150)WARNING: ScsiUid: 411: vmhba35:C0:T0:L0: NAA identifier type has an unknown naa value of 0x3
2014-01-09T15:55:06.820Z cpu1:2148)WARNING: ScsiUid: 411: vmhba35:C0:T2:L0: NAA identifier type has an unknown naa value of 0x3
2014-01-09T15:55:06.822Z cpu1:2151)WARNING: ScsiUid: 411: vmhba35:C1:T1:L0: NAA identifier type has an unknown naa value of 0x3

2. Thank you.

Does anybody have already try LACP on NAS4Free for iSCSI nics ? but MPIO on esxi ?
I'm try this in my lab, but you know may be already the results?

Config 4:
NAS4Free : nic1-> nic4 : lacp nic : 10.1.0.1
portals : 1 portal for all luns : 10.1.0.1
esxi1 : nic1 : 10.1.0.2/24 ; nic2 : 10.1.0.3/24
esxi2 : nic1 : 10.1.0.4/24 ; nic2 : 10.1.0.5/24

[justin]

... for my lab, i have "only" a 3500XL (in prod i have 3750)

so, i don't have LACP support on 3500XL.

i'm try with others lagg modes

[justin]

Hello,

LACP does'nt work well for me and ISCSI. does any body have sucess experience with LACP/ISCSI or other solution?

Thank you,

[STAMSTER]

Forget about LACP, use only MPIO. It's the best of the both worlds!

Different subnet / VLAN for each pair of interfaces, and enable Round Robin mode on each Software storage adapter on ESXi.