[solved] nas4free host is not responding on one out of three interfaces

[selulven]

Hi.

NAS4Free version 10.2.0.2 - Prester (revision 2235).

My issue, in short, is that my nas4free server is responding to ICMP ping and iSCSI on one interface (or from one host), but not the other.

My physical setup is as follows:

1 nas4free server with an Intel X520-DA2 10GbE dual SFP+ NIC
1 vmware host ("vmware1") with an Intel X520-DA2 10GbE dual SFP+ NIC
1 vmware host ("vmware2") with a Broadcom NetXtreme II BCM58700 10GbE dual SFP+ NIC

Needless to say, the first SFP+ port (ix0) on the nas4free server is connected to vmware1 and the other port (ix1) is connected to vmware2, both using the same brand and model SPF+ DAC TwinAx cables.

IPv4 setup is:

nas4free ix0 -> 192.168.2.1/24
nas4free ix1 -> 192.168.3.1/24

vmware1 vmk1 -> 192.168.2.2/255.255.255.0

vmware2 vmk2 -> 192.168.3.2/255.255.255.0

(it's just pure coincidence that the vmkernels were named that way -- it's not a typo and not meant to follow the host numbers)

No one is firewalling their connections.

Here's what happens when I ping from vmware2 to nas4free. This works fine and I have a working iSCSI connection here.

Code: Select all

vmware2# vmkping 192.168.3.1

PING 192.168.3.1 (192.168.3.1): 56 data bytes
64 bytes from 192.168.3.1: icmp_seq=0 ttl=64 time=0.094 ms
64 bytes from 192.168.3.1: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 192.168.3.1: icmp_seq=2 ttl=64 time=0.080 ms

--- 192.168.3.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.080/0.086/0.094 ms


nas4free# tcpdump -i ix1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix1, link-type EN10MB (Ethernet), capture size 65535 bytes
20:55:33.371666 IP 192.168.3.2 > 192.168.3.1: ICMP echo request, id 36606, seq 0, length 64
20:55:33.371673 IP 192.168.3.1 > 192.168.3.2: ICMP echo reply, id 36606, seq 0, length 64
20:55:34.373766 IP 192.168.3.2 > 192.168.3.1: ICMP echo request, id 36606, seq 1, length 64
20:55:34.373770 IP 192.168.3.1 > 192.168.3.2: ICMP echo reply, id 36606, seq 1, length 64
20:55:35.374187 IP 192.168.3.2 > 192.168.3.1: ICMP echo request, id 36606, seq 2, length 64
20:55:35.374191 IP 192.168.3.1 > 192.168.3.2: ICMP echo reply, id 36606, seq 2, length 64

We see the ping coming in and we see the reply going out.

Now see what happens when I try the same operation on vmware1, where there is seemingly no connection to nas4free at all.

Code: Select all

vmware1# vmkping 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes

--- 192.168.2.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss


nas4free# tcpdump -i ix0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix0, link-type EN10MB (Ethernet), capture size 65535 bytes
20:57:15.019560 IP 192.168.2.2 > 192.168.2.1: ICMP echo request, id 38972, seq 0, length 64
20:57:16.021641 IP 192.168.2.2 > 192.168.2.1: ICMP echo request, id 38972, seq 1, length 64
20:57:17.022334 IP 192.168.2.2 > 192.168.2.1: ICMP echo request, id 38972, seq 2, length 64

Oddly enough the ICMP packets come through, but FreeBSD somehow decides not to respond. I tcpdumped the specific interface, so I'm assuming the packet is not being sent via the management network (192.168.0.0/24) somehow.

The reverse, pinging from nas4free to vmware1, works fine:

Code: Select all

nas4free# ping 192.168.2.2
PING 192.168.2.2 (192.168.2.2): 56 data bytes
64 bytes from 192.168.2.2: icmp_seq=0 ttl=64 time=0.211 ms
64 bytes from 192.168.2.2: icmp_seq=1 ttl=64 time=0.094 ms
64 bytes from 192.168.2.2: icmp_seq=2 ttl=64 time=0.090 ms
64 bytes from 192.168.2.2: icmp_seq=3 ttl=64 time=0.093 ms

--- 192.168.2.2 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.090/0.122/0.211/0.051 ms

It's the same story with the vmware1 iSCSI initiator trying to connect to the iSCSI target on nas4free:

Code: Select all

# tcpdump -i ix0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ix0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:39:42.108080 IP 192.168.2.2.59365 > 192.168.2.1.iscsi-target: Flags [S], seq 2211372390, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 1615169 ecr 0], length 0
21:39:45.113183 IP 192.168.2.2.59365 > 192.168.2.1.iscsi-target: Flags [S], seq 2211372390, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 1615470 ecr 0], length 0
21:39:48.323247 IP 192.168.2.2.59365 > 192.168.2.1.iscsi-target: Flags [S], seq 2211372390, win 65535, options [mss 1460,nop,wscale 9,sackOK,TS val 1615791 ecr 0], length 0

There's traffic coming in, but no replies going out.

Like I said, no one is firewalling. The nas4free firewall is disabled.

The nas4free routing table looks fine:

Code: Select all

nas4free# netstat -r
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            fw                 UGS         em1
localhost          link#5             UH          lo0
192.168.0.0        link#4             U           em1
nas4free             link#4             UHS         lo0
192.168.2.0        link#1             U           ix0
192.168.2.1        link#1             UHS         lo0
192.168.3.0        link#2             U           ix1
192.168.3.1        link#2             UHS         lo0

I can see no reason why it can ping vmware1, but not reply to ping from vmware1, and why it's different from the connection to vmware2.

vmware1's routing table looks fine:

Code: Select all

vmware1# esxcfg-route -l
VMkernel Routes:
Network          Netmask          Gateway          Interface
192.168.0.0      255.255.255.0    Local Subnet     vmk0
192.168.2.0      255.255.255.0    Local Subnet     vmk1
default          0.0.0.0          192.168.0.1      vmk0

and it looks similar to vmware2's:

Code: Select all

vmware2# esxcfg-route -l
VMkernel Routes:
Network          Netmask          Gateway          Interface
192.168.0.0      255.255.255.0    Local Subnet     vmk0
192.168.3.0      255.255.255.0    Local Subnet     vmk2
default          0.0.0.0          192.168.0.1      vmk0

Any suggestions on a path of inquiry?

Thanks.

[selulven]

OK, so I have noticed something weird. On nas4free, the Intel X520-DA2 has two MAC addresses, and they are:

ix0: ff:ff:21:ba:e9:70
ix1: 00:1b:21:ba:e9:72

Is it just me or does ix0's MAC address look weird? I guess it should have started with 00:1b like ix1's. At least that how it is on the other X520 NIC in vmware1:

00:1b:21:ba:ef:9e
00:1b:21:ba:ef:9f

I'm wondering if this is the cause of the problems.

[selulven]

That's it! I reconfigured the MAC address using:

Code: Select all

ifconfig ix0 ether 00:1b:21:ba:e9:71

and now everything works perfectly.

That's a first...