scsi connection errors on idle nas

[danypd69]

Hello,
I'm trying to setup an iscsi nas4free with proxmox as scsi initiator.

I keep getting connection errors on the proxmox side and I cannot find the reason for it.

This is the log on the client side:

Code: Select all

Dec  9 17:59:56 px2950-01 kernel: connection7:0: detected conn error (1021)
Dec  9 17:59:57 px2950-01 kernel: connection4:0: detected conn error (1021)
Dec  9 17:59:58 px2950-01 kernel: connection6:0: detected conn error (1021)
Dec  9 18:00:00 px2950-01 kernel: connection8:0: detected conn error (1021)
Dec  9 18:00:00 px2950-01 kernel: connection2:0: detected conn error (1021)
Dec  9 18:00:10 px2950-01 kernel: session8: session recovery timed out after 10 secs
Dec  9 18:00:10 px2950-01 kernel: session2: session recovery timed out after 10 secs
Dec  9 18:00:10 px2950-01 kernel: sd 22:0:0:0: Device offlined - not ready after error recovery
Dec  9 18:00:10 px2950-01 kernel: sd 22:0:0:0: [sdk] Unhandled error code
Dec  9 18:00:10 px2950-01 kernel: sd 22:0:0:0: [sdk] Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
Dec  9 18:00:10 px2950-01 kernel: sd 16:0:0:0: Device offlined - not ready after error recovery
Dec  9 18:00:10 px2950-01 kernel: sd 22:0:0:0: [sdk] CDB: 
Dec  9 18:00:10 px2950-01 kernel: sd 16:0:0:0: [sdg] Unhandled error code
Dec  9 18:00:10 px2950-01 kernel: sd 16:0:0:0: [sdg] Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
Dec  9 18:00:10 px2950-01 kernel: sd 16:0:0:0: [sdg] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Dec  9 18:00:10 px2950-01 kernel: Read(10): 28 00 00 00 00 00 00 00 08 00
Dec  9 18:00:11 px2950-01 kernel: device-mapper: multipath: Failing path 8:96.
Dec  9 18:00:11 px2950-01 kernel: device-mapper: multipath: Failing path 8:160.
Dec  9 18:00:28 px2950-01 kernel: connection6:0: detected conn error (1021)
Dec  9 18:00:38 px2950-01 kernel: session6: session recovery timed out after 10 secs
Dec  9 18:00:38 px2950-01 kernel: sd 20:0:0:0: Device offlined - not ready after error recovery
Dec  9 18:00:38 px2950-01 kernel: sd 20:0:0:0: [sdj] Unhandled error code
Dec  9 18:00:38 px2950-01 kernel: sd 20:0:0:0: [sdj] Result: hostbyte=DID_TRANSPORT_DISRUPTED driverbyte=DRIVER_OK
Dec  9 18:00:38 px2950-01 kernel: sd 20:0:0:0: [sdj] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Dec  9 18:00:38 px2950-01 kernel: device-mapper: multipath: Failing path 8:144.
Dec  9 18:00:49 px2950-01 kernel: connection4:0: detected conn error (1020)
Dec  9 18:00:50 px2950-01 kernel: connection5:0: detected conn error (1020)
Dec  9 18:00:51 px2950-01 kernel: connection7:0: detected conn error (1020)
Dec  9 18:00:52 px2950-01 kernel: connection1:0: detected conn error (1020)
Dec  9 18:00:56 px2950-01 kernel: connection3:0: detected conn error (1020)

On the server side (nas4free) I only see the reconnections from the client

Code: Select all

Dec 9 17:59:41 	nas4free 	kernel: arp: 192.168.200.30 moved from d4:ca:6d:42:2d:84 to 00:15:17:63:56:4e on em0
Dec 9 18:00:00 	nas4free 	istgt[1772]: Login from iqn.2013-01.com.company:px2950-01 (192.168.200.30) on iqn.2013-05.com.company.nas:disk1 LU2 (192.168.201.1:3260,1), ISID=23d070000, TSIH=86, CID=0, HeaderDigest=off, DataDigest=off
Dec 9 18:00:01 	nas4free 	istgt[1772]: Login from iqn.2013-01.com.company:px2950-01 (192.168.200.30) on iqn.2013-05.com.company.nas:disk0 LU1 (192.168.200.1:3260,1), ISID=23d040000, TSIH=86, CID=0, HeaderDigest=off, DataDigest=off
Dec 9 18:00:02 	nas4free 	istgt[1772]: Login from iqn.2013-01.com.company:px2950-01 (192.168.200.30) on iqn.2013-05.com.company.nas:disk1 LU2 (192.168.200.1:3260,1), ISID=23d060000, TSIH=87, CID=0, HeaderDigest=off, DataDigest=off
Dec 9 18:00:02 	nas4free 	kernel: arp: 192.168.200.30 moved from 00:15:17:63:56:4e to 00:15:17:63:56:4f on em1

The computers are connected with 2 nic for each side dedicated to the iscsi traffic, multipath is setup and seems working; the nics are on different subnets (192.168.200.xxx , 192.168.201.xxx)

This is the multipath -ll output on the client

Code: Select all

root@px2950-01:~# multipath -ll
3300000003f4b6f41 dm-3 FreeBSD,iSCSI DISK
size=3.6T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 17:0:0:0 sdd 8:48  active ready  running
  |- 15:0:0:0 sde 8:64  active ready  running
  |- 18:0:0:0 sdf 8:80  active ready  running
  `- 16:0:0:0 sdg 8:96  active ready  running
330000000e20ec5f9 dm-4 FreeBSD,iSCSI DISK
size=931G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 19:0:0:0 sdh 8:112 active ready  running
  |- 21:0:0:0 sdi 8:128 active ready  running
  |- 20:0:0:0 sdj 8:144 active ready  running
  `- 22:0:0:0 sdk 8:160 active ready  running

The proxmox client is not doing any traffic, the only test virtual machine on it is stopped.
I have tried everything I found, changed timeouts, changed cables, replaced network switch but it still happens.

Does someone have an idea on what can be wrong ?

Thanks
Daniele

[gregb]

I notcied that the iSCSI target (istgt) listens with a backlog of 2 (according to the source code). I don't know of this will limit the rate of connections, but it seems low.

Code: Select all

# netstat -anL
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen         Local Address
tcp6  0/0/1          fd0c:898b:471c:6.3260
tcp4  0/0/1          10.20.108.2.3260
tcp4  0/0/1          10.20.107.2.3260
tcp6  0/0/1          fd0c:898b:471c:6.3260
tcp6  0/0/1          fd0c:898b:471c:6.3260
tcp6  0/0/1          fd0c:898b:471c:6.3260
tcp6  0/0/1          fd0c:898b:471c:6.3260
tcp4  0/0/1          127.0.0.1.3261

[mcvillage]

I've been using proxmox with iscsi from freenas 0.7 and nas4free for years, and it worked pretty well. It could be a network config issue, or nas4free iscsi setup issue. Once it is configured it's rock solid.