Strange behaviour when using ZFS stripe
Posted: 17 Dec 2016 16:22
I am experiencing strange issues (CAM errors) when configuring and using 2 identical HGST 6TB HDDs in ZFS Stripe.
Once I start writing to the ZFS volume - I am getting CAM errors on da4 (the 2 drives are /dev/da4 and /dev/da5).
No matter what I do - I can not get rid of this errors and soon da4 is even disconnected and my stripe is of course
corrupted. I did several tests on single drives - both for /dev/da4 and /dev/da5, but the drives are healthy.
Constant SMART monitoring also does not reveal anything - in the end the drives are factory new!
On /dev/da0, da1 and da2 I have my other ZFS Z1 Primary data array (WD Reds - also 6 TB in size) which runs
without problems. It even has a cache device on /dev/da3 (1TB Samsung EVO 840 SSD) .
Another anomaly that I am facing is the enumeration of my SATA devices. All of the above drives are connected to
Avago (LSI-2008) card which is in a Non-RAID mode, but while da0, da1 and da2 drives are indeed connected to
ports 0, 1 and 2 of the controller da3 is connected to port 5. So theoretically it should be enumerated as /dev/da5,
but because it was installed earlier in the system (before the /dev/da4 and /dev/da5) it is always enumerated as
/dev/da3 - (it's initial name), no matter to which port I connect on the LSI-2008! The two HGST drives are connected
to SAS ports 3 and 4, and are enumerated as /dev/da4 and dev/da5 as I stated above.
Can someone throw a suggestion here? What shall I do? For the time being I am using both HGST drives as separate
GPT backup drives for my data, but I would like very much to unite them in a ZFS stripe. No errors are thrown when I
use them as separate drives though! So I guess there are no issues with my LS-2008, SATA ports, SATA cables and HDDs.
It is quite a mystery to me what is wrong here
.
These are my current settings
Version 11.0.0.4 - Sayyadina (revision 3330)
Platform OS FreeBSD 11.0-RELEASE-p5 #0 r309722M: Thu Dec 8 22:52:57 CET 2016
Platform x64-full on Intel(R) Xeon(R) CPU L5640 @ 2.27GHz (Dual CPUs)
System Supermicro X8DT3
RAM 48 GB
Once I start writing to the ZFS volume - I am getting CAM errors on da4 (the 2 drives are /dev/da4 and /dev/da5).
No matter what I do - I can not get rid of this errors and soon da4 is even disconnected and my stripe is of course
corrupted. I did several tests on single drives - both for /dev/da4 and /dev/da5, but the drives are healthy.
Constant SMART monitoring also does not reveal anything - in the end the drives are factory new!
On /dev/da0, da1 and da2 I have my other ZFS Z1 Primary data array (WD Reds - also 6 TB in size) which runs
without problems. It even has a cache device on /dev/da3 (1TB Samsung EVO 840 SSD) .
Another anomaly that I am facing is the enumeration of my SATA devices. All of the above drives are connected to
Avago (LSI-2008) card which is in a Non-RAID mode, but while da0, da1 and da2 drives are indeed connected to
ports 0, 1 and 2 of the controller da3 is connected to port 5. So theoretically it should be enumerated as /dev/da5,
but because it was installed earlier in the system (before the /dev/da4 and /dev/da5) it is always enumerated as
/dev/da3 - (it's initial name), no matter to which port I connect on the LSI-2008! The two HGST drives are connected
to SAS ports 3 and 4, and are enumerated as /dev/da4 and dev/da5 as I stated above.
Can someone throw a suggestion here? What shall I do? For the time being I am using both HGST drives as separate
GPT backup drives for my data, but I would like very much to unite them in a ZFS stripe. No errors are thrown when I
use them as separate drives though! So I guess there are no issues with my LS-2008, SATA ports, SATA cables and HDDs.
It is quite a mystery to me what is wrong here
.These are my current settings
Version 11.0.0.4 - Sayyadina (revision 3330)
Platform OS FreeBSD 11.0-RELEASE-p5 #0 r309722M: Thu Dec 8 22:52:57 CET 2016
Platform x64-full on Intel(R) Xeon(R) CPU L5640 @ 2.27GHz (Dual CPUs)
System Supermicro X8DT3
RAM 48 GB