Latest News:
*New 11.2 series Release:
2019-06-20: XigmaNAS 11.2.0.4.6766 - released!

*New 12.0 series Release:
2019-06-20: XigmaNAS 12.0.0.4.6766 - released!

We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

Disk enumeration

Post/Debate your Suggestions & Requests of XigmaNAS here. This ONLY pertains to XigmaNAS.
Forum rules
Set-Up GuideFAQsForum Rules
Post Reply
User avatar
WTB VPN Jail
Starter
Starter
Posts: 54
Joined: 10 Jul 2018 01:56
Location: Westfall, Azeroth
Contact:
Status: Offline

Disk enumeration

#1

Post by WTB VPN Jail » 10 Jul 2018 05:31

Whenever I go to Disks / Management / HDD Management I am told "Configuration information about devices is different from physical devices. Please remove those devices and re-add them or run import disks with clear configuration option enabled." In fact, nothing has changed on this server for months. It is just that during boot up, disk enumeration order is non-deterministic.

Can the disk management table associate disks declared in the config.xml with the devices found at /dev/ada#, /dev/da#, etc. on the basis of properties (UUID, serial number, etc.) rather than path?

jdjon
Starter
Starter
Posts: 56
Joined: 17 Sep 2018 20:52
Status: Offline

Re: Disk enumeration

#2

Post by jdjon » 03 Oct 2018 14:55

Thumbs up!
Would make managing adding and removing disks so much easier and I guess so much more stable.

User avatar
raulfg3
Site Admin
Site Admin
Posts: 4929
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: Disk enumeration

#3

Post by raulfg3 » 03 Oct 2018 17:32

If you push the button "Clean config and import Disk" WebGUI is revert to desired status.
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

jdjon
Starter
Starter
Posts: 56
Joined: 17 Sep 2018 20:52
Status: Offline

Re: Disk enumeration

#4

Post by jdjon » 03 Oct 2018 17:38

raulfg3 wrote:
03 Oct 2018 17:32
If you push the button "Clean config and import Disk" WebGUI is revert to desired status.
And then you can manually reset everything else ... such as standby, power management etc per disk and ZFS re-import and sync. For every time a disk changes or is added to a port.

User avatar
Maurizio
Starter
Starter
Posts: 55
Joined: 05 Jul 2018 21:49
Location: Linate (MIlan)
Status: Offline

Re: Disk enumeration

#5

Post by Maurizio » 05 Oct 2018 10:03

Probably we can use the disk label eg. /dev/diskid/DISK-125B20100804
Enable the disk label setting kern.geom.label.disk_ident.enable=1 in loader.conf.
A pool can be create using this label eg:

Code: Select all

# zpool create zusb /dev/diskid/DISK-125B20100804
The zpool list command shows the label:

Code: Select all

# zpool list -v zusb
NAME                         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zusb                        3.50G   276K  3.50G        -         -     0%     0%  1.00x  ONLINE  -
  diskid/DISK-125B20100804  3.50G   276K  3.50G        -         -     0%     0%
The label doesn't change if the OS enumerates the disks in different order.
XigmaNAS 11.2.0.4 on Dell R710 144GB RAM - RootOnZFS zroot on 2x 64GB 15k HDDs in mirror, zdata on 3x 1TB SSD in RAIDZ1.
2x XigmaNAS 11.2.0.4 - RootOnZFS on HPE Proliant Microserver gen10 X3216 - 3x 4TB WD RED. In mirror with zrep.

User avatar
raulfg3
Site Admin
Site Admin
Posts: 4929
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: Disk enumeration

#6

Post by raulfg3 » 05 Oct 2018 13:09

Maurizio wrote:
05 Oct 2018 10:03
Probably we can use the disk label eg. /dev/diskid/DISK-125B20100804
Enable the disk label setting kern.geom.label.disk_ident.enable=1 in loader.conf.
A pool can be create using this label eg:

Code: Select all

# zpool create zusb /dev/diskid/DISK-125B20100804
The zpool list command shows the label:

Code: Select all

# zpool list -v zusb
NAME                         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zusb                        3.50G   276K  3.50G        -         -     0%     0%  1.00x  ONLINE  -
  diskid/DISK-125B20100804  3.50G   276K  3.50G        -         -     0%     0%
The label doesn't change if the OS enumerates the disks in different order.
change if you export your pool and then import it again.
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

User avatar
Maurizio
Starter
Starter
Posts: 55
Joined: 05 Jul 2018 21:49
Location: Linate (MIlan)
Status: Offline

Re: Disk enumeration

#7

Post by Maurizio » 05 Oct 2018 14:08

raulfg3 wrote:
05 Oct 2018 13:09
Maurizio wrote:
05 Oct 2018 10:03
Probably we can use the disk label eg. /dev/diskid/DISK-125B20100804
Enable the disk label setting kern.geom.label.disk_ident.enable=1 in loader.conf.
A pool can be create using this label eg:

Code: Select all

# zpool create zusb /dev/diskid/DISK-125B20100804
The zpool list command shows the label:

Code: Select all

# zpool list -v zusb
NAME                         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zusb                        3.50G   276K  3.50G        -         -     0%     0%  1.00x  ONLINE  -
  diskid/DISK-125B20100804  3.50G   276K  3.50G        -         -     0%     0%
The label doesn't change if the OS enumerates the disks in different order.
change if you export your pool and then import it again.
change if you have kern.geom.label.disk_ident.enable=0 and the pool is imported. But if you have kern.geom.label.disk_ident.enable=1 in loader.conf the pool is imported showing the label. The pool can be exported/imported multiple times without loosing the label.
I have tested this in the lastes Xigmanas and in FreeBSD 11.2-RELEASE-p3.
XigmaNAS 11.2.0.4 on Dell R710 144GB RAM - RootOnZFS zroot on 2x 64GB 15k HDDs in mirror, zdata on 3x 1TB SSD in RAIDZ1.
2x XigmaNAS 11.2.0.4 - RootOnZFS on HPE Proliant Microserver gen10 X3216 - 3x 4TB WD RED. In mirror with zrep.

User avatar
raulfg3
Site Admin
Site Admin
Posts: 4929
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: Disk enumeration

#8

Post by raulfg3 » 05 Oct 2018 15:04

good to know , i take note about this.


Thanks a lot.
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

jdjon
Starter
Starter
Posts: 56
Joined: 17 Sep 2018 20:52
Status: Offline

Re: Disk enumeration

#9

Post by jdjon » 06 Oct 2018 22:09

Nice, now, err, solution for us GUI-ers? ;-)

User avatar
Snufkin
Advanced User
Advanced User
Posts: 283
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: Disk enumeration

#10

Post by Snufkin » 06 Oct 2018 22:48

jdjon wrote:
06 Oct 2018 22:09
Nice, now, err, solution for us GUI-ers? ;-)
GPT label (not disk ID) is there since the time when dinosaurs were small. See field Volume label in Disks | Format.

IMHO, GPT labels are the most useful one among other disk labels in FreeBSD.

Would recommend to read Chapter Disk Labeling in Absolute FreeBSD by Michael W. Lucas (free to read in Google books).
XNAS 11.2.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

User avatar
WTB VPN Jail
Starter
Starter
Posts: 54
Joined: 10 Jul 2018 01:56
Location: Westfall, Azeroth
Contact:
Status: Offline

Re: Disk enumeration

#11

Post by WTB VPN Jail » 07 Oct 2018 01:50

For those systems using GPT disks with partitions that have labels, using something like a hash of those labels to look up the user's configuration of name, sleep time, format, SMART enablement, etc. etc. would be a possible fix, true. But remember that GPT labels identify partitions (/dev/da0p1), whereas we're talking about the system responsible for identifying disks (/dev/da0).

There would need to be some fallback, however, for those who either have GPT disks but haven't set labels, or those who are using MBR disks. Ideally a fallback that successfully pairs up the user's disk configuration with the disk discovered during boot based on persistent properties available to both systems. If the device's SMART data can be queried during the boot process, its serial number should be a unique value. Then the webUI can create a config.xml section with that serial number as a key. When loading the config.xml if a section exists for a serial number read from SMART data, then apply that configuration section to that disk, otherwise provide defaults. But providing another different disk's configuration (just because it happens to be at the same /dev/path this boot) is inconvenient. (It's understandable, but it's not what some users would like.)

jdjon
Starter
Starter
Posts: 56
Joined: 17 Sep 2018 20:52
Status: Offline

Re: Disk enumeration

#12

Post by jdjon » 07 Oct 2018 11:15

Basically as a GUI guy one looks to connect a disk with a cable in a port, regardless which port (SATA, USB; internal or external), and be able to configure it (format, raid, etc.), and assign features to it (shares), without needing to worry about plugging the disk into another port on next reboot. Xigmanas would recognize it and it would be assigned correctly, regardless of the port, or any change of ports, PCIe cards swapped, etc. But then again, I'm a simple mind, a GUI guy.

User avatar
Maurizio
Starter
Starter
Posts: 55
Joined: 05 Jul 2018 21:49
Location: Linate (MIlan)
Status: Offline

Deterministic Disk enumeration

#13

Post by Maurizio » 09 Oct 2018 22:25

Snufkin wrote:
06 Oct 2018 22:48
IMHO, GPT labels are the most useful one among other disk labels in FreeBSD.
You are right, GPT labels are the solution!

In the mean time is possible to let the FreeBSD Disk Enumeration deterministic.
Please look at this message in the FreeBSD List.
I have tested this in my FreeBSD server and now I have: ada10, ada11 and ada15 8-)

Code: Select all

# zpool status -v pool2tb
  pool: pool2tb
 state: ONLINE
  scan: scrub repaired 0 in 10h27m with 0 errors on Mon Oct  8 13:34:55 2018
config:

        NAME                 STATE     READ WRITE CKSUM
        pool2tb              ONLINE       0     0     0
          raidz1-0           ONLINE       0     0     0
            ada10            ONLINE       0     0     0
            ada11            ONLINE       0     0     0
            ada15            ONLINE       0     0     0
        cache
          gpt/SSD_L2ARC      ONLINE       0     0     0
          gpt/SSD_L2ARC_BIS  ONLINE       0     0     0

errors: No known data errors

Adding these lines to /boot/loader.conf.local

Code: Select all

hint.scbus.0.at="ahcich0"
hint.scbus.1.at="ahcich1"
hint.scbus.2.at="ahcich2"
hint.scbus.3.at="ahcich3"
hint.scbus.4.at="ahcich4"
hint.scbus.5.at="ahcich5"
hint.scbus.6.at="ahcich6"
hint.scbus.7.at="ahcich7"
hint.scbus.8.at="ahcich8"
hint.scbus.9.at="ahcich9"
hint.scbus.10.at="ahcich10"
hint.scbus.11.at="ahcich11"
hint.scbus.12.at="ahcich12"
hint.scbus.13.at="ahcich13"
hint.scbus.14.at="ahcich14"
hint.scbus.15.at="ahcich15"
hint.scbus.16.at="ahcich16"
hint.scbus.17.at="ahcich17"
hint.scbus.18.at="ahcich18"
hint.scbus.19.at="ahcich19"
hint.scbus.20.at="ahcich20"

hint.ada.0.at="scbus0"
hint.ada.1.at="scbus1"
hint.ada.2.at="scbus2"
hint.ada.3.at="scbus3"
hint.ada.4.at="scbus4"
hint.ada.5.at="scbus5"
hint.ada.6.at="scbus6"
hint.ada.7.at="scbus7"
hint.ada.8.at="scbus8"
hint.ada.9.at="scbus9"
hint.ada.10.at="scbus10"
hint.ada.11.at="scbus11"
hint.ada.12.at="scbus12"
hint.ada.13.at="scbus13"
hint.ada.14.at="scbus14"
hint.ada.15.at="scbus15"
hint.ada.16.at="scbus16"
hint.ada.17.at="scbus17"
hint.ada.18.at="scbus18"
hint.ada.19.at="scbus19"
hint.ada.20.at="scbus20"
XigmaNAS 11.2.0.4 on Dell R710 144GB RAM - RootOnZFS zroot on 2x 64GB 15k HDDs in mirror, zdata on 3x 1TB SSD in RAIDZ1.
2x XigmaNAS 11.2.0.4 - RootOnZFS on HPE Proliant Microserver gen10 X3216 - 3x 4TB WD RED. In mirror with zrep.

User avatar
WTB VPN Jail
Starter
Starter
Posts: 54
Joined: 10 Jul 2018 01:56
Location: Westfall, Azeroth
Contact:
Status: Offline

Re: Deterministic Disk enumeration

#14

Post by WTB VPN Jail » 11 Oct 2018 01:03

Maurizio wrote:
09 Oct 2018 22:25
You are right, GPT labels are the solution!
GPT labels are the solution to a different problem. They help ZFS to consistently find the devices participating in a pool.

What this thread was about was:
WTB VPN Jail wrote:
10 Jul 2018 05:31
Whenever I go to Disks / Management / HDD Management I am told "Configuration information about devices is different from physical devices."
WTB VPN Jail wrote:
07 Oct 2018 01:50
remember that GPT labels identify partitions (/dev/da0p1), whereas we're talking about the system responsible for identifying disks (/dev/da0).
For example, imagine the scenario where I share an entire disk device (/dev/da, not /dev/da0p1) over iSCSI from the NAS. If disk enumeration is non-deterministic, the block device shared over iSCSI will be non-deterministic, leading to unreliable service to the client system and / or data corruption. This is no good. GPT partition labels cannot help because they only label partitions, not disks. The fact that GPT labels can solve a different problem involving ZFS does not help here because this scenario does not involve ZFS.

So while GPT labels are certainly a solution to a certain problem, they are not a solution to this problem.
Maurizio wrote:
09 Oct 2018 22:25
In the mean time is possible to let the FreeBSD Disk Enumeration deterministic.
Please look at this message in the FreeBSD List.
Adding these lines to /boot/loader.conf.local

Code: Select all

hint.scbus.0.at="ahcich0"
hint.scbus.1.at="ahcich1"
hint.scbus.2.at="ahcich2"
hint.scbus.3.at="ahcich3"
hint.scbus.4.at="ahcich4"
hint.scbus.5.at="ahcich5"
hint.scbus.6.at="ahcich6"
hint.scbus.7.at="ahcich7"
hint.scbus.8.at="ahcich8"
hint.scbus.9.at="ahcich9"
hint.scbus.10.at="ahcich10"
hint.scbus.11.at="ahcich11"
hint.scbus.12.at="ahcich12"
hint.scbus.13.at="ahcich13"
hint.scbus.14.at="ahcich14"
hint.scbus.15.at="ahcich15"
hint.scbus.16.at="ahcich16"
hint.scbus.17.at="ahcich17"
hint.scbus.18.at="ahcich18"
hint.scbus.19.at="ahcich19"
hint.scbus.20.at="ahcich20"

hint.ada.0.at="scbus0"
hint.ada.1.at="scbus1"
hint.ada.2.at="scbus2"
hint.ada.3.at="scbus3"
hint.ada.4.at="scbus4"
hint.ada.5.at="scbus5"
hint.ada.6.at="scbus6"
hint.ada.7.at="scbus7"
hint.ada.8.at="scbus8"
hint.ada.9.at="scbus9"
hint.ada.10.at="scbus10"
hint.ada.11.at="scbus11"
hint.ada.12.at="scbus12"
hint.ada.13.at="scbus13"
hint.ada.14.at="scbus14"
hint.ada.15.at="scbus15"
hint.ada.16.at="scbus16"
hint.ada.17.at="scbus17"
hint.ada.18.at="scbus18"
hint.ada.19.at="scbus19"
hint.ada.20.at="scbus20"
Now this looks more like it! Thankyou for finding and linking that! I will try it next time I can reboot my NAS. If this works as you say, then it will fix:
  • Disks / Management / HDD Management claiming "Configuration information about devices is different from physical devices." and assigning the wrong name, spin-down, SMART enablement, etc. for drives
  • which device iSCSI shares
  • and by happy convenience, ZPools that are not based on GPT labels

User avatar
Earendil
Moderator
Moderator
Posts: 44
Joined: 23 Jun 2012 15:57
Location: near Boston, MA, USA.
Status: Offline

Re: Disk enumeration

#15

Post by Earendil » 22 Feb 2019 15:36

Using the reference https://books.google.com/books?id=C0MyD ... nt&f=false, I got this from from my /var/run/dmesg.boot file with judicious cutting&pasting:

Code: Select all

ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
ada6 at ahcich6 bus 0 scbus6 target 0 lun 0
ada7 at ahcich7 bus 0 scbus7 target 0 lun 0
ada8 at ahcich8 bus 0 scbus8 target 0 lun 0
ada9 at ahcich9 bus 0 scbus9 target 0 lun 0
da0 at umass-sim0 bus 0 scbus12 target 0 lun 0
da1 at umass-sim1 bus 1 scbus13 target 0 lun 0
For those that are curious, in the list above in order: the first 10 entries are my 10 HDDs, then the USB 3.0 thumb flash drive with embedded XigmaNAS followed by the Orico 5 bay RAID5 external enclosure through USB 3.0.

Moving on, using the instructions I created the following hints:

Code: Select all

hint.scbus.0.at="ahcich0"
hint.scbus.1.at="ahcich1"
hint.scbus.2.at="ahcich2"
hint.scbus.3.at="ahcich3"
hint.scbus.4.at="ahcich4"
hint.scbus.5.at="ahcich5"
hint.scbus.6.at="ahcich6"
hint.scbus.7.at="ahcich7"
hint.scbus.8.at="ahcich8"
hint.scbus.9.at="ahcich9"
hint.scbus.12.at="umass-sim0"
hint.scbus.13.at="umass-sim1"
hint.ada.0.at="scbus0"
hint.ada.1.at="scbus1"
hint.ada.2.at="scbus2"
hint.ada.3.at="scbus3"
hint.ada.4.at="scbus4"
hint.ada.5.at="scbus5"
hint.ada.6.at="scbus6"
hint.ada.7.at="scbus7"
hint.ada.8.at="scbus8"
hint.ada.9.at="scbus9"
hint.da.0.at="scbus12"
hint.da.1.at="scbus13"
To add a hint, say:

Code: Select all

hint.scbus.0.at="ahcich0"
through "System > Advanced > loader.conf > Add" in the XigmaNAS GUI, place

Code: Select all

hint.scbus.0.at
(no equal sign) as the "Name of the variable" and

Code: Select all

ahcich0
(no double quotes) as "The value of the variable".

I have a question though, my original source in the HTML link above says put these in the /boot/device.hints file yet the https://lists.freebsd.org/pipermail/fre ... 71851.html source previously mentioned say put them in the loader.conf file ONLY, never in device.hints file. Which is right? Since it appears to work in both places and the loader.conf will follow any changes to XigmaNAS (and by extent any changes to FreeBSD) by being in the XigmaNAS configuration file, perhaps I've already answered my own question. :)
-----------------------------------------------------------------------------------------
EDIT:
OK, I haven't tested to ensure the wiring down actually works but there was one device that actually changed even though I had that device name wired down. da1 became da2 after reboot. I tried adding some more code for both da0 and da1 to loader.conf:

Code: Select all

hint.da.0.bus="0"
hint.da.1.bus="1"
hint.da.0.target="0"
hint.da.1.target="0"
I was thinking hint.da.#.target (a variable i've seen used used elsewhere) was needed to be declared even though they are both zero. Nope. I tried to add hint.da.#.boot (which I've never seen as a loader.conf entry, in other words I made it up) since that has different values between da0 and da1. Nope.

When I disabled all da1 entries:

Code: Select all

hint.scbus.13.at="umass-sim1"
hint.da.1.at="scbus13"
hint.da.1.bus="1"
hint.da.1.target="0"
The Orico came back to da1. Huh?

Then I saw this line in the older dmesg.boot file, where I got da2; "Ambiguous scbus configuration for umass-sim1 bus 1, cannot wire down. The kernel config entry for scbus13 should specify a controller bus.". This led me thinking for the bus variable maybe I have to use scbus13 instead of da1. I therefore re-enabled all disabled entries and changed that one variable to:

Code: Select all

hint.scbus.13.bus="1"
This appeared to work.

Final list of hints added to load.conf through "System > Advanced > loader.conf > Add" in the XigmaNAS GUI: (Target variables may be superfluous)

Code: Select all

hint.scbus.0.at="ahcich0"
hint.scbus.1.at="ahcich1"
hint.scbus.2.at="ahcich2"
hint.scbus.3.at="ahcich3"
hint.scbus.4.at="ahcich4"
hint.scbus.5.at="ahcich5"
hint.scbus.6.at="ahcich6"
hint.scbus.7.at="ahcich7"
hint.scbus.8.at="ahcich8"
hint.scbus.9.at="ahcich9"
hint.scbus.12.at="umass-sim0"
hint.scbus.13.at="umass-sim1"
hint.scbus.12.bus="0"
hint.scbus.13.bus="1"
hint.ada.0.at="scbus0"
hint.ada.1.at="scbus1"
hint.ada.2.at="scbus2"
hint.ada.3.at="scbus3"
hint.ada.4.at="scbus4"
hint.ada.5.at="scbus5"
hint.ada.6.at="scbus6"
hint.ada.7.at="scbus7"
hint.ada.8.at="scbus8"
hint.ada.9.at="scbus9"
hint.da.0.at="scbus12"
hint.da.1.at="scbus13"
hint.da.0.target="0"
hint.da.1.target="0"
Earendil

XigmaNAS server:
-AMD A10-7860K APU
-Gigabyte F2A88XM-D3HP w/16GB RAM
-Green & Red HDDs
--4x 2TB
--6x 4TB
-Syba SI-PEX40064 PCI-e 1x
-External Orico USB 3.0 5 bay HDD external enclosure set at RAID 5
--5x 2TB
-650W power supply

Post Reply

Return to “Suggestions & Requests”