ZFS "Synchronize" breaks vdev after adding new drive (ada vs gpt identifiers)

[User6872]

Hello,

I'm running NAS4Free 11.0.0.4.4490 on a HP MicroServer ProLiant Gen8. I've started out with a ZFS mirror RAID consisting of 2 WD Red disks. This mirror RAID was initially created using the WebGUI of NAS4Free 11.0.0.4.3460.

• I upgraded this by putting another, same sized WD Red in a third slot
• I imported the disk in the disk management
• I formatted the disk as ZFS storage pool and chose to create ZFS on a GPT partition like I did for the first 2 drives as well
• In the ZFS pool tools I used "Attach a data device" to attach the new disk to one of the first and it started resilvering

The resilvering finished and my ZFS mirror pool successfully expanded to 3 drives:

Code: Select all

  pool: ZFSpool
 state: ONLINE
  scan: resilvered 3.92T in 9h24m with 0 errors on Thu Aug  3 03:56:57 2017
config:

    NAME                  STATE     READ WRITE CKSUM
    ZFSpool               ONLINE       0     0     0
      mirror-0            ONLINE       0     0     0
        gpt/WX11D76N****  ONLINE       0     0     0
        gpt/WX11D76D****  ONLINE       0     0     0
        gpt/WXQ1H26Y****  ONLINE       0     0     0

errors: No known data errors

(disk serial numbers redacted)

Now the problem: the NAS4Free configuration doesn't know about the third drive yet. Usually one would synchronize this within Disks > ZFS > Configuration > Synchronize. Doing so however will reproducibly break the ZFS vdev. The reason is that my stored configuration (which works) is using "ada" identifiers for the vdev (e.g. "/dev/ada0p1") instead of the "/dev/gpt/*" identifiers which is the "Detected" configuration. The synchronization will replace the ada identifiers with the gpt ones, leading to my vdev losing all disks - becoming essentially empty within "Disks > ZFS > Pools > Virtual Device".

The way I fixed it is, I took the backed up configuration from before the Synchronize operation and added in the additional third "ada" drive manually:

Code: Select all

        <vdevices>
            <vdevice>
                <uuid>********-****-****-****-************</uuid>
                <name>ZFSvdev</name>
                <type>mirror</type>
                <device>/dev/ada0p1</device>
                <device>/dev/ada1p1</device>
                <device>/dev/ada2p1</device>
                <aft4k/>
                <desc/>
            </vdevice>
        </vdevices>

(uuid redacted)

This works. If I load this configuration and reboot, everything works fine. All 3 disks are recognized within the ZFS vdev and the ZFS mirror pool works fine.

The configuraion file that is created by NAS4Free after the Synchronize operation looks as follows:

Code: Select all

        <vdevices>
            <vdevice>
                <uuid>********-****-****-****-************</uuid>
                <name>ZFSpool_mirror_0</name>
                <type>mirror</type>
                <device>/dev/gpt/WX11D76N****</device>
                <device>/dev/gpt/WX11D76D****</device>
                <device>/dev/gpt/WXQ1H26Y****</device>
                <desc/>
            </vdevice>
        </vdevices>

(uuid & disk serial numbers redacted)

This does not work! The vdev within "Disks > ZFS > Pools > Virtual Device" is displayed as empty, not containing any disks. Also the vdev name and uuid changes, however the new name is correctly reflected in the "pool" substructure of the configuration XML.

I can rinse and repeat this infinite amount of times. The result is always the same. The Synchronize operation breaks the vdev. Only manually reverting the configuration file to the "ada" devices within the vdev fixes it. Despite "Disks > ZFS > Configuration > Detected" showing the same "/dev/gpt/<serial>" entries as the "zdb" command does.

NAS4Free did initially create my 2 disk ZFS mirror RAID with the "ada" device identifiers as it seems. Now it wants to use the "/dev/gpt/<serial>" identifiers when using Synchronize, which doesn't work despite the gpt identifiers being correct (same as zdb command). I have to manually edit the configuration to use the "ada" identifiers again. I configured and formatted my new third disk the same way I did the first two.

Does anyone have a clue what's wrong here?

[User6872]

Update: It seems the issue is just the WebGUI not listing the vdev's disks after ZFS Synchronize (and the associated migration from the "ada" disk identifiers to the gpt ones).

If I use the Synchronize option within "Disks > ZFS > Configuration > Synchronize" and reboot, the "Device List" of the vdev within "Disks > ZFS > Pools > Virtual Device" turns empty and stays like that (see screenshot below). However, the ZFS pool seems to be working correctly nonetheless. Here are some outputs:

Code: Select all

nas4free: ~# zpool status -v ZFSpool
  pool: ZFSpool
 state: ONLINE
  scan: resilvered *** in *** with 0 errors on ***
config:

	NAME                  STATE     READ WRITE CKSUM
	ZFSpool               ONLINE       0     0     0
	  mirror-0            ONLINE       0     0     0
	    gpt/WX11D76N****  ONLINE       0     0     0
	    gpt/WX11D76D****  ONLINE       0     0     0
	    gpt/WXQ1H26Y****  ONLINE       0     0     0

errors: No known data errors

Code: Select all

nas4free: ~# zdb
ZFSpool:
    version: 5000
    name: 'ZFSpool'
    state: 0
    txg: 143050
    pool_guid: 843*************134
    hostname: ''
    com.delphix:has_per_vdev_zaps
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 843*************134
        children[0]:
            type: 'mirror'
            id: 0
            guid: 578*************090
            metaslab_array: 38
            metaslab_shift: 35
            ashift: 12
            asize: 6001163501568
            is_log: 0
            create_txg: 4
            com.delphix:vdev_zap_top: 35
            children[0]:
                type: 'disk'
                id: 0
                guid: 160*************183
                path: '/dev/gpt/WX11D76N****'
                whole_disk: 1
                DTL: 54
                create_txg: 4
                com.delphix:vdev_zap_leaf: 36
            children[1]:
                type: 'disk'
                id: 1
                guid: 165*************471
                path: '/dev/gpt/WX11D76D****'
                whole_disk: 1
                DTL: 53
                create_txg: 4
                com.delphix:vdev_zap_leaf: 37
            children[2]:
                type: 'disk'
                id: 2
                guid: 160*************842
                path: '/dev/gpt/WXQ1H26Y****'
                whole_disk: 1
                DTL: 195
                create_txg: 4
                com.delphix:vdev_zap_leaf: 194
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

(some info redacted)

Those look rather healthy to me.
However, the WebGUI shows the Virtual Device as empty:

empty-vdev.png

This is not in sync with the "Current" and "Detected" configuration within "Disks > ZFS > Configuration", which both list the drives correctly:

vdev-config-view.png

So essentially I have the choice between 2 working configuration files:

1. NAS4Free's auto-generated configuration from the Synchronize command - uses "/dev/gpt/*" disk identifiers for vdev - WebGUI shows vdev as empty - ZFS seems healthy according to CLI (outputs cited above)
2. my manually edited configuration (edited locally and uploaded via restore option) - uses "/dev/ada*" disk identifiers for vdev - WebGUI shows everything correctly - ZFS is healthy

I did compare both XML configuration files with a diff tool and they semantically do not differ except for the ZFS uuids, the vdev name (the Synchronize renames it for whatever reason) and the disk identifiers (ada vs gpt).

I have not yet accessed and used the ZFS dataset with the NAS4Free configuration file (option #1). I don't feel well with the WebGUI showing me an empty vdev although the ZFS CLI commands don't complain. I will stay with my manually edited config for now until I absolutely know for sure what is going on here.

Am I simply encountering a WebGUI bug of NAS4Free here?

[benoliver999]

FWIW I am experiencing exactly the same issue.

Background:

- I came over from FreeNAS
- I have 9 disks in a RAIDZ3 setup
- 1 vdev

Status:

- 'Current' and 'Detected' configuration are correct.
- Info screen is also correct:

Code: Select all

  pool: main
 state: ONLINE
  scan: scrub repaired 0 in 11h29m with 0 errors on Wed Dec  6 20:53:50 2017
config:

	NAME        STATE     READ WRITE CKSUM
	main        ONLINE       0     0     0
	  raidz3-0  ONLINE       0     0     0
	    ada6p2  ONLINE       0     0     0
	    ada7p2  ONLINE       0     0     0
	    ada3p2  ONLINE       0     0     0
	    ada5p2  ONLINE       0     0     0
	    ada0p2  ONLINE       0     0     0
	    ada8p2  ONLINE       0     0     0
	    ada2p2  ONLINE       0     0     0
	    ada4p2  ONLINE       0     0     0
	    ada1p2  ONLINE       0     0     0

errors: No known data errors

However in spite of this when I go to ZFS -> Pools -> Virtual Device -> Edit, my vdev shows up but the device list is empty.

[sleid]

did you activate the option "Import discs used in the configuration"?

[tony1]

there is two places to synchronize the gui.
Disks > ZFS > Configuration > Synchronize (check all three boxes at the bottom of the page)
also
Disks > Management > HDD Management (check Clear configuration information before importing disks.)
you shouldn't need to manually edit the config file.
try Disks > ZFS > Pools > Tools (Export a pool from the system)
then
Disks > ZFS > Pools > Tools (List or import pools) check the first three boxes
the third box will import the GPT labels if you are using them.

that should get everything to display correctly.

if your config is that far out of wack and as a last resort you could use System > Factory Defaults then import your pool and recreate your system settings, shares or other services.