*New 11.3 series Release:
2019-10-19: XigmaNAS 11.3.0.4.7014 - released

*New 12.0 series Release:
2019-10-05: XigmaNAS 12.0.0.4.6928 - released!

*New 11.2 series Release:
2019-09-23: XigmaNAS 11.2.0.4.6881 - released!

We really need "Your" help on XigmaNAS https://translations.launchpad.net/xigmanas translations. Please help today!

Producing and hosting XigmaNAS costs money. Please consider donating for our project so that we can continue to offer you the best.
We need your support! eg: PAYPAL

RAID 5 major fail, really need help

Software RAID information and help
Forum rules
Set-Up GuideFAQsForum Rules
milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

RAID 5 major fail, really need help

#1

Post by milan187 » 16 Mar 2018 05:12

I was running Nas4Free 9.x and something major happened, I think a power outage that corrupted my USB drive (OS) and one of the hard drives in the RAID5 config. i HAVE NO BACKUP OF CONFIG OR MY DATA (big mistake I know).

I am desperate now.

I've installed latest 11.x Nas4Free release and trying to configure from scratch.

My two drives that should be good are showing unknown or unformatted when added to disk management.
Trying to import a software raid just tells me no new software raid disk found.
I am also seeing GPT rejected on bootup for the two "good" drives.

Now what can I do?
What are the chances I will be able to recover my data and rebuild the raid 5 with one new drive if 2 are ok?

I really hope someone can help.
If I can send a donation to the project and/or person helping me I would be happy to do so (if not against the rules).

I am busy father and my system has been running fine for 5 years. Really lots of precious data on there.

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#2

Post by milan187 » 16 Mar 2018 18:38

Any chance I can get any of my data?

Image

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#3

Post by Snufkin » 16 Mar 2018 19:05

Worked with geom (mirror) two years ago, trying to help.

Let's check where we are (correct me if I'm wrong):
  1. geom RAID5, three disks
  2. one of RAID5 disks corrupted
  3. boot USB corrupted
  4. no config backup
How did you get info about corruption (provide additional info, i.e. 'hang on boot' or 'log's full of errors', etc):
  1. disk in RAID5
  2. boot USB
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#4

Post by milan187 » 16 Mar 2018 19:30

Snufkin wrote:
16 Mar 2018 19:05
Worked with geom (mirror) two years ago, trying to help.

Let's check where we are (correct me if I'm wrong):
  1. geom RAID5, three disks
  2. one of RAID5 disks corrupted
  3. boot USB corrupted
  4. no config backup
How did you get info about corruption (provide additional info, i.e. 'hang on boot' or 'log's full of errors', etc):
  1. disk in RAID5
  2. boot USB

Correct with assumptions.

The boot USB would not boot up there where some error during boot.
I could not get the config file with the few methods I found.
Then I tried to do an upgrade with Live CD to the same USB, that failed.

I then did a fresh install (embedded).
So now it boot fine and I can access the webgui.
I can see the drives. I formatted the corrupted hard drive, it kept clicking. After format it sounds ok now but I don't know if its okay or not. I will get a new drive anyway.
My other 2 drives should be good but the show as unknown or unformatted.

Importing software raid does not work.
I really really hope something can be done.


Does Software RAID format wipe data? If not that could possibly help.

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#5

Post by Snufkin » 16 Mar 2018 20:55

milan187 wrote:
16 Mar 2018 19:30
The boot USB would not boot up there where some error during boot.
Those errors could give us additional info what was exactly happened.
Suspect there were errors that failed disk can not be read.
milan187 wrote:
16 Mar 2018 19:30
I could not get the config file with the few methods I found.
System USB disk may fail to boot but could be mounted and read (in some cases).
There was a chance to make new boot N4F USB, boot the machine, connect corrupted USB and read config.xml
milan187 wrote:
16 Mar 2018 19:30
Then I tried to do an upgrade with Live CD to the same USB, that failed.
Direct N4F upgrade from 9.x.x.x to 11.x.x.x is not supported.
Suggest not taking futher actions with old 'corrupted' USB.
milan187 wrote:
16 Mar 2018 19:30
I then did a fresh install (embedded).
Hope it was new USB drive, isn't it? If not I see no way to recover config.xml.
milan187 wrote:
16 Mar 2018 19:30
I formatted the corrupted hard drive, it kept clicking. After format it sounds ok now but I don't know if its okay or not. I will get a new drive anyway.
Very often drive begins to click when tries to read demaged sectors.
When you formatted it demaged sectors were marked and skipped, but new demages will appear very soon.
If possible connect the drive to freshly installed N4F and copy here its S.M.A.R.T table.
milan187 wrote:
16 Mar 2018 19:30
My other 2 drives should be good but the show as unknown or unformatted.
Importing software raid does not work.
Let's see what we've lost permanently:
  1. original config.xml
  2. one of the three drive in RAID5
What we still have:
  • all data on the rest of RAID5 drives
And this data could be recovered.

I would ask you to connect the rest two drives to freshly installed N4F and copy here their S.M.A.R.T tables.
This info gives us the idea about their health, because RAID5 recovery is very stressfull for all disks in RAID5. And one of the disks may fail during RAID rebuild.

May be it will be better not to rebuild RAID but just copy all info from degraded RAID to the safe location.
And probably not to use RAID5 in the future (for example Why RAID 5 stops working in 2009)
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#6

Post by milan187 » 16 Mar 2018 21:57

I've made a bunch of stupid moved before I got here.
Yes I did over write the original USB where I had 9.x.
It was probably possible to at least recover the config.xml but I gave up after spending a few hours.
My second USB drive was being weird so I used the original (yes a big mistake).

So config and old system is lost permanently. :(

I just hope I can somehow recover at least some of the data and then I will take your suggestion and not use RAID5

I hope these are the SMART tables you are looking for.

Code: Select all

Device /dev/ada1 - Western Digital Green
Information	
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD20EARX-00PASB0
Serial Number:    WD-WCAZAF888671
LU WWN Device Id: 5 0014ee 2b1edc7c9
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Mar 16 16:40:01 2018 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
ID	Attribute Name	Raw Value	Description
1	Raw_Read_Error_Rate	0	(Vendor specific raw value.) Stores data related to the rate of hardware read errors that occurred when reading data from a disk surface. The raw value has different structure for different vendors and is often not meaningful as a decimal number.
3	Spin_Up_Time	6175	Average time of spindle spin up (from zero RPM to fully operational).
4	Start_Stop_Count	1940	A tally of spindle start/stop cycles. The spindle turns on, and hence the count is increased, both when the hard disk is turned on after having before been turned entirely off (disconnected from power source) and when the hard disk returns from having previously been put to sleep mode.
5	Reallocated_Sector_Ct	0	Count of reallocated sectors. When the hard drive finds a read/write/verification error, it marks that sector as 'reallocated' and transfers data to a special reserved area (spare area). This process is also known as remapping, and reallocated sectors are called 'remaps'. The raw value normally represents a count of the bad sectors that have been found and remapped. Thus, the higher the attribute value, the more sectors the drive has had to reallocate. This allows a drive with bad sectors to continue operation; however, a drive which has had any reallocations at all is significantly more likely to fail in the near future. While primarily used as a metric of the life expectancy of the drive, this number also affects performance. As the count of reallocated sectors increases, the read/write speed tends to become worse because the drive head is forced to seek to the reserved area whenever a remap is accessed. If sequential access speed is critical, the remapped sectors can be manually marked as bad blocks in the file system in order to prevent their use.
7	Seek_Error_Rate	0	(Vendor specific raw value.) Rate of seek errors of the magnetic heads. If there is a partial failure in the mechanical positioning system, then seek errors will arise. Such a failure may be due to numerous factors, such as damage to a servo, or thermal widening of the hard disk. The raw value has different structure for different vendors and is often not meaningful as a decimal number.
9	Power_On_Hours	46405	Count of hours in power-on state. The raw value of this attribute shows total count of hours (or minutes, or seconds, depending on manufacturer) in power-on state.
10	Spin_Retry_Count	0	Count of retry of spin start attempts. This attribute stores a total count of the spin start attempts to reach the fully operational speed (under the condition that the first attempt was unsuccessful). An increase of this attribute value is a sign of problems in the hard disk mechanical subsystem.
11	Calibration_Retry_Count	0	This attribute indicates the count that recalibration was requested (under the condition that the first attempt was unsuccessful). An increase of this attribute value is a sign of problems in the hard disk mechanical subsystem.
12	Power_Cycle_Count	51	This attribute indicates the count of full disk power on/off cycles.
192	Power-Off_Retract_Count	40	Count of times the heads are loaded off the media. Heads can be unloaded without actually powering off.
193	Load_Cycle_Count	3814348	Count of load/unload cycles into head landing zone position.
194	Temperature_Celsius	28	Current internal temperature.
196	Reallocated_Event_Count	0	Count of remap operations. The raw value of this attribute shows the total count of attempts to transfer data from reallocated sectors to a spare area. Both successful & unsuccessful attempts are counted.
197	Current_Pending_Sector	0	Count of 'unstable' sectors (waiting to be remapped, because of unrecoverable read errors). If an unstable sector is subsequently read successfully, the sector is remapped and this value is decreased. Read errors on a sector will not remap the sector immediately (since the correct value cannot be read and so the value to remap is not known, and also it might become readable later); instead, the drive firmware remembers that the sector needs to be remapped, and will remap it the next time it's written. However some drives will not immediately remap such sectors when written; instead the drive will first attempt to write to the problem sector and if the write operation is successful then the sector will be marked good (in this case, the 'Reallocation Event Count' (0xC4) will not be increased). This is a serious shortcoming, for if such a drive contains marginal sectors that consistently fail only after some time has passed following a successful write operation, then the drive will never remap these problem sectors.
198	Offline_Uncorrectable	0	The total count of uncorrectable errors when reading/writing a sector. A rise in the value of this attribute indicates defects of the disk surface and/or problems in the mechanical subsystem.
199	UDMA_CRC_Error_Count	0	The count of errors in data transfer via the interface cable as determined by ICRC (Interface Cyclic Redundancy Check).
200	Multi_Zone_Error_Rate	0	The total count of errors when writing a sector.
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		(38880) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 375) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x3035)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   176   169   021    Pre-fail  Always       -       6175
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       1940
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   037   037   000    Old_age   Always       -       46405
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       51
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       40
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       3814348
194 Temperature_Celsius     0x0022   120   101   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     11545         -
# 2  Short offline       Completed without error       00%     11544         -
# 3  Short offline       Completed without error       00%     11543         -
# 4  Short offline       Completed without error       00%     11542         -
# 5  Short offline       Completed without error       00%     11541         -
# 6  Short offline       Completed without error       00%     11540         -
# 7  Short offline       Completed without error       00%     11539         -
# 8  Short offline       Completed without error       00%     11538         -
# 9  Short offline       Completed without error       00%     11537         -
#10  Short offline       Completed without error       00%     11536         -
#11  Short offline       Completed without error       00%     11535         -
#12  Short offline       Completed without error       00%     11534         -
#13  Short offline       Completed without error       00%     11533         -
#14  Short offline       Completed without error       00%     11532         -
#15  Short offline       Completed without error       00%     11531         -
#16  Short offline       Completed without error       00%     11530         -
#17  Short offline       Completed without error       00%     11529         -
#18  Short offline       Completed without error       00%     11528         -
#19  Short offline       Completed without error       00%     11527         -
#20  Short offline       Completed without error       00%     11526         -
#21  Short offline       Completed without error       00%     11525         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code: Select all

Device /dev/ada2 - Western Digital Green
Information	
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD20EARX-00PASB0
Serial Number:    WD-WCAZAF953292
LU WWN Device Id: 5 0014ee 2b1edd127
Firmware Version: 51.0AB51
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Mar 16 16:40:01 2018 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
ID	Attribute Name	Raw Value	Description
1	Raw_Read_Error_Rate	0	(Vendor specific raw value.) Stores data related to the rate of hardware read errors that occurred when reading data from a disk surface. The raw value has different structure for different vendors and is often not meaningful as a decimal number.
3	Spin_Up_Time	6075	Average time of spindle spin up (from zero RPM to fully operational).
4	Start_Stop_Count	1934	A tally of spindle start/stop cycles. The spindle turns on, and hence the count is increased, both when the hard disk is turned on after having before been turned entirely off (disconnected from power source) and when the hard disk returns from having previously been put to sleep mode.
5	Reallocated_Sector_Ct	0	Count of reallocated sectors. When the hard drive finds a read/write/verification error, it marks that sector as 'reallocated' and transfers data to a special reserved area (spare area). This process is also known as remapping, and reallocated sectors are called 'remaps'. The raw value normally represents a count of the bad sectors that have been found and remapped. Thus, the higher the attribute value, the more sectors the drive has had to reallocate. This allows a drive with bad sectors to continue operation; however, a drive which has had any reallocations at all is significantly more likely to fail in the near future. While primarily used as a metric of the life expectancy of the drive, this number also affects performance. As the count of reallocated sectors increases, the read/write speed tends to become worse because the drive head is forced to seek to the reserved area whenever a remap is accessed. If sequential access speed is critical, the remapped sectors can be manually marked as bad blocks in the file system in order to prevent their use.
7	Seek_Error_Rate	0	(Vendor specific raw value.) Rate of seek errors of the magnetic heads. If there is a partial failure in the mechanical positioning system, then seek errors will arise. Such a failure may be due to numerous factors, such as damage to a servo, or thermal widening of the hard disk. The raw value has different structure for different vendors and is often not meaningful as a decimal number.
9	Power_On_Hours	46348	Count of hours in power-on state. The raw value of this attribute shows total count of hours (or minutes, or seconds, depending on manufacturer) in power-on state.
10	Spin_Retry_Count	0	Count of retry of spin start attempts. This attribute stores a total count of the spin start attempts to reach the fully operational speed (under the condition that the first attempt was unsuccessful). An increase of this attribute value is a sign of problems in the hard disk mechanical subsystem.
11	Calibration_Retry_Count	0	This attribute indicates the count that recalibration was requested (under the condition that the first attempt was unsuccessful). An increase of this attribute value is a sign of problems in the hard disk mechanical subsystem.
12	Power_Cycle_Count	52	This attribute indicates the count of full disk power on/off cycles.
192	Power-Off_Retract_Count	41	Count of times the heads are loaded off the media. Heads can be unloaded without actually powering off.
193	Load_Cycle_Count	3892693	Count of load/unload cycles into head landing zone position.
194	Temperature_Celsius	29	Current internal temperature.
196	Reallocated_Event_Count	0	Count of remap operations. The raw value of this attribute shows the total count of attempts to transfer data from reallocated sectors to a spare area. Both successful & unsuccessful attempts are counted.
197	Current_Pending_Sector	0	Count of 'unstable' sectors (waiting to be remapped, because of unrecoverable read errors). If an unstable sector is subsequently read successfully, the sector is remapped and this value is decreased. Read errors on a sector will not remap the sector immediately (since the correct value cannot be read and so the value to remap is not known, and also it might become readable later); instead, the drive firmware remembers that the sector needs to be remapped, and will remap it the next time it's written. However some drives will not immediately remap such sectors when written; instead the drive will first attempt to write to the problem sector and if the write operation is successful then the sector will be marked good (in this case, the 'Reallocation Event Count' (0xC4) will not be increased). This is a serious shortcoming, for if such a drive contains marginal sectors that consistently fail only after some time has passed following a successful write operation, then the drive will never remap these problem sectors.
198	Offline_Uncorrectable	0	The total count of uncorrectable errors when reading/writing a sector. A rise in the value of this attribute indicates defects of the disk surface and/or problems in the mechanical subsystem.
199	UDMA_CRC_Error_Count	0	The count of errors in data transfer via the interface cable as determined by ICRC (Interface Cyclic Redundancy Check).
200	Multi_Zone_Error_Rate	1	The total count of errors when writing a sector.
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		(37800) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 364) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x3035)	SCT Status supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   178   171   021    Pre-fail  Always       -       6075
  4 Start_Stop_Count        0x0032   099   099   000    Old_age   Always       -       1934
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   037   037   000    Old_age   Always       -       46348
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       52
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       41
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       3892693
194 Temperature_Celsius     0x0022   120   105   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       1

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     11545         -
# 2  Short offline       Completed without error       00%     11544         -
# 3  Short offline       Completed without error       00%     11543         -
# 4  Short offline       Completed without error       00%     11542         -
# 5  Short offline       Completed without error       00%     11541         -
# 6  Short offline       Completed without error       00%     11540         -
# 7  Short offline       Completed without error       00%     11539         -
# 8  Short offline       Completed without error       00%     11538         -
# 9  Short offline       Completed without error       00%     11537         -
#10  Short offline       Completed without error       00%     11536         -
#11  Short offline       Completed without error       00%     11535         -
#12  Short offline       Completed without error       00%     11534         -
#13  Short offline       Completed without error       00%     11533         -
#14  Short offline       Completed without error       00%     11532         -
#15  Short offline       Completed without error       00%     11531         -
#16  Short offline       Completed without error       00%     11530         -
#17  Short offline       Completed without error       00%     11529         -
#18  Short offline       Completed without error       00%     11528         -
#19  Short offline       Completed without error       00%     11527         -
#20  Short offline       Completed without error       00%     11526         -
#21  Short offline       Completed without error       00%     11525         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.



Here is the SMART info for the drive that was clicking, that I formatted

Code: Select all

Device /dev/ada0 - Seagate Barracuda LP
Information	
=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda LP
Device Model:     ST32000542AS
Serial Number:    5XW1F657
LU WWN Device Id: 5 000c50 02a8fb514
Firmware Version: CC34
User Capacity:    2,000,397,852,160 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    5900 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Fri Mar 16 16:40:00 2018 UTC

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/213915en

SMART support is: Available - device has SMART capability.
SMART support is: Enabled
ID	Attribute Name	Raw Value	Description
1	Raw_Read_Error_Rate	55093216	(Vendor specific raw value.) Stores data related to the rate of hardware read errors that occurred when reading data from a disk surface. The raw value has different structure for different vendors and is often not meaningful as a decimal number.
3	Spin_Up_Time	0	Average time of spindle spin up (from zero RPM to fully operational).
4	Start_Stop_Count	8817	A tally of spindle start/stop cycles. The spindle turns on, and hence the count is increased, both when the hard disk is turned on after having before been turned entirely off (disconnected from power source) and when the hard disk returns from having previously been put to sleep mode.
5	Reallocated_Sector_Ct	0	Count of reallocated sectors. When the hard drive finds a read/write/verification error, it marks that sector as 'reallocated' and transfers data to a special reserved area (spare area). This process is also known as remapping, and reallocated sectors are called 'remaps'. The raw value normally represents a count of the bad sectors that have been found and remapped. Thus, the higher the attribute value, the more sectors the drive has had to reallocate. This allows a drive with bad sectors to continue operation; however, a drive which has had any reallocations at all is significantly more likely to fail in the near future. While primarily used as a metric of the life expectancy of the drive, this number also affects performance. As the count of reallocated sectors increases, the read/write speed tends to become worse because the drive head is forced to seek to the reserved area whenever a remap is accessed. If sequential access speed is critical, the remapped sectors can be manually marked as bad blocks in the file system in order to prevent their use.
7	Seek_Error_Rate	4453806906	(Vendor specific raw value.) Rate of seek errors of the magnetic heads. If there is a partial failure in the mechanical positioning system, then seek errors will arise. Such a failure may be due to numerous factors, such as damage to a servo, or thermal widening of the hard disk. The raw value has different structure for different vendors and is often not meaningful as a decimal number.
9	Power_On_Hours	60834	Count of hours in power-on state. The raw value of this attribute shows total count of hours (or minutes, or seconds, depending on manufacturer) in power-on state.
10	Spin_Retry_Count	0	Count of retry of spin start attempts. This attribute stores a total count of the spin start attempts to reach the fully operational speed (under the condition that the first attempt was unsuccessful). An increase of this attribute value is a sign of problems in the hard disk mechanical subsystem.
12	Power_Cycle_Count	66	This attribute indicates the count of full disk power on/off cycles.
183	Runtime_Bad_Block	0	Western Digital, Samsung or Seagate attribute: Total number of data blocks with detected, uncorrectable errors encountered during normal operation.
184	End-to-End_Error	0	This attribute is a part of Hewlett-Packard's SMART IV technology, as well as part of other vendors' IO Error Detection and Correction schemas, and it contains a count of parity errors which occur in the data path to the media via the drive's cache RAM.
187	Reported_Uncorrect	105	The count of errors that could not be recovered using hardware ECC (see attribute 195).
Consider replacing this drive
188	Command_Timeout	0	The count of aborted operations due to HDD timeout. Normally this attribute value should be equal to zero and if the value is far above zero, then most likely there will be some serious problems with power supply or an oxidized data cable.
189	High_Fly_Writes	0	HDD producers implement a Fly Height Monitor that attempts to provide additional protections for write operations by detecting when a recording head is flying outside its normal operating range. If an unsafe fly height condition is encountered, the write process is stopped, and the information is rewritten or reallocated to a safe region of the hard drive. This attribute indicates the count of these errors detected over the lifetime of the drive. This feature is implemented in most modern Seagate drives.
190	Airflow_Temperature_Cel	32	Value is equal to (100-temp. Celsius), allowing manufacturer to set a minimum threshold which corresponds to a maximum temperature.
194	Temperature_Celsius	32	Current internal temperature.
195	Hardware_ECC_Recovered	55093216	(Vendor-specific raw value.) The raw value has different structure for different vendors and is often not meaningful as a decimal number.
197	Current_Pending_Sector	0	Count of 'unstable' sectors (waiting to be remapped, because of unrecoverable read errors). If an unstable sector is subsequently read successfully, the sector is remapped and this value is decreased. Read errors on a sector will not remap the sector immediately (since the correct value cannot be read and so the value to remap is not known, and also it might become readable later); instead, the drive firmware remembers that the sector needs to be remapped, and will remap it the next time it's written. However some drives will not immediately remap such sectors when written; instead the drive will first attempt to write to the problem sector and if the write operation is successful then the sector will be marked good (in this case, the 'Reallocation Event Count' (0xC4) will not be increased). This is a serious shortcoming, for if such a drive contains marginal sectors that consistently fail only after some time has passed following a successful write operation, then the drive will never remap these problem sectors.
198	Offline_Uncorrectable	0	The total count of uncorrectable errors when reading/writing a sector. A rise in the value of this attribute indicates defects of the disk surface and/or problems in the mechanical subsystem.
199	UDMA_CRC_Error_Count	0	The count of errors in data transfer via the interface cable as determined by ICRC (Interface Cyclic Redundancy Check).
240	Head_Flying_Hours	51819	Time spent during the positioning of the drive heads.
241	Total_LBAs_Written	2753848765	Total count of LBAs written.
242	Total_LBAs_Read	1532321673	Total count of LBAs read. Some S.M.A.R.T. utilities will report a negative number for the raw value since in reality it has 48 bits rather than 32.
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection: 		(  633) seconds.
Offline data collection
capabilities: 			 (0x73) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					No Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 456) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x103f)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   113   099   006    Pre-fail  Always       -       55093216
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   092   092   020    Old_age   Always       -       8817
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   081   060   030    Pre-fail  Always       -       4453806906
  9 Power_On_Hours          0x0032   031   031   000    Old_age   Always       -       60834
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       66
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       105
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   068   052   045    Old_age   Always       -       32 (Min/Max 32/32)
194 Temperature_Celsius     0x0022   032   048   000    Old_age   Always       -       32 (0 19 0 0 0)
195 Hardware_ECC_Recovered  0x001a   051   018   000    Old_age   Always       -       55093216
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       51819 (33 213 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2753848765
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       1532321673

SMART Error Log Version: 1
ATA Error Count: 105 (device log contains only the most recent five errors)
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 105 occurred at disk power-on lifetime: 60816 hours (2534 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 10 00 00 00  Error: UNC at LBA = 0x00000010 = 16

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 10 00 00 40 00      01:00:17.170  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      01:00:17.161  READ LOG EXT
  60 00 01 10 00 00 40 00      01:00:13.489  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      01:00:13.408  READ LOG EXT
  60 00 01 10 00 00 40 00      01:00:09.755  READ FPDMA QUEUED

Error 104 occurred at disk power-on lifetime: 60816 hours (2534 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 10 00 00 00  Error: UNC at LBA = 0x00000010 = 16

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 10 00 00 40 00      01:00:13.489  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      01:00:13.408  READ LOG EXT
  60 00 01 10 00 00 40 00      01:00:09.755  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      01:00:09.674  READ LOG EXT
  60 00 01 10 00 00 40 00      01:00:06.012  READ FPDMA QUEUED

Error 103 occurred at disk power-on lifetime: 60816 hours (2534 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 10 00 00 00  Error: UNC at LBA = 0x00000010 = 16

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 10 00 00 40 00      01:00:09.755  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      01:00:09.674  READ LOG EXT
  60 00 01 10 00 00 40 00      01:00:06.012  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      01:00:05.941  READ LOG EXT
  60 00 01 10 00 00 40 00      01:00:02.333  READ FPDMA QUEUED

Error 102 occurred at disk power-on lifetime: 60816 hours (2534 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 10 00 00 00  Error: UNC at LBA = 0x00000010 = 16

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 10 00 00 40 00      01:00:06.012  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      01:00:05.941  READ LOG EXT
  60 00 01 10 00 00 40 00      01:00:02.333  READ FPDMA QUEUED
  60 00 01 02 00 00 40 00      01:00:02.333  READ FPDMA QUEUED
  c8 00 01 00 00 00 40 00      01:00:02.289  READ DMA

Error 101 occurred at disk power-on lifetime: 60816 hours (2534 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 10 00 00 00  Error: UNC at LBA = 0x00000010 = 16

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 01 10 00 00 40 00      01:00:02.333  READ FPDMA QUEUED
  60 00 01 02 00 00 40 00      01:00:02.333  READ FPDMA QUEUED
  c8 00 01 00 00 00 40 00      01:00:02.289  READ DMA
  2f 00 01 10 00 00 00 00      01:00:02.207  READ LOG EXT
  60 00 01 40 00 00 40 00      00:59:58.565  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     26390         -
# 2  Short offline       Completed without error       00%     26389         -
# 3  Short offline       Completed without error       00%     26388         -
# 4  Short offline       Completed without error       00%     26387         -
# 5  Short offline       Completed without error       00%     26386         -
# 6  Short offline       Completed without error       00%     26385         -
# 7  Short offline       Completed without error       00%     26384         -
# 8  Short offline       Completed without error       00%     26383         -
# 9  Short offline       Completed without error       00%     26382         -
#10  Short offline       Completed without error       00%     26381         -
#11  Short offline       Completed without error       00%     26380         -
#12  Short offline       Completed without error       00%     26379         -
#13  Short offline       Completed without error       00%     26378         -
#14  Short offline       Completed without error       00%     26377         -
#15  Short offline       Completed without error       00%     26376         -
#16  Short offline       Completed without error       00%     26375         -
#17  Short offline       Completed without error       00%     26374         -
#18  Short offline       Completed without error       00%     26373         -
#19  Short offline       Completed without error       00%     26372         -
#20  Short offline       Completed without error       00%     26371         -
#21  Short offline       Completed without error       00%     26370         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


Thank you again for all your help!!!

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#7

Post by Snufkin » 16 Mar 2018 22:49

Devices health:
  • ada1 good
  • ada2 average
    200 Multi_Zone_Error_Rate 1
  • ada3 bad
    7 Seek_Error_Rate 4453806906
    Error 105 occurred at disk power-on lifetime: 60816 hours - that was the error which stoped boot sequence (to investigate the cause).
More details about S.M.A.R.T.

Taking into account all above suggest starting RAID5 in degraded state (ada1, ada2 only) and copy files to the brand new disk.

Suspect Web GUI will reject RAID5 creation in degraded mode, need some time to recreate your case in virtual environment and check shell command sequence.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

User avatar
ms49434
Developer
Developer
Posts: 720
Joined: 03 Sep 2015 18:49
Location: Neuenkirchen-Vörden, Germany - GMT+1
Contact:
Status: Offline

Re: RAID 5 major fail, really need help

#8

Post by ms49434 » 16 Mar 2018 23:48

Snufkin wrote:
16 Mar 2018 22:49
Devices health:
  • ada1 good
  • ada2 average
    200 Multi_Zone_Error_Rate 1
  • ada3 bad
    7 Seek_Error_Rate 4453806906
    Error 105 occurred at disk power-on lifetime: 60816 hours - that was the error which stoped boot sequence (to investigate the cause).
More details about S.M.A.R.T.

Taking into account all above suggest starting RAID5 in degraded state (ada1, ada2 only) and copy files to the brand new disk.

Suspect Web GUI will reject RAID5 creation in degraded mode, need some time to recreate your case in virtual environment and check shell command sequence.
ada1 and ada2 are WD Green's with the LCC bug...
1) XigmaNAS 12.0.0.4 amd64-embedded on a Dell T20 running in a VM on ESXi 6.7U2, 22GB out of 32GB ECC RAM, LSI 9300-8i IT mode in passthrough mode. Pool 1: 2x HGST 10TB, mirrored, SLOG: Samsung 850 Pro, L2ARC: Samsung 850 Pro, Pool 2: 1x Samsung 860 EVO 1TB , services: Samba AD, CIFS/SMB, ftp, ctld, rsync, syncthing, zfs snapshots.
2) XigmaNAS 12.0.0.4 amd64-embedded on a Dell T20 running in a VM on ESXi 6.7U2, 8GB out of 32GB ECC RAM, IBM M1215 crossflashed, IT mode, passthrough mode, 2x HGST 10TB , services: rsync.

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#9

Post by milan187 » 17 Mar 2018 02:30

Snufkin wrote:
16 Mar 2018 22:49
Devices health:
  • ada1 good
  • ada2 average
    200 Multi_Zone_Error_Rate 1
  • ada3 bad
    7 Seek_Error_Rate 4453806906
    Error 105 occurred at disk power-on lifetime: 60816 hours - that was the error which stoped boot sequence (to investigate the cause).
More details about S.M.A.R.T.

Taking into account all above suggest starting RAID5 in degraded state (ada1, ada2 only) and copy files to the brand new disk.

Suspect Web GUI will reject RAID5 creation in degraded mode, need some time to recreate your case in virtual environment and check shell command sequence.
Yeah I can't get Web GUI to recognize any sort of RAID.
I will buy a drive(s) tomorrow to be able to copy recovered that if we can get it.

It's mostly some important documents I am after and some pictures. Some movies and other stuff I could care less about.

I really appreciate your help with this. Your help will not be unnoticed.

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#10

Post by Snufkin » 17 Mar 2018 09:25

ms49434 wrote:
16 Mar 2018 23:48
...ada1 and ada2 are WD Green's with the LCC bug...
Good point ms49434, thank you for reminding me about this bug.

Code: Select all

Device /dev/ada1 - Western Digital Green
...
9	Power_On_Hours	46405
...
193	Load_Cycle_Count	3814348

Code: Select all

Device /dev/ada2 - Western Digital Green
...
9	Power_On_Hours	46348
...
193	Load_Cycle_Count	3892693
Heads are placed into landing zone every 43 seconds!

But really bad guy is

Code: Select all

Device /dev/ada0 - Seagate Barracuda LP
...
9	Power_On_Hours	60834
...
Error 105 occurred at disk power-on lifetime: 60816 hours
By the way S.M.A.R.T. points to

Code: Select all

==> WARNING: A firmware update for this drive may be available,
see the following Seagate web pages:
http://knowledge.seagate.com/articles/en_US/FAQ/207931en
http://knowledge.seagate.com/articles/en_US/FAQ/213915en
Not sure if it's good idea to update firmware and continue using 60 000 hour old drive.



There have been made a lot of changes in N4F since my last visit to the Software RAID configuration page.
And now it's very easy to recreate degraded RAID5 without touching the shell.
The guys who keep N4F project alive are really great developers!

Here is suggested step by step procedure.
  1. Make fresh N4F boot USB using latest release (11.1.0.4.5313 embedded)
  2. Insert USB and connect two WD disks to the machine. Do not connect Seagate disk.
  3. Boot machine up and save settings (config.xml)
  4. Look at Diagnostics > Information > Disks and note new Device names for both WD disks.
    Futher steps contain virtual machine device names ada1 and ada2 as an example.
  5. Look at System log and find 4 records with 'GEOM'. You'll see something like

    Code: Select all

    kernel: GEOM: ada1: corrupt or invalid GPT detected.
    kernel: GEOM: ada1: GPT rejected -- may not be recoverable.
    kernel: GEOM: ada2: corrupt or invalid GPT detected.
    kernel: GEOM: ada2: GPT rejected -- may not be recoverable.
    It means that N4F (FreeBSD actually) recognized metadata on both disks.
    Do not be suprised about 'corrupt or invalid', it's traces of old FreeBSD bug-or-feature holywar.
  6. Go to Disks > Management > HDD Management and MANUALLY (!) add both WD disks.
    Check S.M.A.R.T. and select Preformatted file system to Software RAID.
    Apply changes.
  7. On the same page Disks > Management > HDD Management press Import button under Import Software RAID Disks section.
    Clear Configuration checkbox could be skipped this time because we've started from scratch with default config.xml.
    Message All software RAID disks are imported. will appear on the page top.
  8. Look at System log and find 5 records with 'GEOM_RAID5'. You'll see something like

    Code: Select all

    kernel: GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
    kernel: GEOM_RAID5: R53: device created (stripesize=131072).
    kernel: GEOM_RAID5: R53: ada2(1): disk attached.
    kernel: GEOM_RAID5: R53: ada1(0): disk attached.
    kernel: GEOM_RAID5: R53: activated (forced) (need about 57MiB kmem (max)).
  9. Look at Diagnostics > Information > Software RAID and see detailed RAID info in GEOM raid5 section.
    Note the line Status: Total=3, Online=2, this info was retrived from RAID metadata written on WD disks.
  10. Save settings (config.xml)
That's actually it.

Then it's better mount the RAID and copy all data to the safe location.
It could be done by network using shares (SMB, NFS, FTP) or directly to the new internal disk with rsync.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#11

Post by milan187 » 17 Mar 2018 15:33

Thanks for detailed explanation. I think I am getting somewhere but not quite there.
Step 7 to Import Software raid throws no new software raid found.


Here is the system log:

Code: Select all

Mar 17 10:01:40	nas4free	kernel: GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
Mar 17 10:04:34	nas4free	kernel: (aprobe0:ahcich1:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 17 10:04:34	nas4free	kernel: (aprobe0:ahcich1:0:15:0): CAM status: Command timeout
Mar 17 10:04:34	nas4free	kernel: (aprobe0:ahcich1:0:15:0): Error 5, Retries exhausted
Mar 17 10:04:34	nas4free	kernel: (aprobe0:ahcich2:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 17 10:04:34	nas4free	kernel: (aprobe0:ahcich2:0:15:0): CAM status: Command timeout
Mar 17 10:04:34	nas4free	kernel: (aprobe0:ahcich2:0:15:0): Error 5, Retries exhausted
Mar 17 10:12:11	nas4free	kernel: (aprobe0:ahcich1:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 17 10:12:11	nas4free	kernel: (aprobe0:ahcich1:0:15:0): CAM status: Command timeout
Mar 17 10:12:11	nas4free	kernel: (aprobe0:ahcich1:0:15:0): Error 5, Retries exhausted
Mar 17 10:12:11	nas4free	kernel: (aprobe0:ahcich2:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 17 10:12:11	nas4free	kernel: (aprobe0:ahcich2:0:15:0): CAM status: Command timeout
Mar 17 10:12:11	nas4free	kernel: (aprobe0:ahcich2:0:15:0): Error 5, Retries exhausted

If I go to GEOM I can at least see ada0 and ada1 there but can't create raid5 with 2 disks. I don't want to touch anything beyond this, but it seems I may need to recreate raid 5 manually?

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#12

Post by Snufkin » 17 Mar 2018 16:47

milan187 wrote:
17 Mar 2018 15:33
Step 7 to Import Software raid throws no new software raid found.

Here is the system log:

Code: Select all

Mar 17 10:01:40	nas4free	kernel: GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
Mar 17 10:04:34	nas4free	kernel: (aprobe0:ahcich1:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
...
Please show boot log.
It's easy to get it in Tools > Execute Command
Command is

Code: Select all

cat /var/run/dmesg.boot
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#13

Post by milan187 » 17 Mar 2018 17:29

Here is the bootlog:

Code: Select all

$ cat /var/run/dmesg.boot
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p8 #0 r330934M: Wed Mar 14 18:13:21 CET 2018
    root@dev.nas4free.org:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64 amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
CPU: AMD Athlon(tm) 7750 Dual-Core Processor (2705.82-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x100f23  Family=0x10  Model=0x2  Stepping=3
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x7ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS>
  SVM: (disabled in BIOS) NP,NAsids=64
  TSC: P-state invariant
real memory  = 4294967296 (4096 MB)
avail memory = 3417681920 (3259 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <GBT    GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.1> irqs 0-23 on motherboard
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1352909701 Hz quality 800
random: entropy device external interface
md0: Preloaded image </mfsroot> 134217728 bytes at 0xffffffff82a97b78
kbd1 at kbdmux0
netmap: loaded module
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <GBT GBTUACPI> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 450
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: Length mismatch for 3 range: 2ec00000 vs 4ed10000
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xee00-0xeeff mem 0xd0000000-0xdfffffff,0xfdfe0000-0xfdfeffff,0xfde00000-0xfdefffff irq 18 at device 5.0 on pci1
vgapci0: Boot video device
hdac0: <ATI RS780 HDA Controller> mem 0xfdffc000-0xfdffffff irq 19 at device 5.1 on pci1
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 10.0 on pci0
pci2: <ACPI PCI bus> on pcib2
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xde00-0xdeff mem 0xfdaff000-0xfdafffff,0xfdae0000-0xfdaeffff irq 18 at device 0.0 on pci2
re0: Using 1 MSI-X message
re0: Chip rev. 0x3c000000
re0: MAC rev. 0x00400000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 00:1f:d0:d7:7e:71
re0: netmap queues/slots: TX 1/256, RX 1/256
ahci0: <AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller> port 0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f mem 0xfe02f000-0xfe02f3ff irq 22 at device 17.0 on pci0
ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported
ahci0: quirks=0x22000<ATI_PMP_BUG,1MSI>
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02e000-0xfe02efff irq 16 at device 18.0 on pci0
usbus0 on ohci0
usbus0: 12Mbps Full Speed USB v1.0
ohci1: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02d000-0xfe02dfff irq 16 at device 18.1 on pci0
usbus1 on ohci1
usbus1: 12Mbps Full Speed USB v1.0
ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe02c000-0xfe02c0ff irq 17 at device 18.2 on pci0
ehci0: AMD SB600/700 quirk applied
usbus2: EHCI version 1.0
usbus2 on ehci0
usbus2: 480Mbps High Speed USB v2.0
ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02b000-0xfe02bfff irq 18 at device 19.0 on pci0
usbus3 on ohci2
usbus3: 12Mbps Full Speed USB v1.0
ohci3: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02a000-0xfe02afff irq 18 at device 19.1 on pci0
usbus4 on ohci3
usbus4: 12Mbps Full Speed USB v1.0
ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe029000-0xfe0290ff irq 19 at device 19.2 on pci0
ehci1: AMD SB600/700 quirk applied
usbus5: EHCI version 1.0
usbus5 on ehci1
usbus5: 480Mbps High Speed USB v2.0
atapci0: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfa00-0xfa0f at device 20.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
hdac1: <ATI SB600 HDA Controller> mem 0xfe024000-0xfe027fff irq 16 at device 20.2 on pci0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib3: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci3: <ACPI PCI bus> on pcib3
pci3: <serial bus, FireWire> at device 14.0 (no driver attached)
ohci4: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe028000-0xfe028fff irq 18 at device 20.5 on pci0
usbus6 on ohci4
usbus6: 12Mbps Full Speed USB v1.0
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
amdtemp0: Erratum 319: temperature measurement may be inaccurate
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
iSCSI boot driver version 0.2.13
nvme cam probe device init
hdacc0: <ATI RS690/780 HDA CODEC> at cad 0 on hdac0
hdaa0: <ATI RS690/780 Audio Function Group> at nid 1 on hdacc0
pcm0: <ATI RS690/780 (HDMI)> at nid 3 on hdaa0
hdacc1: <Realtek ALC889A HDA CODEC> at cad 0 on hdac1
hdaa1: <Realtek ALC889A Audio Function Group> at nid 1 on hdacc1
pcm1: <Realtek ALC889A (Rear Analog 7.1/2.0)> at nid 20,22,21,23 and 24,26 on hdaa1
pcm2: <Realtek ALC889A (Front Analog)> at nid 27 and 25 on hdaa1
pcm3: <Realtek ALC889A (Rear Digital)> at nid 30 and 31 on hdaa1
ugen6.1: <ATI OHCI root HUB> at usbus6
ugen5.1: <ATI EHCI root HUB> at usbus5
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen2.1: <ATI EHCI root HUB> at usbus2
ugen4.1: <ATI OHCI root HUB> at usbus4
uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen0.1: <ATI OHCI root HUB> at usbus0
ugen3.1: <ATI OHCI root HUB> at usbus3
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen1.1: <ATI OHCI root HUB> at usbus1
uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
(aprobe1:ahcich1:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe1:ahcich1:0:15:0): CAM status: Command timeout
(aprobe1:ahcich1:0:15:0): Error 5, Retries exhausted
(aprobe0:ahcich2:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
(aprobe0:ahcich2:0:15:0): CAM status: Command timeout
(aprobe0:ahcich2:0:15:0): Error 5, Retries exhausted
uhub0: 2 ports with 2 removable, self powered
uhub4: 3 ports with 3 removable, self powered
uhub3: 3 ports with 3 removable, self powered
uhub5: 3 ports with 3 removable, self powered
uhub6: 3 ports with 3 removable, self powered
uhub2: 6 ports with 6 removable, self powered
uhub1: 6 ports with 6 removable, self powered
ugen2.2: <Lexar JumpDrive> at usbus2
umass0 on uhub2
umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus2
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:6:0: Attached to scbus6
ugen0.2: <vendor 0x04f3 product 0x0103> at usbus0
ukbd0 on uhub4
ukbd0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.07, addr 2> on usbus0
kbd2 at ukbd0
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <WDC WD20EARX-00PASB0 51.0AB51> ATA8-ACS SATA 3.x device
ada0: Serial Number WD-WCAZAF888671
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors)
ada0: quirks=0x1<4K>
ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA8-ACS SATA 3.x device
da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
ada1: Serial Number WD-WCAZAF953292
da0: <Lexar JumpDrive 1100> Removable Direct Access SCSI device
ada1: 300.000MB/s transfersda0: Serial Number AAIFIJERPYNUVZNE
 (SATA 2.x, UDMA6, PIO 8192bytesda0: 40.000MB/s transfers)
ada1: Command Queueing enabled

ada1: 1907728MB (3907027055 512 byte sectors)
da0: 7648MB (15663104 512 byte sectors)
ada1: quirks=0x1<4K>
da0: quirks=0x2<NO_6_BYTE>
random: unblocking device.
Trying to mount root from ufs:/dev/md0 []...
GEOM: ada0: corrupt or invalid GPT detected.
GEOM: ada0: GPT rejected -- may not be recoverable.
GEOM: ada1: corrupt or invalid GPT detected.
GEOM: ada1: GPT rejected -- may not be recoverable.
re0: link state changed to DOWN
re0: link state changed to UP
uhid0 on uhub4
uhid0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.07, addr 2> on usbus0

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#14

Post by Snufkin » 17 Mar 2018 17:39

Please copy here info from Diagnostics > Information > Partitions page.
Include boot USB as well.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#15

Post by milan187 » 17 Mar 2018 17:50

Here we go

Code: Select all

Device /dev/ada0 - Western Digital Green
GEOM Partition Information	
No partition information found.
Fdisk Partition Information	
******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=3876021 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=3876021 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 4294967295 (2097151 Meg), flag 80 (active)
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
Device /dev/ada1 - Western Digital Green
GEOM Partition Information	
No partition information found.
Fdisk Partition Information	
******* Working on device /dev/ada1 *******
parameters extracted from in-core disklabel are:
cylinders=3876018 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=3876018 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 4294967295 (2097151 Meg), flag 80 (active)
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
Device /dev/da0 - Lexar JumpDrive 1100
GEOM Partition Information	
=>      40  15663024  da0  GPT  (7.5G)
        40      1024    1  freebsd-boot  (512K)
      1064      7128       - free -  (3.5M)
      8192   2097152    2  freebsd-ufs  (1.0G)
   2105344   8388608    3  freebsd-swap  (4.0G)
  10493952   5160960    4  freebsd-ufs  (2.5G)
  15654912      8152       - free -  (4.0M)
Fdisk Partition Information	
******* Working on device /dev/da0 *******
parameters extracted from in-core disklabel are:
cylinders=974 heads=255 sectors/track=63 (16065 blks/cyl)

parameters to be used for BIOS calculations are:
cylinders=974 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 15663103 (7647 Meg), flag 0
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#16

Post by Snufkin » 17 Mar 2018 18:18

Do not see critical errors in boot log and partitions.

Ok, let's roll back and rethink what we have:
  • Consumer grade system's worked fine non stop for 5 years
  • The story began with hangs on boot and clicking Seagate. S.M.A.R.T. reported

    Code: Select all

    1	Raw_Read_Error_Rate	55093216
    ...
    195	Hardware_ECC_Recovered	55093216
    Numbers are equal, and in the same time

    Code: Select all

    187	Reported_Uncorrect	105	The count of errors that could not be recovered using hardware ECC (see attribute 195)
    
  • Now we have low-level errors with the rest two WD, when trying to access them.
All three disks, of different vendors, at the same time have failed due to hardware reason?

Most of the time low-level errors are caused by cables, chipset and PSU.
I would bet on PSU.

Do you have spare PSU or simply another desktop to insert two disks and one USB flash?
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#17

Post by milan187 » 17 Mar 2018 18:40

The original machine suffered multiple power failures in one day. This never happened before where I live in Canada.

Anyway I put them in another machine. Is it possible raid was deleted from GEOM management? Thats where I would see it before.

Here is the bootlog:

Code: Select all

 cat /var/run/dmesg.boot
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p8 #0 r330934M: Wed Mar 14 18:13:21 CET 2018
    root@dev.nas4free.org:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64 amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
CPU: AMD A8-3850 APU with Radeon(tm) HD Graphics (2900.00-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x300f10  Family=0x12  Model=0x1  Stepping=0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  SVM: (disabled in BIOS) NP,NRIP,NAsids=64
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3430170624 (3271 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <GBT    GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.1> irqs 0-23 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 1449997653 Hz quality 800
random: entropy device external interface
md0: Preloaded image </mfsroot> 134217728 bytes at 0xffffffff82a97b78
kbd1 at kbdmux0
netmap: loaded module
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <GBT GBTUACPI> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf800-0xf8ff mem 0xd0000000-0xdfffffff,0xfdfc0000-0xfdffffff irq 18 at device 1.0 on pci0
vgapci0: Boot video device
hdac0: <ATI (0x1714) HDA Controller> mem 0xfe01c000-0xfe01ffff irq 19 at device 1.1 on pci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xee00-0xeeff mem 0xfdeff000-0xfdefffff,0xfdef8000-0xfdefbfff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 50:e5:49:54:f8:88
re0: netmap queues/slots: TX 1/256, RX 1/256
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfe028000-0xfe029fff irq 18 at device 16.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Unable to map MSI-X table 
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
xhci1: <XHCI (generic) USB 3.0 controller> mem 0xfe026000-0xfe027fff irq 17 at device 16.1 on pci0
xhci1: 32 bytes context size, 64-bit DMA
xhci1: Unable to map MSI-X table 
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
ahci0: <AMD Hudson-2 AHCI SATA controller> port 0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f mem 0xfe02f000-0xfe02f7ff irq 19 at device 17.0 on pci0
ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ohci0: <AMD FCH USB Controller> mem 0xfe02e000-0xfe02efff irq 18 at device 18.0 on pci0
usbus2 on ohci0
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <AMD FCH USB 2.0 controller> mem 0xfe02d000-0xfe02d0ff irq 17 at device 18.2 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
ohci1: <AMD FCH USB Controller> mem 0xfe02c000-0xfe02cfff irq 18 at device 19.0 on pci0
usbus4 on ohci1
usbus4: 12Mbps Full Speed USB v1.0
ehci1: <AMD FCH USB 2.0 controller> mem 0xfe02b000-0xfe02b0ff irq 17 at device 19.2 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci1
usbus5: 480Mbps High Speed USB v2.0
atapci0: <AMD Hudson-2 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfa00-0xfa0f at device 20.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
hdac1: <AMD Hudson-2 HDA Controller> mem 0xfe020000-0xfe023fff irq 16 at device 20.2 on pci0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib2: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci2: <ACPI PCI bus> on pcib2
ohci2: <OHCI (generic) USB controller> mem 0xfe02a000-0xfe02afff irq 18 at device 20.5 on pci0
usbus6 on ohci2
usbus6: 12Mbps Full Speed USB v1.0
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcefff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
iSCSI boot driver version 0.2.13
nvme cam probe device init
hdacc0: <ATI R6xx HDA CODEC> at cad 0 on hdac0
hdaa0: <ATI R6xx Audio Function Group> at nid 1 on hdacc0
pcm0: <ATI R6xx (HDMI)> at nid 3 on hdaa0
hdacc1: <Realtek ALC889 HDA CODEC> at cad 0 on hdac1
hdaa1: <Realtek ALC889 Audio Function Group> at nid 1 on hdacc1
pcm1: <Realtek ALC889 (Rear Analog)> at nid 20 and 24,26 on hdaa1
pcm2: <Realtek ALC889 (Front Analog)> at nid 27 and 25 on hdaa1
pcm3: <Realtek ALC889 (Rear Digital)> at nid 30 on hdaa1
pcm4: <Realtek ALC889 (Onboard Digital)> at nid 17 on hdaa1
ugen1.1: <0x1022 XHCI root HUB> at usbus1
ugen4.1: <AMD OHCI root HUB> at usbus4
ugen5.1: <AMD EHCI root HUB> at usbus5
ugen3.1: <AMD EHCI root HUB> at usbus3
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
uhub2: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
uhub3: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen0.1: <0x1022 XHCI root HUB> at usbus0
ugen2.1: <AMD OHCI root HUB> at usbus2
ugen6.1: <AMD OHCI root HUB> at usbus6
uhub4: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub5: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub6: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub6: 2 ports with 2 removable, self powered
uhub2: 5 ports with 5 removable, self powered
uhub5: 5 ports with 5 removable, self powered
uhub0: 4 ports with 4 removable, self powered
uhub4: 4 ports with 4 removable, self powered
ugen1.2: <vendor 0x04f3 product 0x0103> at usbus1
ukbd0 on uhub0
ukbd0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.07, addr 1> on usbus1
kbd2 at ukbd0
uhub1: 5 ports with 5 removable, self powered
uhub3: 5 ports with 5 removable, self powered
ugen3.2: <Lexar JumpDrive> at usbus3
umass0 on uhub3
umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus3
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:6:0: Attached to scbus6
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD20EARX-00PASB0 51.0AB51> ATA8-ACS SATA 3.x device
ada0: Serial Number WD-WCAZAF953292
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907728MB (3907027055 512 byte sectors)
da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
ada0: quirks=0x1<4K>
ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
da0: <Lexar JumpDrive 1100> Removable Direct Access SCSI device
ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA8-ACS SATA 3.x device
da0: Serial Number AAIFIJERPYNUVZNE
ada1: Serial Number WD-WCAZAF888671
ada1: 600.000MB/s transfersda0: 40.000MB/s transfers (SATA 3.x, UDMA6, 
da0: 7648MB (15663104 512 byte sectors)
PIO 8192bytesda0: quirks=0x2<NO_6_BYTE>
)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors)
ada1: quirks=0x1<4K>
random: unblocking device.
Trying to mount root from ufs:/dev/md0 []...
GEOM: ada0: corrupt or invalid GPT detected.
GEOM: ada0: GPT rejected -- may not be recoverable.
GEOM: ada1: corrupt or invalid GPT detected.
GEOM: ada1: GPT rejected -- may not be recoverable.
re0: link state changed to DOWN
re0: link state changed to UP
uhid0 on uhub0
uhid0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.07, addr 1> on usbus1
and partition table:

Code: Select all

Device /dev/ada0 - Western Digital Green
GEOM Partition Information	
No partition information found.
Fdisk Partition Information	
******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=3876018 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=3876018 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 4294967295 (2097151 Meg), flag 80 (active)
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
Device /dev/ada1 - Western Digital Green
GEOM Partition Information	
No partition information found.
Fdisk Partition Information	
******* Working on device /dev/ada1 *******
parameters extracted from in-core disklabel are:
cylinders=3876021 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=3876021 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 4294967295 (2097151 Meg), flag 80 (active)
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
Device /dev/da0 - Lexar JumpDrive 1100
GEOM Partition Information	
=>      40  15663024  da0  GPT  (7.5G)
        40      1024    1  freebsd-boot  (512K)
      1064      7128       - free -  (3.5M)
      8192   2097152    2  freebsd-ufs  (1.0G)
   2105344   8388608    3  freebsd-swap  (4.0G)
  10493952   5160960    4  freebsd-ufs  (2.5G)
  15654912      8152       - free -  (4.0M)
Fdisk Partition Information	
******* Working on device /dev/da0 *******
parameters extracted from in-core disklabel are:
cylinders=974 heads=255 sectors/track=63 (16065 blks/cyl)

parameters to be used for BIOS calculations are:
cylinders=974 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 15663103 (7647 Meg), flag 0
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

Looks like I am screwed?

If you want to login with a teamviewer session we can do that (if you want to poke around)?

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#18

Post by Snufkin » 17 Mar 2018 19:51

milan187 wrote:
17 Mar 2018 18:40
The original machine suffered multiple power failures in one day. This never happened before where I live in Canada.
Not sure if there's any connection .
First suspect in the list is PSU, modern capacitors do not sustain 45 000 hours at the temperature little bit higher than average. And under small load you get huge spikes of PSU voltage, that is exactly happened when you interact with HDD.
I repaired PSU in my own NAS two times (88 000 hours overall) with capacitors replacement.
milan187 wrote:
17 Mar 2018 18:40
...Is it possible raid was deleted from GEOM management? Thats where I would see it before.
We've lost original configuration file that's why no info about RAID in Web GUI.
The only source of RAID configuration is metadata on two WD disks, and we still have no full access to it.
Once we read it RAID5 will be recreated automatically.
milan187 wrote:
17 Mar 2018 18:40
Here is the bootlog:
...
and partition table:
...
All is looking fine
milan187 wrote:
17 Mar 2018 18:40
Looks like I am screwed?
Too early to give up, we are moving along fault tree and step by step eliminating failure possible causes.
I think there is good chance to recover your RAID.

Do no forget to reset configuration in System > Factory Defaults when you start the recovery on the new hardware.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#19

Post by milan187 » 17 Mar 2018 21:47

You are very optimistic, I hope you are right! :)

I did a factory reset with new HW config.
Still same, doesn't find the software raid.

Here is the boot info now.

Code: Select all

$ cat /var/run/dmesg.boot
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p8 #0 r330934M: Wed Mar 14 18:13:21 CET 2018
    root@dev.nas4free.org:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64 amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
CPU: AMD A8-3850 APU with Radeon(tm) HD Graphics (2900.00-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x300f10  Family=0x12  Model=0x1  Stepping=0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  SVM: (disabled in BIOS) NP,NRIP,NAsids=64
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3430170624 (3271 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <GBT    GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.1> irqs 0-23 on motherboard
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1449999294 Hz quality 800
random: entropy device external interface
md0: Preloaded image </mfsroot> 134217728 bytes at 0xffffffff82a97b78
kbd1 at kbdmux0
netmap: loaded module
nexus0
cryptosoft0: <software crypto> on motherboard
acpi0: <GBT GBTUACPI> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf800-0xf8ff mem 0xd0000000-0xdfffffff,0xfdfc0000-0xfdffffff irq 18 at device 1.0 on pci0
vgapci0: Boot video device
hdac0: <ATI (0x1714) HDA Controller> mem 0xfe01c000-0xfe01ffff irq 19 at device 1.1 on pci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xee00-0xeeff mem 0xfdeff000-0xfdefffff,0xfdef8000-0xfdefbfff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 50:e5:49:54:f8:88
re0: netmap queues/slots: TX 1/256, RX 1/256
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfe028000-0xfe029fff irq 18 at device 16.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Unable to map MSI-X table 
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
xhci1: <XHCI (generic) USB 3.0 controller> mem 0xfe026000-0xfe027fff irq 17 at device 16.1 on pci0
xhci1: 32 bytes context size, 64-bit DMA
xhci1: Unable to map MSI-X table 
usbus1 on xhci1
usbus1: 5.0Gbps Super Speed USB v3.0
ahci0: <AMD Hudson-2 AHCI SATA controller> port 0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f mem 0xfe02f000-0xfe02f7ff irq 19 at device 17.0 on pci0
ahci0: AHCI v1.30 with 4 6Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ohci0: <AMD FCH USB Controller> mem 0xfe02e000-0xfe02efff irq 18 at device 18.0 on pci0
usbus2 on ohci0
usbus2: 12Mbps Full Speed USB v1.0
ehci0: <AMD FCH USB 2.0 controller> mem 0xfe02d000-0xfe02d0ff irq 17 at device 18.2 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
usbus3: 480Mbps High Speed USB v2.0
ohci1: <AMD FCH USB Controller> mem 0xfe02c000-0xfe02cfff irq 18 at device 19.0 on pci0
usbus4 on ohci1
usbus4: 12Mbps Full Speed USB v1.0
ehci1: <AMD FCH USB 2.0 controller> mem 0xfe02b000-0xfe02b0ff irq 17 at device 19.2 on pci0
usbus5: EHCI version 1.0
usbus5 on ehci1
usbus5: 480Mbps High Speed USB v2.0
atapci0: <AMD Hudson-2 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfa00-0xfa0f at device 20.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
hdac1: <AMD Hudson-2 HDA Controller> mem 0xfe020000-0xfe023fff irq 16 at device 20.2 on pci0
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib2: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci2: <ACPI PCI bus> on pcib2
ohci2: <OHCI (generic) USB controller> mem 0xfe02a000-0xfe02afff irq 18 at device 20.5 on pci0
usbus6 on ohci2
usbus6: 12Mbps Full Speed USB v1.0
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcefff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
iSCSI boot driver version 0.2.13
nvme cam probe device init
hdacc0: <ATI R6xx HDA CODEC> at cad 0 on hdac0
hdaa0: <ATI R6xx Audio Function Group> at nid 1 on hdacc0
pcm0: <ATI R6xx (HDMI)> at nid 3 on hdaa0
hdacc1: <Realtek ALC889 HDA CODEC> at cad 0 on hdac1
hdaa1: <Realtek ALC889 Audio Function Group> at nid 1 on hdacc1
pcm1: <Realtek ALC889 (Rear Analog)> at nid 20 and 24,26 on hdaa1
pcm2: <Realtek ALC889 (Front Analog)> at nid 27 and 25 on hdaa1
pcm3: <Realtek ALC889 (Rear Digital)> at nid 30 on hdaa1
pcm4: <Realtek ALC889 (Onboard Digital)> at nid 17 on hdaa1
ugen0.1: <0x1022 XHCI root HUB> at usbus0
ugen2.1: <AMD OHCI root HUB> at usbus2
ugen5.1: <AMD EHCI root HUB> at usbus5
ugen4.1: <AMD OHCI root HUB> at usbus4
uhub0: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub2: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
uhub3: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen3.1: <AMD EHCI root HUB> at usbus3
ugen6.1: <AMD OHCI root HUB> at usbus6
ugen1.1: <0x1022 XHCI root HUB> at usbus1
uhub5: <0x1022 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
uhub4: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub6: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub6: 2 ports with 2 removable, self powered
uhub2: 5 ports with 5 removable, self powered
uhub3: 5 ports with 5 removable, self powered
uhub0: 4 ports with 4 removable, self powered
uhub5: 4 ports with 4 removable, self powered
uhub1: 5 ports with 5 removable, self powered
uhub4: 5 ports with 5 removable, self powered
ugen4.2: <vendor 0x04f3 product 0x0103> at usbus4
ukbd0 on uhub2
ukbd0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.07, addr 2> on usbus4
kbd2 at ukbd0
ugen3.2: <Lexar JumpDrive> at usbus3
umass0 on uhub4
umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus3
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:6:0: Attached to scbus6
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD20EARX-00PASB0 51.0AB51> ATA8-ACS SATA 3.x device
ada0: Serial Number WD-WCAZAF953292
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907728MB (3907027055 512 byte sectors)
da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
ada0: quirks=0x1<4K>
ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
da0: ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA8-ACS SATA 3.x device
ada1: Serial Number WD-WCAZAF888671
ada1: 600.000MB/s transfers<Lexar JumpDrive 1100> Removable Direct Access SCSI device
 (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors)
da0: Serial Number AAIFIJERPYNUVZNE
ada1: quirks=0x1<4K>
da0: 40.000MB/s transfers
da0: 7648MB (15663104 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
random: unblocking device.
Trying to mount root from ufs:/dev/md0 []...
GEOM: ada0: corrupt or invalid GPT detected.
GEOM: ada0: GPT rejected -- may not be recoverable.
GEOM: ada1: corrupt or invalid GPT detected.
GEOM: ada1: GPT rejected -- may not be recoverable.
re0: link state changed to DOWN
uhid0 on uhub2
uhid0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.07, addr 2> on usbus4

Code: Select all

Device /dev/ada0 - Western Digital Green
GEOM Partition Information	
No partition information found.
Fdisk Partition Information	
******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=3876018 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=3876018 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 4294967295 (2097151 Meg), flag 80 (active)
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
Device /dev/ada1 - Western Digital Green
GEOM Partition Information	
No partition information found.
Fdisk Partition Information	
******* Working on device /dev/ada1 *******
parameters extracted from in-core disklabel are:
cylinders=3876021 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=3876021 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 4294967295 (2097151 Meg), flag 80 (active)
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>
Device /dev/da0 - Lexar JumpDrive 1100
GEOM Partition Information	
=>      40  15663024  da0  GPT  (7.5G)
        40      1024    1  freebsd-boot  (512K)
      1064      7128       - free -  (3.5M)
      8192   2097152    2  freebsd-ufs  (1.0G)
   2105344   8388608    3  freebsd-swap  (4.0G)
  10493952   5160960    4  freebsd-ufs  (2.5G)
  15654912      8152       - free -  (4.0M)
Fdisk Partition Information	
******* Working on device /dev/da0 *******
parameters extracted from in-core disklabel are:
cylinders=974 heads=255 sectors/track=63 (16065 blks/cyl)

parameters to be used for BIOS calculations are:
cylinders=974 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 15663103 (7647 Meg), flag 0
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

I think this is still all the same.

Good job with the power supply. Most people would throw it out.

Anyway what would you suggest next?

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#20

Post by Snufkin » 17 Mar 2018 22:13

milan187 wrote:
17 Mar 2018 21:47
I did a factory reset with new HW config.
Still same, doesn't find the software raid.
Don't see in the System log lines

Code: Select all

nas4free	kernel: (aprobe0:ahcich2:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
nas4free	kernel: (aprobe0:ahcich2:0:15:0): CAM status: Command timeout
What messages did you get at Step 7? Step 8?
milan187 wrote:
17 Mar 2018 21:47
Anyway what would you suggest next?
I'd go to the shell to recreate RAID manually.
Need some time to refresh all the options to do it safely.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#21

Post by milan187 » 17 Mar 2018 22:28

Snufkin wrote:
17 Mar 2018 22:13

What messages did you get at Step 7? Step 8?
Still no new software RAID is found when trying to add.
In system LOG there is only this one line.

Mar 17 16:35:57 nas4free kernel: GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)

We can try the commands next when you figure out whats safe.
Thanks again!!!

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#22

Post by Snufkin » 18 Mar 2018 13:46

Some thoughts on the next steps.

Restore degraded RAID using shell.
  • I am by no means an expert or advanced user of FreeBSD and have very basic knowledge in this area.
    Working at the command prompt I have to read FreeBSD manual pages even for the simplest actions.
    And I think it's better not enter the thin ice in this particular case.
  • Guys in professional labs who offer service for data recovery never work with original disks.
    They always make bit-by-bit copy even in case of healthy disks like yours.
    And they have very deep knowledge of particular file system's low level formats.
Summing above and in case futher actions give nothing I'd recommend looking for professional data recovery service.



However, there is still another possibility to recover your data.
And ironically all hints and answers are in the neighboring thread and this pdf.

I was able to reproduce your case in virtual machine (VM).
  1. First VM contained RAID5 virtual disks (VD) in the following order

    Code: Select all

    sata1 - VD1
    sata2 - VD2
    sata3 - VD3
  2. Then VD3 has been completely removed from VM, and VD1, VD2 remained in place, connected to the same sata ports.
  3. Degraded RAID5 was successfully recovered on two disks after pressing Import button in Step 7.
  4. Second VM contained same RAID5 VDs but in reverse order with VD3 already removed.

    Code: Select all

    sata1 - VD2
    sata2 - VD1
  5. This time Import button gives nothing but No new software RAID disk found., exactly what you had.
  6. It turns out that it was necessary to press Import button second time, and degraded RAID5 was successfully recovered.
So, try to press Import button at step 7, wait for the message, then press it second time.

If it works then great, if not then it's better to turn to pro service or ask somebody experienced here.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#23

Post by milan187 » 18 Mar 2018 17:21

Selecting import multiple times doesn't have any effect. Still no luck.

Can anyone tell me if this might work?
Disks > Software RAID > GEOM > Management then go ADD
Select all drives and go RAID 5
I don't have to initialize (even though I've read that doesn't even wipe).
Can that possibly recreate my RAID5?

I will look into pro help but I am guessing thats going to be 1000s of dollars.

User avatar
raulfg3
Site Admin
Site Admin
Posts: 4925
Joined: 22 Jun 2012 22:13
Location: Madrid (ESPAÑA)
Contact:
Status: Offline

Re: RAID 5 major fail, really need help

#24

Post by raulfg3 » 18 Mar 2018 17:34

sorry , no help for my side, because no experince with RAID only with ZFS,
12.0.0.4 (revision 6766)+OBI on SUPERMICRO X8SIL-F 8GB of ECC RAM, 12x3TB disk in 3 vdev in RaidZ1 = 32TB Raw size only 22TB usable

Wiki
Last changes

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#25

Post by milan187 » 18 Mar 2018 17:37

Well I added it in GEOM and now its rebuilding, fingers crossed this works, looks like it will take a while.

If I get my data and build a new system what is recommended to do? ZFS and some sort of mirroring (RAD1)?


Here is whats happening:

Code: Select all

Log
Date & Time	User	Event
Mar 18 12:19:35	nas4free	kernel: usbus3: 12Mbps Full Speed USB v1.0
Mar 18 12:19:35	nas4free	kernel: ohci3: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02a000-0xfe02afff irq 18 at device 19.1 on pci0
Mar 18 12:19:35	nas4free	kernel: usbus4 on ohci3
Mar 18 12:19:35	nas4free	kernel: usbus4: 12Mbps Full Speed USB v1.0
Mar 18 12:19:35	nas4free	kernel: ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe029000-0xfe0290ff irq 19 at device 19.2 on pci0
Mar 18 12:19:35	nas4free	kernel: ehci1: AMD SB600/700 quirk applied
Mar 18 12:19:35	nas4free	kernel: usbus5: EHCI version 1.0
Mar 18 12:19:35	nas4free	kernel: usbus5 on ehci1
Mar 18 12:19:35	nas4free	kernel: usbus5: 480Mbps High Speed USB v2.0
Mar 18 12:19:35	nas4free	kernel: atapci0: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfa00-0xfa0f at device 20.1 on pci0
Mar 18 12:19:35	nas4free	kernel: ata0: <ATA channel> at channel 0 on atapci0
Mar 18 12:19:35	nas4free	kernel: ata1: <ATA channel> at channel 1 on atapci0
Mar 18 12:19:35	nas4free	kernel: hdac1: <ATI SB600 HDA Controller> mem 0xfe024000-0xfe027fff irq 16 at device 20.2 on pci0
Mar 18 12:19:35	nas4free	kernel: isab0: <PCI-ISA bridge> at device 20.3 on pci0
Mar 18 12:19:35	nas4free	kernel: isa0: <ISA bus> on isab0
Mar 18 12:19:35	nas4free	kernel: pcib3: <ACPI PCI-PCI bridge> at device 20.4 on pci0
Mar 18 12:19:35	nas4free	kernel: pci3: <ACPI PCI bus> on pcib3
Mar 18 12:19:35	nas4free	kernel: pci3: <serial bus, FireWire> at device 14.0 (no driver attached)
Mar 18 12:19:35	nas4free	kernel: ohci4: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe028000-0xfe028fff irq 18 at device 20.5 on pci0
Mar 18 12:19:35	nas4free	kernel: usbus6 on ohci4
Mar 18 12:19:35	nas4free	kernel: usbus6: 12Mbps Full Speed USB v1.0
Mar 18 12:19:35	nas4free	kernel: amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
Mar 18 12:19:35	nas4free	kernel: amdtemp0: Erratum 319: temperature measurement may be inaccurate
Mar 18 12:19:35	nas4free	kernel: fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
Mar 18 12:19:35	nas4free	kernel: uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
Mar 18 12:19:35	nas4free	kernel: ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
Mar 18 12:19:35	nas4free	kernel: ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
Mar 18 12:19:35	nas4free	kernel: ppbus0: <Parallel port bus> on ppc0
Mar 18 12:19:35	nas4free	kernel: lpt0: <Printer> on ppbus0
Mar 18 12:19:35	nas4free	kernel: lpt0: Interrupt-driven port
Mar 18 12:19:35	nas4free	kernel: ppi0: <Parallel I/O> on ppbus0
Mar 18 12:19:35	nas4free	kernel: sc0: <System console> at flags 0x100 on isa0
Mar 18 12:19:35	nas4free	kernel: sc0: VGA <16 virtual consoles, flags=0x300>
Mar 18 12:19:35	nas4free	kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Mar 18 12:19:35	nas4free	kernel: atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
Mar 18 12:19:35	nas4free	kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
Mar 18 12:19:35	nas4free	kernel: kbd0 at atkbd0
Mar 18 12:19:35	nas4free	kernel: atkbd0: [GIANT-LOCKED]
Mar 18 12:19:35	nas4free	kernel: hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Mar 18 12:19:35	nas4free	kernel: ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
Mar 18 12:19:35	nas4free	kernel: to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
Mar 18 12:19:35	nas4free	kernel: ZFS filesystem version: 5
Mar 18 12:19:35	nas4free	kernel: ZFS storage pool version: features support (5000)
Mar 18 12:19:35	nas4free	kernel: Timecounters tick every 1.000 msec
Mar 18 12:19:35	nas4free	kernel: iSCSI boot driver version 0.2.13
Mar 18 12:19:35	nas4free	kernel: nvme cam probe device init
Mar 18 12:19:35	nas4free	kernel: hdacc0: <ATI RS690/780 HDA CODEC> at cad 0 on hdac0
Mar 18 12:19:35	nas4free	kernel: hdaa0: <ATI RS690/780 Audio Function Group> at nid 1 on hdacc0
Mar 18 12:19:35	nas4free	kernel: pcm0: <ATI RS690/780 (HDMI)> at nid 3 on hdaa0
Mar 18 12:19:35	nas4free	kernel: hdacc1: <Realtek ALC889A HDA CODEC> at cad 0 on hdac1
Mar 18 12:19:35	nas4free	kernel: hdaa1: <Realtek ALC889A Audio Function Group> at nid 1 on hdacc1
Mar 18 12:19:35	nas4free	kernel: pcm1: <Realtek ALC889A (Rear Analog 7.1/2.0)> at nid 20,22,21,23 and 24,26 on hdaa1
Mar 18 12:19:35	nas4free	kernel: pcm2: <Realtek ALC889A (Front Analog)> at nid 27 and 25 on hdaa1
Mar 18 12:19:35	nas4free	kernel: pcm3: <Realtek ALC889A (Rear Digital)> at nid 30 and 31 on hdaa1
Mar 18 12:19:35	nas4free	kernel: ugen6.1: <ATI OHCI root HUB> at usbus6
Mar 18 12:19:35	nas4free	kernel: ugen5.1: <ATI EHCI root HUB> at usbus5
Mar 18 12:19:35	nas4free	kernel: uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
Mar 18 12:19:35	nas4free	kernel: uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
Mar 18 12:19:35	nas4free	kernel: ugen2.1: <ATI EHCI root HUB> at usbus2
Mar 18 12:19:35	nas4free	kernel: ugen4.1: <ATI OHCI root HUB> at usbus4
Mar 18 12:19:35	nas4free	kernel: uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
Mar 18 12:19:35	nas4free	kernel: uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
Mar 18 12:19:35	nas4free	kernel: ugen0.1: <ATI OHCI root HUB> at usbus0
Mar 18 12:19:35	nas4free	kernel: ugen3.1: <ATI OHCI root HUB> at usbus3
Mar 18 12:19:35	nas4free	kernel: uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Mar 18 12:19:35	nas4free	kernel: uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
Mar 18 12:19:35	nas4free	kernel: ugen1.1: <ATI OHCI root HUB> at usbus1
Mar 18 12:19:35	nas4free	kernel: uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
Mar 18 12:19:35	nas4free	kernel: (aprobe1:ahcich1:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe1:ahcich1:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe1:ahcich1:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: uhub0: 2 ports with 2 removable, self powered
Mar 18 12:19:35	nas4free	kernel: uhub4: 3 ports with 3 removable, self powered
Mar 18 12:19:35	nas4free	kernel: uhub3: 3 ports with 3 removable, self powered
Mar 18 12:19:35	nas4free	kernel: uhub5: 3 ports with 3 removable, self powered
Mar 18 12:19:35	nas4free	kernel: uhub6: 3 ports with 3 removable, self powered
Mar 18 12:19:35	nas4free	kernel: uhub2: 6 ports with 6 removable, self powered
Mar 18 12:19:35	nas4free	kernel: uhub1: 6 ports with 6 removable, self powered
Mar 18 12:19:35	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2
Mar 18 12:19:35	nas4free	kernel: umass0 on uhub2
Mar 18 12:19:35	nas4free	kernel: umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus2
Mar 18 12:19:35	nas4free	kernel: umass0: SCSI over Bulk-Only; quirks = 0x8100
Mar 18 12:19:35	nas4free	kernel: umass0:6:0: Attached to scbus6
Mar 18 12:19:35	nas4free	kernel: ugen0.2: <vendor 0x04f3 product 0x0103> at usbus0
Mar 18 12:19:35	nas4free	kernel: ukbd0 on uhub4
Mar 18 12:19:35	nas4free	kernel: ukbd0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.07, addr 2> on usbus0
Mar 18 12:19:35	nas4free	kernel: kbd2 at ukbd0
Mar 18 12:19:35	nas4free	kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Mar 18 12:19:35	nas4free	kernel: ada0: <ST32000542AS CC34> ATA8-ACS SATA 2.x device
Mar 18 12:19:35	nas4free	kernel: ada0: Serial Number 5XW1F657
Mar 18 12:19:35	nas4free	kernel: ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Mar 18 12:19:35	nas4free	kernel: ada0: Command Queueing enabled
Mar 18 12:19:35	nas4free	kernel: ada0: 1907728MB (3907027055 512 byte sectors)
Mar 18 12:19:35	nas4free	kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Mar 18 12:19:35	nas4free	kernel: ada1: <WDC WD20EARX-00PASB0 51.0AB51> ATA8-ACS SATA 3.x device
Mar 18 12:19:35	nas4free	kernel: ada1: Serial Number WD-WCAZAF953292
Mar 18 12:19:35	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:19:35	nas4free	kernel: ada1: 300.000MB/s transfersda0: (<Lexar JumpDrive 1100> Removable Direct Access SCSI device
Mar 18 12:19:35	nas4free	kernel: SATA 2.x, da0: Serial Number AAIFIJERPYNUVZNE
Mar 18 12:19:35	nas4free	kernel: UDMA6, da0: 40.000MB/s transfersPIO 8192bytes
Mar 18 12:19:35	nas4free	kernel: )da0: 7648MB (15663104 512 byte sectors)
Mar 18 12:19:35	nas4free	kernel:
Mar 18 12:19:35	nas4free	kernel: da0: quirks=0x2<NO_6_BYTE>
Mar 18 12:19:35	nas4free	kernel: ada1: Command Queueing enabled
Mar 18 12:19:35	nas4free	kernel: ada1: 1907728MB (3907027055 512 byte sectors)
Mar 18 12:19:35	nas4free	kernel: ada1: quirks=0x1<4K>
Mar 18 12:19:35	nas4free	kernel: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
Mar 18 12:19:35	nas4free	kernel: ada2: <WDC WD20EARX-00PASB0 51.0AB51> ATA8-ACS SATA 3.x device
Mar 18 12:19:35	nas4free	kernel: ada2: Serial Number WD-WCAZAF888671
Mar 18 12:19:35	nas4free	kernel: ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Mar 18 12:19:35	nas4free	kernel: ada2: Command Queueing enabled
Mar 18 12:19:35	nas4free	kernel: ada2: 1907729MB (3907029168 512 byte sectors)
Mar 18 12:19:35	nas4free	kernel: ada2: quirks=0x1<4K>
Mar 18 12:19:35	nas4free	kernel: random: unblocking device.
Mar 18 12:19:35	nas4free	kernel: Trying to mount root from ufs:/dev/md0 []...
Mar 18 12:19:35	nas4free	kernel: GEOM: ada1: corrupt or invalid GPT detected.
Mar 18 12:19:35	nas4free	kernel: GEOM: ada1: GPT rejected -- may not be recoverable.
Mar 18 12:19:35	nas4free	kernel: GEOM: ada2: corrupt or invalid GPT detected.
Mar 18 12:19:35	nas4free	kernel: GEOM: ada2: GPT rejected -- may not be recoverable.
Mar 18 12:19:35	nas4free	kernel: 3 1 s 100 % 111.9 MiB / 768.0 MiB = 0.146 57 MiB/s 0:13
Mar 18 12:19:35	nas4free	kernel: GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
Mar 18 12:19:35	nas4free	kernel: re0: link state changed to DOWN
Mar 18 12:19:35	nas4free	kernel: uhid0 on uhub4
Mar 18 12:19:35	nas4free	kernel: uhid0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.07, addr 2> on usbus0
Mar 18 12:19:35	nas4free	kernel: re0: link state changed to UP
Mar 18 12:19:36	nas4free	kernel: ada1
Mar 18 12:19:36	nas4free	kernel: ada2
Mar 18 12:19:37	nas4free	lighttpd[2139]: (server.c.1423) server started (lighttpd/1.4.49)
Mar 18 12:19:38	nas4free	login: login on ttyv0 as root
Mar 18 12:19:45	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2 (disconnected)
Mar 18 12:19:45	nas4free	kernel: umass0: at uhub2, port 1, addr 2 (disconnected)
Mar 18 12:19:45	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:19:45	nas4free	kernel: da0: <Lexar JumpDrive 1100> s/n AAIFIJERPYNUVZNE detached
Mar 18 12:19:45	nas4free	kernel: (da0:umass-sim0:0:0:0): Periph destroyed
Mar 18 12:19:45	nas4free	kernel: umass0: detached
Mar 18 12:19:47	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2
Mar 18 12:19:47	nas4free	kernel: umass0 on uhub2
Mar 18 12:19:47	nas4free	kernel: umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus2
Mar 18 12:19:47	nas4free	kernel: umass0: SCSI over Bulk-Only; quirks = 0x8100
Mar 18 12:19:47	nas4free	kernel: umass0:6:0: Attached to scbus6
Mar 18 12:19:47	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:19:47	nas4free	kernel: da0: <Lexar JumpDrive 1100> Removable Direct Access SCSI device
Mar 18 12:19:47	nas4free	kernel: da0: Serial Number AAIFIJERPYNUVZNE
Mar 18 12:19:47	nas4free	kernel: da0: 40.000MB/s transfers
Mar 18 12:19:47	nas4free	kernel: da0: 7648MB (15663104 512 byte sectors)
Mar 18 12:19:47	nas4free	kernel: da0: quirks=0x2<NO_6_BYTE>
Mar 18 12:19:55	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2 (disconnected)
Mar 18 12:19:55	nas4free	kernel: umass0: at uhub2, port 1, addr 2 (disconnected)
Mar 18 12:19:55	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:19:55	nas4free	kernel: da0: <Lexar JumpDrive 1100> s/n AAIFIJERPYNUVZNE detached
Mar 18 12:19:55	nas4free	kernel: (da0:umass-sim0:0:0:0): Periph destroyed
Mar 18 12:19:55	nas4free	kernel: umass0: detached
Mar 18 12:19:57	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2
Mar 18 12:19:57	nas4free	kernel: umass0 on uhub2
Mar 18 12:19:57	nas4free	kernel: umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus2
Mar 18 12:19:57	nas4free	kernel: umass0: SCSI over Bulk-Only; quirks = 0x8100
Mar 18 12:19:57	nas4free	kernel: umass0:6:0: Attached to scbus6
Mar 18 12:19:57	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:19:57	nas4free	kernel: da0: <Lexar JumpDrive 1100> Removable Direct Access SCSI device
Mar 18 12:19:57	nas4free	kernel: da0: Serial Number AAIFIJERPYNUVZNE
Mar 18 12:19:57	nas4free	kernel: da0: 40.000MB/s transfers
Mar 18 12:19:57	nas4free	kernel: da0: 7648MB (15663104 512 byte sectors)
Mar 18 12:19:57	nas4free	kernel: da0: quirks=0x2<NO_6_BYTE>
Mar 18 12:19:58	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2 (disconnected)
Mar 18 12:19:58	nas4free	kernel: umass0: at uhub2, port 1, addr 2 (disconnected)
Mar 18 12:19:58	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:19:58	nas4free	kernel: da0: <Lexar JumpDrive 1100> s/n AAIFIJERPYNUVZNE detached
Mar 18 12:19:58	nas4free	kernel: (da0:umass-sim0:0:0:0): Periph destroyed
Mar 18 12:19:58	nas4free	kernel: umass0: detached
Mar 18 12:20:00	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2
Mar 18 12:20:00	nas4free	kernel: umass0 on uhub2
Mar 18 12:20:00	nas4free	kernel: umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus2
Mar 18 12:20:00	nas4free	kernel: umass0: SCSI over Bulk-Only; quirks = 0x8100
Mar 18 12:20:00	nas4free	kernel: umass0:6:0: Attached to scbus6
Mar 18 12:20:00	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:20:00	nas4free	kernel: da0: <Lexar JumpDrive 1100> Removable Direct Access SCSI device
Mar 18 12:20:00	nas4free	kernel: da0: Serial Number AAIFIJERPYNUVZNE
Mar 18 12:20:00	nas4free	kernel: da0: 40.000MB/s transfers
Mar 18 12:20:00	nas4free	kernel: da0: 7648MB (15663104 512 byte sectors)
Mar 18 12:20:00	nas4free	kernel: da0: quirks=0x2<NO_6_BYTE>
Mar 18 12:20:00	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2 (disconnected)
Mar 18 12:20:00	nas4free	kernel: umass0: at uhub2, port 1, addr 2 (disconnected)
Mar 18 12:20:00	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:20:00	nas4free	kernel: da0: <Lexar JumpDrive 1100> s/n AAIFIJERPYNUVZNE detached
Mar 18 12:20:00	nas4free	kernel: (da0:umass-sim0:0:0:0): Periph destroyed
Mar 18 12:20:00	nas4free	kernel: umass0: detached
Mar 18 12:20:03	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2
Mar 18 12:20:03	nas4free	kernel: umass0 on uhub2
Mar 18 12:20:03	nas4free	kernel: umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus2
Mar 18 12:20:03	nas4free	kernel: umass0: SCSI over Bulk-Only; quirks = 0x8100
Mar 18 12:20:03	nas4free	kernel: umass0:6:0: Attached to scbus6
Mar 18 12:20:03	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:20:03	nas4free	kernel: da0: <Lexar JumpDrive 1100> Removable Direct Access SCSI device
Mar 18 12:20:03	nas4free	kernel: da0: Serial Number AAIFIJERPYNUVZNE
Mar 18 12:20:03	nas4free	kernel: da0: 40.000MB/s transfers
Mar 18 12:20:03	nas4free	kernel: da0: 7648MB (15663104 512 byte sectors)
Mar 18 12:20:03	nas4free	kernel: da0: quirks=0x2<NO_6_BYTE>
Mar 18 12:20:05	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2 (disconnected)
Mar 18 12:20:05	nas4free	kernel: umass0: at uhub2, port 1, addr 2 (disconnected)
Mar 18 12:20:05	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:20:05	nas4free	kernel: da0: <Lexar JumpDrive 1100> s/n AAIFIJERPYNUVZNE detached
Mar 18 12:20:05	nas4free	kernel: (da0:umass-sim0:0:0:0): Periph destroyed
Mar 18 12:20:05	nas4free	kernel: umass0: detached
Mar 18 12:20:07	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2
Mar 18 12:20:07	nas4free	kernel: umass0 on uhub2
Mar 18 12:20:07	nas4free	kernel: umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 2> on usbus2
Mar 18 12:20:07	nas4free	kernel: umass0: SCSI over Bulk-Only; quirks = 0x8100
Mar 18 12:20:07	nas4free	kernel: umass0:6:0: Attached to scbus6
Mar 18 12:20:07	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:20:07	nas4free	kernel: da0: <Lexar JumpDrive 1100> Removable Direct Access SCSI device
Mar 18 12:20:07	nas4free	kernel: da0: Serial Number AAIFIJERPYNUVZNE
Mar 18 12:20:07	nas4free	kernel: da0: 40.000MB/s transfers
Mar 18 12:20:07	nas4free	kernel: da0: 7648MB (15663104 512 byte sectors)
Mar 18 12:20:07	nas4free	kernel: da0: quirks=0x2<NO_6_BYTE>
Mar 18 12:20:10	nas4free	kernel: ugen2.2: <Lexar JumpDrive> at usbus2 (disconnected)
Mar 18 12:20:10	nas4free	kernel: umass0: at uhub2, port 1, addr 2 (disconnected)
Mar 18 12:20:10	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:20:10	nas4free	kernel: da0: <Lexar JumpDrive 1100> s/n AAIFIJERPYNUVZNE detached
Mar 18 12:20:10	nas4free	kernel: (da0:umass-sim0:0:0:0): Periph destroyed
Mar 18 12:20:10	nas4free	kernel: umass0: detached
Mar 18 12:20:12	nas4free	kernel: ugen0.3: <Lexar JumpDrive> at usbus0
Mar 18 12:20:12	nas4free	kernel: umass0 on uhub4
Mar 18 12:20:12	nas4free	kernel: umass0: <Lexar JumpDrive, class 0/0, rev 2.00/11.00, addr 3> on usbus0
Mar 18 12:20:12	nas4free	kernel: umass0: SCSI over Bulk-Only; quirks = 0x8100
Mar 18 12:20:12	nas4free	kernel: umass0:6:0: Attached to scbus6
Mar 18 12:20:13	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:20:13	nas4free	kernel: da0: <Lexar JumpDrive 1100> Removable Direct Access SCSI device
Mar 18 12:20:13	nas4free	kernel: da0: Serial Number AAIFIJERPYNUVZNE
Mar 18 12:20:13	nas4free	kernel: da0: 1.000MB/s transfers
Mar 18 12:20:13	nas4free	kernel: da0: 7648MB (15663104 512 byte sectors)
Mar 18 12:20:13	nas4free	kernel: da0: quirks=0x2<NO_6_BYTE>
Mar 18 12:20:30	nas4free	kernel: (da0:umass-sim0:0:0:0): got CAM status 0x44
Mar 18 12:20:30	nas4free	kernel: (da0:umass-sim0:0:0:0): fatal error, failed to attach to device
Mar 18 12:20:30	nas4free	kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 18 12:20:30	nas4free	kernel: da0:
Mar 18 12:20:30	nas4free	kernel: <Lexar JumpDrive 1100>
Mar 18 12:20:30	nas4free	kernel: s/n AAIFIJERPYNUVZNE detached
Mar 18 12:20:30	nas4free	kernel: g_access(918): provider da0 has error
Mar 18 12:20:30	nas4free	kernel: g_access(918): provider da0 has error
Mar 18 12:20:36	nas4free	kernel: (da0:umass-sim0:0:0:0): Periph destroyed
Mar 18 12:22:10	nas4free	root: geom load raid5 service executed
Mar 18 12:22:10	nas4free	root: Detected >= 512MB of RAM... No need to optimize RAM usage for GEOM raid5.
Mar 18 12:22:10	nas4free	root: geom tune raid5 service executed
Mar 18 12:22:11	nas4free	kernel: GEOM: ada1: corrupt or invalid GPT detected.
Mar 18 12:22:11	nas4free	kernel: GEOM: ada1: GPT rejected -- may not be recoverable.
Mar 18 12:22:11	nas4free	kernel: GEOM: ada2: corrupt or invalid GPT detected.
Mar 18 12:22:11	nas4free	kernel: GEOM: ada2: GPT rejected -- may not be recoverable.
Mar 18 12:22:12	nas4free	kernel: GEOM_RAID5: MCRaid: device created (stripesize=131072).
Mar 18 12:22:12	nas4free	kernel:
Mar 18 12:22:12	nas4free	kernel: GEOM_RAID5: MCRaid: ada0(0): newest disk data (HOT): -1.
Mar 18 12:22:12	nas4free	kernel: GEOM_RAID5: MCRaid: ada0(0): disk attached.
Mar 18 12:22:12	nas4free	kernel: GEOM_RAID5: MCRaid: ada1(1): newest disk data (HOT): -1.
Mar 18 12:22:12	nas4free	kernel: GEOM_RAID5: MCRaid: ada1(1): disk attached.
Mar 18 12:22:12	nas4free	kernel: GEOM_RAID5
Mar 18 12:22:12	nas4free	kernel: : MCRaid: ada2(2): newest disk data (HOT): -1.
Mar 18 12:22:12	nas4free	kernel:
Mar 18 12:22:12	nas4free	kernel: GEOM_RAID5: MCRaid: ada2(2): disk attached.
Mar 18 12:22:12	nas4free	kernel: GEOM_RAID5: MCRaid: activated (need about 57MiB kmem (max)).
Mar 18 12:22:12	nas4free	kernel: GEOM: raid5/MCRaid: corrupt or invalid GPT detected.
Mar 18 12:22:12	nas4free	kernel: GEOM: raid5/MCRaid: GPT rejected -- may not be recoverable.
Mar 18 12:25:32	nas4free	kernel: GEOM_RAID5: MCRaid: all(-1): re-sync in progress: 0.96% p:114 ETA:344min (cause: store verify progress).
Mar 18 12:28:53	nas4free	kernel: GEOM_RAID5: MCRaid: all(-1): re-sync in progress: 1.99% p:71 ETA:321min (cause: store verify progress).
Mar 18 12:32:14	nas4free	kernel: GEOM_RAID5: MCRaid: all(-1): re-sync in progress: 3.06% p:38 ETA:303min (cause: store verify progress).

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#26

Post by Snufkin » 18 Mar 2018 18:12

milan187 wrote:
18 Mar 2018 17:37
Well I added it in GEOM and now its rebuilding, fingers crossed this works, looks like it will take a while.
Great! My congratulations
milan187 wrote:
18 Mar 2018 17:37
Here is whats happening:

Code: Select all

...
Mar 18 12:19:35	nas4free	kernel: (aprobe1:ahcich1:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe1:ahcich1:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe1:ahcich1:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): Error 5, Retries exhausted
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): NOP FLUSHQUEUE. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): CAM status: Command timeout
Mar 18 12:19:35	nas4free	kernel: (aprobe2:ahcich2:0:15:0): Error 5, Retries exhausted
...
In healthy machine there should be no above records in the log.

When you finish to download files from degraded RAID check their integrity, i.e. open them in corresponding applications.
milan187 wrote:
18 Mar 2018 17:37
If I get my data and build a new system what is recommended to do?
ZFS and some sort of mirroring (RAD1)?
Using ZFS with mirror VDevs (zmirror), straight and simple.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#27

Post by milan187 » 18 Mar 2018 18:32

Once rebuilt. I need to mount and create SMB right?
I do hope my data is still there.

Also should I stop using these WD drives?

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#28

Post by Snufkin » 18 Mar 2018 19:16

milan187 wrote:
18 Mar 2018 18:32
Once rebuilt. I need to mount and create SMB right?
I do hope my data is still there.
Mount the RAID and copy all data to the safe location.
It could be done by network using shares (SMB, NFS, FTP) or directly to the new internal disk with rsync.
milan187 wrote:
18 Mar 2018 18:32
Also should I stop using these WD drives?
It depends on NAS role, number of new/modified files per time slot (minute, hour, day) and your new backup schedule (once a day, week, month). For example
  1. You have daily backup of small company file server, and files are changing 100 times per hour.
    In case of complete loss of RAID you'll miss 2 400 new/modified files and spend several hours to restore files from backup to the new RAID on brand new disks.
  2. Your NAS is dedicated on-site backup storage, and you have 2nd off-site (cloud in my case) backup. If on-site backup machine will somehow disappear, you simply order new box, connect it and check overnight backup logs next day.
If risk is acceptable why not use WDs?
As temporary solution to restore RAID5 redundancy I'd even upgrade Seagate HDD firmware, fully format it and put it in place. But first of all I'd double check machine PSU and connect good UPS.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

milan187
Starter
Starter
Posts: 19
Joined: 15 Mar 2018 13:52
Status: Offline

Re: RAID 5 major fail, really need help

#29

Post by milan187 » 18 Mar 2018 19:27

This is a home server solution so changes do not happen vert often, maybe some files written to a few time a week at most.

Will probably reuse the drives and setup ZFS.
Difference this time will be that I will backup to cloud all the important stuff.

Anyway I will be in touch once my RAID rebuild completes. It looks like the data should be there?
Should be done in next 4h or so.

User avatar
Snufkin
Advanced User
Advanced User
Posts: 289
Joined: 01 Jul 2012 11:27
Location: Etc/GMT-3 (BSD style)
Status: Offline

Re: RAID 5 major fail, really need help

#30

Post by Snufkin » 18 Mar 2018 19:44

milan187 wrote:
18 Mar 2018 19:27
Will probably reuse the drives and setup ZFS.
Before you start setting new configuration try to resolve NOP FLUSHQUEUE errors.
milan187 wrote:
18 Mar 2018 19:27
It looks like the data should be there?
Actually you can mount your MCRaid right now and look at files in Tools > File Manager.
But I'd not recommend download them immediatelly to avoid extra stress on WDs.
XNAS 11.3.0.4 embedded, ASUS P5B-E, Intel DC E6600, 4 GB DDR2, 2 x HGST HDN726040ALE614, 2 x WDC WD5000AAKS, Ippon Back Power Pro 400

Post Reply

Return to “Software RAID”