http://www.45drives.com/
Here's some background on the concept/product:
http://blog.backblaze.com/2009/09/01/pe ... d-storage/
http://blog.backblaze.com/2011/07/20/pe ... e-secrets/
And, of course, two weeks after we got the equipment, here's what Blaze posts:
http://blog.backblaze.com/2013/02/20/18 ... e-pod-3-0/
After a few delays in getting the equipment, a couple days of testing, here's the short version: they work like a charm, no problems so far.
We had Protocase up the CPU to a quad-core, instead of the standard dual-core.
We ordered and paid for 16gb of RAM instead of the standard 8gb... but the units shipped with 8gb. Protocase has apologized, and promised to have the correct RAM in the mail to us ASAP.
Version - 9.1.0.1 - Sandstorm (revision 573)
Built date - Sun Dec 16 14:58:27 JST 2012
Platform OS - FreeBSD 9.1-RELEASE (reldate 901000)
Platform - x64-embedded on Intel(R) Xeon(R) CPU X3470 @ 2.93GHz
System - Supermicro X8SIA Bios: 1.2a 06/27/2012
Rather than start with the planned 20 drives per unit (two units), my client opted to "play it safe" and only go with 10 drives per unit... additional drives to be installed as needed in the future (10 at a time).
The ZFS configuration is a 9-drive RaidZ2, with a hot spare, and an SSD L2ARC drive (note that the SSD was not installed when the "dmesg" and "zpool status" below were generated). One drive per backplane in each RaidZ2... or a maximum of (5) 9-drive vdevs. If put into a single pool that's 5 * (9-2) * 3 or 105 TB per pod using 3TB drives. Given the limited RAM, we are leaving deduplication and compression turned off.
The motherboard used has two standard USB ports directly on the board, so we put the NAS4Free USB Flash drive inside the case, rather than plugged into an external port.
For a "quick test" of the system, we configured the ZFS array, then ran this script until the array filled to capacity:
Code: Select all
#!/bin/bash
while true
do
date
dd if=/dev/zero of=/vault0/$RANDOM.$RANDOM.dat bs=1048576 count=1024
done
The first set of 10 drives had two drives fail before the array filled... replaced and retested without any problems.
Rather than use the device names when creating the ZFS pool, I labeled the drives with their serial numbers, and used those labels instead. I've seen ZFS pools have trouble when the pool drives suddenly change names... and adding drives to these pods in the future would trigger a change in names.
We disabled hyper-threading on the Xeon CPU.
Protocase ships the unit with the RAID card BIOS enabled... adds about five minutes to the boot time, and ZFS doesn't like hardware RAID controllers anyway... so we disabled that as well, leaving the RAID cards as simple SATA controllers.
Code: Select all
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-RELEASE #0 r244224M: Fri Dec 14 19:53:48 JST 2012
aoyama@nas4free.local:/usr/obj/nas4free/usr/src/sys/NAS4FREE-amd64 amd64
CPU: Intel(R) Xeon(R) CPU X3470 @ 2.93GHz (2933.40-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106e5 Family = 6 Model = 1e Stepping = 5
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x98e3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,POPCNT>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant, performance statistics
real memory = 8589934592 (8192 MB)
avail memory = 7989415936 (7619 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <062712 APIC1351>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 6
ioapic0: Changing APIC ID to 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
acpi0: <SMCI > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0x5E, should be 0x5B (20110527/tbutils-282)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 5.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 2.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <PCI-PCI bridge> irq 18 at device 0.0 on pci4
pci5: <PCI bus> on pcib5
siis0: <SiI3124 SATA controller> port 0xac00-0xac0f mem 0xfb277c00-0xfb277c7f,0xfb278000-0xfb27ffff irq 18 at device 4.0 on pci5
siisch0: <SIIS channel> at channel 0 on siis0
siisch1: <SIIS channel> at channel 1 on siis0
siisch2: <SIIS channel> at channel 2 on siis0
siisch3: <SIIS channel> at channel 3 on siis0
pcib6: <ACPI PCI-PCI bridge> at device 4.0 on pci3
pci6: <ACPI PCI bus> on pcib6
pcib7: <PCI-PCI bridge> irq 16 at device 0.0 on pci6
pci7: <PCI bus> on pcib7
siis1: <SiI3124 SATA controller> port 0xbc00-0xbc0f mem 0xfb377c00-0xfb377c7f,0xfb378000-0xfb37ffff irq 16 at device 4.0 on pci7
siisch4: <SIIS channel> at channel 0 on siis1
siisch5: <SIIS channel> at channel 1 on siis1
siisch6: <SIIS channel> at channel 2 on siis1
siisch7: <SIIS channel> at channel 3 on siis1
pci0: <base peripheral> at device 8.0 (no driver attached)
pci0: <base peripheral> at device 8.1 (no driver attached)
pci0: <base peripheral> at device 8.2 (no driver attached)
pci0: <base peripheral> at device 8.3 (no driver attached)
pci0: <base peripheral> at device 16.0 (no driver attached)
pci0: <base peripheral> at device 16.1 (no driver attached)
ehci0: <Intel PCH USB 2.0 controller USB-B> mem 0xfb1fc000-0xfb1fc3ff irq 21 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <PCI-PCI bridge> irq 16 at device 0.0 on pci8
pci9: <PCI bus> on pcib9
siis2: <SiI3124 SATA controller> port 0xcc00-0xcc0f mem 0xfb477c00-0xfb477c7f,0xfb478000-0xfb47ffff irq 16 at device 4.0 on pci9
siisch8: <SIIS channel> at channel 0 on siis2
siisch9: <SIIS channel> at channel 1 on siis2
siisch10: <SIIS channel> at channel 2 on siis2
siisch11: <SIIS channel> at channel 3 on siis2
pcib10: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci10: <ACPI PCI bus> on pcib10
em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xdc00-0xdc1f mem 0xfb5e0000-0xfb5fffff,0xfb5dc000-0xfb5dffff irq 16 at device 0.0 on pci10
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: 00:25:90:9a:d0:ae
pcib11: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci11: <ACPI PCI bus> on pcib11
em1: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0xec00-0xec1f mem 0xfb6e0000-0xfb6fffff,0xfb6dc000-0xfb6dffff irq 17 at device 0.0 on pci11
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 00:25:90:9a:d0:af
ehci1: <Intel PCH USB 2.0 controller USB-A> mem 0xfb1fa000-0xfb1fa3ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci12: <ACPI PCI bus> on pcib12
vgapci0: <VGA-compatible display> mem 0xfa000000-0xfaffffff,0xfb7fc000-0xfb7fffff,0xfb800000-0xfbffffff irq 19 at device 3.0 on pci12
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel 5 Series/3400 Series AHCI SATA controller> port 0x9400-0x9407,0x9c00-0x9c03,0x9880-0x9887,0x9800-0x9803,0x9480-0x949f mem 0xfb1f8000-0xfb1f87ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 5 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
ctl: CAM Target Layer loaded
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
ZFS filesystem version 5
ZFS storage pool version 28
Timecounters tick every 1.000 msec
(noperiph:siisch0:0:-1:-1): rescan already queued
(noperiph:siisch2:0:-1:-1): rescan already queued
(noperiph:siisch3:0:-1:-1): rescan already queued
(noperiph:siisch4:0:-1:-1): rescan already queued
(noperiph:siisch6:0:-1:-1): rescan already queued
(noperiph:siisch7:0:-1:-1): rescan already queued
(noperiph:siisch8:0:-1:-1): rescan already queued
(noperiph:siisch10:0:-1:-1): rescan already queued
(noperiph:siisch11:0:-1:-1): rescan already queued
ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to accept, logging disabled
iSCSI boot driver version 0.2.6
md0: Preloaded image </mfsroot> 230686720 bytes at 0xffffffff819a3c58
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
pmp0 at siisch0 bus 0 scbus0 target 15 lun 0
pmp0: <Port Multiplier 37261095 1706> ATA-0 device
pmp0: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp0: 5 fan-out ports
pmp1 at siisch2 bus 0 scbus2 target 15 lun 0
pmp1: <Port Multiplier 37261095 1706> ATA-0 device
pmp1: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp2 at siisch3 bus 0 scbus3 target 15 lun 0
pmp2: <Port Multiplier 37261095 1706> ATA-0 device
pmp2: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp3 at siisch4 bus 0 scbus4 target 15 lun 0
pmp3: <Port Multiplier 37261095 1706> ATA-0 device
pmp3: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp4 at siisch6 bus 0 scbus6 target 15 lun 0
pmp4: <Port Multiplier 37261095 1706> ATA-0 device
pmp4: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp5 at siisch7 bus 0 scbus7 target 15 lun 0
pmp5: <Port Multiplier 37261095 1706> ATA-0 device
pmp5: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp6 at siisch8 bus 0 scbus8 target 15 lun 0
pmp6: <Port Multiplier 37261095 1706> ATA-0 device
pmp6: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp7 at siisch10 bus 0 scbus10 target 15 lun 0
pmp7: <Port Multiplier 37261095 1706> ATA-0 device
pmp7: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp8 at siisch11 bus 0 scbus11 target 15 lun 0
pmp8: <Port Multiplier 37261095 1706> ATA-0 device
pmp8: 300.000MB/s transfers (SATA 2.x, NONE, PIO 8192bytes)
pmp1: 5 fan-out ports
pmp2: 5 fan-out ports
pmp3: 5 fan-out ports
pmp4: 5 fan-out ports
pmp5: 5 fan-out ports
pmp6: 5 fan-out ports
pmp7: 5 fan-out ports
pmp8: 5 fan-out ports
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0020, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0020, class 9/0, rev 2.00/0.00, addr 2> on usbus1
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen0.3: <Winbond Electronics Corp> at usbus0
ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
kbd2 at ukbd0
ugen1.3: <vendor 0x0461> at usbus1
ums1: <vendor 0x0461 USB Optical Mouse, class 0/0, rev 2.00/2.00, addr 3> on usbus1
ums1: 3 buttons and [XYZ] coordinates ID=0
ugen1.4: <Dell> at usbus1
ukbd1: <Dell Dell USB Entry Keyboard, class 0/0, rev 1.10/1.15, addr 4> on usbus1
kbd3 at ukbd1
ugen1.5: <FLASH> at usbus1
umass0: <FLASH Drive SKUSB20, class 0/0, rev 2.00/0.00, addr 5> on usbus1
umass0: SCSI over Bulk-Only; quirks = 0x0100
umass0:19:0:-1: Attached to scbus19
da0 at umass-sim0 bus 0 scbus19 target 0 lun 0
da0: <FLASH Drive SK_USB20 1.00> Removable Direct Access SCSI-2 device
da0: 40.000MB/s transfers
da0: 3915MB (8017920 512 byte sectors: 255H 63S/T 499C)
ada0 at siisch0 bus 0 scbus0 target 2 lun 0
ada0: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1 at siisch2 bus 0 scbus2 target 2 lun 0
ada1: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada2 at siisch3 bus 0 scbus3 target 2 lun 0
ada2: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada3 at siisch4 bus 0 scbus4 target 2 lun 0
ada3: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada4 at siisch6 bus 0 scbus6 target 0 lun 0
ada4: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada4: Previously was known as ad16
ada5 at siisch6 bus 0 scbus6 target 2 lun 0
ada5: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada6 at siisch7 bus 0 scbus7 target 2 lun 0
ada6: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada6: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada6: Command Queueing enabled
ada6: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada7 at siisch8 bus 0 scbus8 target 2 lun 0
ada7: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada7: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada7: Command Queueing enabled
ada7: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada8 at siisch10 bus 0 scbus10 target 2 lun 0
ada8: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada8: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada8: Command Queueing enabled
ada8: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada9 at siisch11 bus 0 scbus11 target 2 lun 0
ada9: <ST3000DM001-1CH166 CC43> ATA-8 SATA 3.x device
ada9: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada9: Command Queueing enabled
ada9: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 11458586 Hz quality 1000
GEOM: da0: geometry does not match label (16h,63s != 255h,63s).
GEOM: da0: media size does not match label.
Trying to mount root from ufs:/dev/md0 []...
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
Code: Select all
pool: vault1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
vault1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
label/z1f17j1a ONLINE 0 0 0
label/z1f16pwv ONLINE 0 0 0
label/z1f17j5w ONLINE 0 0 0
label/z1f13j35 ONLINE 0 0 0
label/z1f17jkt ONLINE 0 0 0
label/z1f17e96 ONLINE 0 0 0
label/z1f16ps7 ONLINE 0 0 0
label/z1f0zstr ONLINE 0 0 0
label/z1f17j2s ONLINE 0 0 0
spares
label/z1f100ke AVAIL
errors: No known data errors
