RAID-Z2 pool keeps getting new permanent errors (SOLVED)

[PaganGod]

I have a RAID-Z2 pool which seems to keep getting new permanent errors. I was nervous that it was related to swapping in larger disks to repalce functional smaller ones, with my ultimate goal being to expand the pool once all smaller disks are repalced. None of the disks show read or write errors, only checksum. If I wait 1-2 days after doing extensive I/O (like migrating VMware files into the NFS share running on top of one of the ZFS datastores) and run a scrub, I get typically one file each time, always a VMDK file, with permanent errors. I don't believe the motherboard is the problem. Unfortuntely I am not using ECC memory, but my understanding is that the type of error that protects you from should be very very uncommon, and not something happening every 1-2 days.

None of the devices are reporting SMART errors. I clear the error counters on the zpool each time I repalce a drive and successfully resilver or scrub. At this time I see checksum errors on every drive in the vdev holding the RAID-Z2 pool. Any suggestions?

[b0ssman]

just to be sure run a memtest for 24 hours

[PaganGod]

Thanks, b0ssman! The problem was indeed a bad stick of memory. Live and learn, I should have run a couple full passes with MemText x86+ on this box before putting any production data on it. Since getting the bad stick out (which also caused NFSd to stop responding yesterday) it is stable, and a scrub only found fixable errors. The one virtual disk file I lost is for an easily replacable VM.