I’ve been reading about ZFS ever since it hit slashdot over a year ago, and finally decided to move a bunch of data into ZFS. We bought a Sun x4100 M2 server (two dual-core Opterons with 8GB of RAM), a pair of dual-channel PCI Express Ultra320 SCSI controllers, and two Aberdeen XDAS SCSI-to-SATA shelves. Total raw disk space: 24TB. We bought all the hardware for a little over $2/GB.
I should note that the custom solution easily beat the offers from Network Appliance ($10/GB), EMC ($9.40/GB), BlueArc ($20/GB), Panasas ($5.60/GB) and Sun StorageTek ($9.40/GB).
I racked the two shelves and went to connect the SCSI cables and ran into a little problem. The SCSI cables that I bought (VHDCI) were too fat to fit side-by-side on the PCI Express cards. There is a special type of cable called a VHDCI Offset Cable that has the connector offset to one side. The off-the-shelf cache configuration on the XDAS boxes was 512MB of battery-backed RAM. I didn’t think that was going to be a problem until I found out that I couldn’t expose each of the SATA disks as an individual LUN to make full use of ZFS. I could get 16 LUN’s mapped, but the system didn’t have “slots” available to map any more LUN’s. Aberdeen tech support put me in direct contact with InforTrend’s support group, and they said I simply needed to upgrade the RAM in the shelf to 1GB.
After the 1GB upgrade, I performed a factory-reset of the disk shelf and the shelf showed a full 128 slots for LUN mappings. I booted Solaris and saw no drives. Since the default Solaris LSI driver (MPT) was a bit older, I installed the “unsupported” Solaris drivers from LSI Logic (itmpt-x86-5.07.01) and Solaris was able to see a whole bunch of drives. A very short time later, I had a 24TB ZFS pool comprised of four raidz2 pools of 12 drives each. Carving up the drives into the four raidz2 pools allows me to lose two drives per pool, or an entire SCSI chain without losing the ability to serve data. God forbid I lose an entire SCSI chain, but it’s nice to know that it can survive something that bad.
Given the parity loss of 8 drives worth of data, and 500GB per drive, I have a usable storage pool of 20TB. Awesome.
Post a Comment
You must be logged in to post a comment.