WHS - First build

I found out about Windows Home Server (WHS). As of this writing, they are just coming out with version 2 (code named Vail) of this product.

The product allows you to:

1) Backup up to 10 PC's in your home (very nice backup by the way) with bare metal restore CD capability
2) Provide RAID-1 like file level duplication of shared folders
3) Provides Media sharing capabilities to devices like the XBox 360.

While is it not 100% DLNA compatibile, it is possible to add in components which will make it DLNA compatibile. Also, there are other add-ins that let you do things like Torrents etc.

The RAID-1 like functionality is called Drive Extender - very cool in that you can just use any drive lying around, plug it into the WHS and make it part of the storage pool. The Drive Extender will automatically make a duplicate copy of the file onto another drive in the pool. Microsoft does not support the use of RAID controllers at all with WHS. However, you can easily build a system with them.

When building a WHS system and considering using RAID instead of the builtin Drive Extender you have to think about a few things:

- Drive extender is automatic and works across different sized drives
- Drive extender means that you end up with 50% of the space
- Mixing hardware RAID and Drive Extender will be a big space waster

There is no way to say that a specific shared folder should reside on a certain volume - thus if you make a RAID-5 and combine it with other drives, you will still end up losing 50% of the space on top of the single drive you lost with the RAID-5.

You should either go all with Hardware RAID or all with Drive Extender. The exception might be the boot drive...I would seriously consider using RAID-1 for the boot drive.

Anyway...I took an old machine, put 5 320 GB drives I had lying around, added 3 2 TB drives I bought from Newegg along with a Highpoint JBOD controller and a couple of 1 TB USB hard drives. I now have a system with 8 TB of space. After about a month, I have 1 TB free, 3.6 TB of used space and 3 TB of space used for Duplication.

For my next buiild, I want to get the Norco 4220 case and use a real Hardware RAID controller like the Highpoint RR3540 controller - 16 drives.

Since the biggest cost in the whole thing is hard drives, I want to maximize my cost per unit of space.

Here is the math (I am only listing the items that would be different in the build):

If I do 20 drives using the WHS Drive Extender, I need 2 Supermicro AOC-SASLP-MV8 cards and then the rest of the drives are connected via the MB.

Cost - 2 * $110 = $220
Cables - $100
Total = $320

If I do 20 drives using the Highpoint card above and then one other 4 port card:

Highpoint RR3540 = $685
Highpoint 2640 = $120
Total: $805

Now....the cost of the drives is the same - 20 2 TB drives @ $140 each = $2800

So...total RAW space is 40 TB.

Using Drive Extender, I essentially end up with 20 TB of usable space.
Using RAID 6 on the RR3540 and RAID 5 on the 2640 I end up with 17 drives of usable space for a total of 34 TB. Or I could use RAID 1 on the 2640 for more redundancy. That still gives me 16 drives of space for a total of 32 TB.

Now...cost per TB

With Drive Extender = ($2800 + $320)/20TB = $156 per TB

With Harware RAID = ($2800 + $805)/32TB = $112.66 per TB
or with RAID5 instead = ($2800 + $805)/34TB = $106.03 per TB

Yes, I know I didn't include things like the case, power supply, motherboard, RAM, etc...but those things would remain the same for either solution.

So, while you might spend more on the equipment, you actually spend less per unit of storage with the hardware RAID. I also think that it is quite reliable - the drives have 3 year warranty, are hot-swappable with the Norco case and Highpoint solution, and with RAID 6 you can lose 2 drives and still have a working system. I think your odds of losing 3 simultaneously are very low. Even if you decided to leave one hot spare, you would have a total of 15 drives of space which is 30 TB and comes out to a cost of about $120 per TB which is still less than the Drive Extender method. You do lose the ability to just mix and match your drives and it doesn't make sense to try and use external drives (USB) and mix Drive Extender duplication with the hardware RAID. However, with a purpose built case and solution, I am prepared for sacrifice some of the flexibiilty for space and cost.

Also...to those that say you lose the ability to dynamically expand the storage pool, I disagree. The Highpoint card has the ability to do Online Capacity Expansion.

When I start actually building this, I will post some details.

Proper Openfiler Installation

Here is my recommended Openfiler installation guide. I initially was going to use pen/flash/usb/thumb drives, but I found out that they are a) very slow, b) somewhat unreliable and c) have a limited write-cycle life.

This is part one of two.

Openfiler Installation - Part I

This setup assumes a configuration of a Chenbro case, motherboard with 2 GB nics on board, an extra NIC for management, a PCI-X slot, 2 small hard drives for booting attached to the motherboard, an Areca 1230 RAID Card,2 GB of RAM and a bunch of SATA disks attached to the Areca card.

At the end of this installation, you will end up with a reliable NFS/iSCSI box that can be used with VMware ESX with good performance. The machine will boot from the hard drives attached to the motherboard, so all hard drive slots (12 available) can be used for storage drives. We did try to use USB sticks to boot from, but they were a) somewhat unreliable and b) have a limited write-cycle (1-5 million writes) and c) quite slow.

You need to have 3 IP’s for the machine – one for management and one for the monitoring of the Areca card. Then you need 1 IP for the iSCSI/NFS traffic – this one is usually on a different subnet.

1) Setup your hardware.
2) Boot the machine – set BIOS appropriately (make boot order CD, then motherboard attached hard drive boot disks)
3) Go into Areca RAID – Press F6
4) Setup IP for Areca card in Ethernet config – default password is 0000
5) Setup RAID volume as one large volume across all disks – either use Quick Volume & Raid setup, or create a Raid and then a Volume
6) Press F10 to Reset/Exit
7) Boot from Openfiler 2.3 64 bit
8) From boot prompt type: expert text
9) Skip the media test
10) Press OK on Welcome to Openfiler
11) Keyboard – US
12) Use Disk Druid
13) Initialize – Yes
14) If any partitions – wipe them out
15) Create a new 200 MB with Filesystem Type: software raid on the first boot disk (usually sda – these will be the smaller disks) – Note that you need to pick the correct hard drive to apply to – only one at a time. Use TAB to move from selection to selection and cursor keys to move up/down.
16) Do the same for the 2nd boot disk (usually sdb)
17) Create a new 10000 MB with Filesystem Type: software raid on the first boot disk
18) Do the same for the 2nd boot disk (usually sdb)
19) Create a new 5000 MB swap partition on each of the boot disks (sda and sdb)
20) Before you begin this step, note which partitions are the 200 MB ones. Now use the RAID option on the partition overview screen. Type “/boot” in the mount point (only type what is inside the quotes). Filesystem type should be ext3. Use RAID1 and only include the 200 MB partitions on each of the 2 boot disks.
21) Again use the RAID option to create a “/” mount point. Again use RAID1, ext3 and use the remaining 2 partitions - one on each disk.
22) Click OK
23) Configure the Ethernet appropriately (usually we don’t use DHCP) and make sure to Activate on Boot for all of them. Also…only put the management IP in on one of them and put the other two as fake IPs for now as they will be replaced by a bonded connection later.
24) Set the hostname – ex: openfiler1.lexcom.local
25) Set the Timezone to America/Regina (unless you are somewhere else!)
26) Set the password
27) Click OK on Log screen
28) Let it install – takes about 15 minutes
29) When it is finished, it may eject the CD. Put it back in and then reboot to the Openfiler CD. These next steps ensure that the machine will boot from either software raided drive.
30) At the boot prompt, type: linux rescue
31) Click OK on next two screens.
32) Answer No to network interface question
33) Click Continue and then OK on next screen
34) At the shell prompt, type: fdisk –l (this is a lowercase “L”)
35) Note which partitions are the 200 MB boot partitions and which ones are the 10000 MB root partitions. (Usually the boot drives are sda and sdb)
36) We want to mount one of the 10000 MB. So type (replace sda2 in the line below with the correct partition):
mount /dev/sda2 /mnt/source
37) Type: cd /mnt/source/sbin
38) Type: ./grub
39) For these next lines, make sure you use the drives that are the boot drives (again, usually sda and sdb) At the grub prompt type (again, replace sda with the correct drive – this will not have a number as that indicates the partition, only the letter):
device (hd0) /dev/sda
40) Type: root (hd0,0) ß-after this expect unknown partition message
41) Type: setup (hd0)
42) Type: device (hd1) /dev/sdb
43) Type: root (hd1,0) ß-after this expect unknown partition message
44) Type: setup (hd1)
45) Type: quit
46) At the shell prompt, type exit
47) Eject the CD and ensure that the BIOS is set to boot from the motherboard hard drives.
48) Reboot and continue with Openfiler configuration.

I will post part two as soon as it is done.

VMware ESX 3.5, MSCS and iSCSI

Trying to setup Microsoft Clustering Services on VMware ESX 3.5 with iSCSI turned out to be a real pain in the *ss.

PDF from VMware (http://www.vmware.com/pdf/vi3_35/esx_3/vi3_35_25_u1_mscs.pdf) indicates the "proper" and supported way of doing it. However, I don't have Fiber channel and if you follow these instructions, you can't VMotion any machine that has Clustered disk (unless you have all VM's on the same ESX server, but then what is the point of the MSCS?)

Anyway, I wanted a solution that would allow me to have clustering, but yet still get the benefit of VMotion.

So...my solution was to use the Microsoft iSCSI Initiator inside the Windows VM's and connect to iSCSI LUNs directly from the VM's.

First I setup the iSCSI SAN. For this I used the newest Openfiler 2.3 as the iSCSI target. I set it up to boot from a pen drive (USB memory stick) following some instructions here: http://ha.nnes.be/index.php/install-openfiler-on-usb-stick/
Mind you....I didn't do the chroot and I had created a separate /boot and / directories, so my install was slightly different. Anyway....got it all working.

Next....setup Windows 2003 VM's with Clustering and install the Microsoft iSCSI initator.

Tired now....gonna post the rest later....


Edit: August 14th, 2008

Found out that USB drives are somewhat flakey for this purpose. Plus they have a limited write cycle. It can be from 100,000 to 1-5 million writes, but on a system that is supposed to be up 24x7 and be extremely reliable, I don't feel comfortable using a $10 USB stick that could wear out.

So...I have redone this all with Openfiler 2.3, Areca 1230 RAID card and 2 small internal hard drives attached via SATA to the motherboard using Software RAID for the boot drives. I have been running this setup for over a month and done extensive testing on various settings/configs to find the one that worked the best (for me).

I use iSCSI for the clustered disks (with Microsoft iSCSI Initiator) and NFS mounts for all regular disks (from VMware ESX). Suprisingly, the NFS performs pretty much the same as iSCSI plus a) it is readable from the Openfiler directly and b) it doesn't have the 2 TB LUN restriction. Only downside I have seen thus far is that I don't get disk performance stats for the NFS disks from ESX. Not a huge thing.

Two important things to get the best speed from NFS - set async mode on the NFS (this assumes you are using a good UPS) - this will significantly improve your write speed. The second thing is to use alb bonding mode for your two gb ethernet nics. This mode does not have any special switch requirements but allows good speed both receiving and sending.

I will try to document my steps here in a little while.