cyborgworkshop.org

Now that I have the LAN almost 100% virtual, I thought I would put a little post together on my ESXi 4.1 whitebox. Most of you won’t care, but ESXi whitebox info is kinda hard to come by, so maybe someone on google will find this useful someday. First, the VM Load. I’m currently running 7 VMs.

Boomer      ubuntu 10.04 utility box that lives in my DMZ
Hybrid       pfSense firewall with 4 physical NICs and 1 virtual attached
GLaDOS     OpenFiler NAS with 3 2TB drives setup for raw sata access
Vault         ubuntu 10.10 server with 3 1TB drives setup for raw sata access for backups of GLaDOS
Serenity    XP VM that I use when I’m oncall
Sleven      Windows 7 that I use for nefarious deeds
DC1          Windows 2008 Server setup as a DC to experiment with.

The ESXi server itself (Jane, Ender fan anyone?) is a quad core Xeon X3220 with 16 gigs of DDR3 RAM on a gigabyte motherboard that has an ICH10 chipset allowing for local SATA storage. I’ve had to add in an additional SIL3114 SATA 1 PCI controller to support the additional drives that are attached to the vault. SATA 1 drive performance hasn’t really been an issue since backups are mostly occurring over a WAN link anyhow. I was given an Intel quad port gigabit PCIEx4 adapter, so that’s my primary NIC and then I have a few old Intel dual 10/100 cards installed just in case. The server itself boots off of a USB thumbdrive and everything is stored in a 4U case with 7 Kingwin trayless racks holding the drives in place. Here are the actual parts I’m using

GIGABYTE GA-P43T-ES3G motherboard
Intel X3220 Quad core Xeon CPU
GSkill RipJaws DDR3 1333 4GB DIMMS x 4 (16GB)
Seagate 2TB SATA drive x3
Seagate 1TB SATA Drive x3
Vantec 6 port SATA II 150 PCI SATA card
Intel EXPI9404PTG2L20 quad port ethernet adapter (freebie)
Intel Dual port 10/100 ethernet adapter (junk parts store)
Kingwin SATA hotswap rack x7
generic 4U 7 5.25 bay rack mount enclosure

The whole system was put together piece by piece over time, but if I had to buy it all again, I would anticipate around $900 if you had no parts at all. My CPU load is almost always below 25% and RAM is around 50%. The amazing part though is that this setup has replaced at least 7 physical machines and very likely more around 10 when you figure in the “I want to try out solaris again” whims. I literally have a 7 foot rack in my basement that has a single server in it. Looks kinda sad really. I have only 1 upgrade planned for the year and that is to replace the USB boot drive with an SSD that I have, but otherwise this server is setup to carry my needs for a good long time.

This is the easiest esxi thumb install yet. Just download the iso file, grab a thumbdrive and dd a single file onto the drive. Assuming that your thumbdrive is /dev/sdd, just run the following.

mount -t iso9660 -o loop VMware-VMvisor-Installer-4.1.0-260247.x86_64.iso /mnt/cdrom/
bzip2 -d /mnt/cdrom/imagedd.bz2 -c | dd of=/dev/sdd


Done, boot it up!

I’m not going to say that this is a really embarrassing bug, but a whole bunch of other people will. If you are using openfiler 2.3 64 bit and select a network’s access to a share to be read only, samba will panic and create a core every time you try to access that share. The bug does appear to be upstream, not openfiler specific, but damn man! The fix is to revert back to an older version of samba and stay there. To revert back, ssh into your openfiler and run the following.

conary update samba=3.2.6-0.0.1-1
conary update samba-client=3.2.6-0.0.1-1
conary update samba-server=3.2.6-0.0.1-1
/etc/init.d/smb restart

That should take care of it.

I have recently moved us from having two boxes, the ESXi server and a file server, down to just the ESXi host. This is a great thing because what used to be 14 servers in a rack in the basement, all with howling fans and clunking harddrives, has been reduced to a single quad core chassis sitting lonely in a 7 foot rack. Where I’m not as comfortable though is with running a vm as my file server and having all of my data tied up in vmdks without a solid, enterprise storage array behind it.  If something goes wrong I want the peace of mind of being able to rip the drives out of my server, pop them into another box and be back on my feet in a matter of minutes. I’m not certain I can do that if I have all of my data spread across 9 or 10 vmdk files on 3 drives, but I know I can pop a linux based, 3 disk RAID-5 set into any old linux machine and be online. My solution is to create a linux file server in vmware, but give it raw access to some attached SATA drives so the drives end up looking and working just like they had come out of a physical machine. No vmware vmdk or other info anywhere in sight. VMWare will allow you to setup raw disk maps to storage that is located on a SAN using the VI client, but not to local SATA storage. Fortunately it’s a very simple process to hack and works like a charm.

You’ll need ssh access to your esxi host, a vmfs that you are not doing raw access to in order to store a mapping file and drives to map.

First, ssh into your esxi server and run fdisk -l to get a listing of all of the drives that you esxi host sees. The format will look something like this

~ # fdisk -l

Disk /dev/disks/t10.ATA_____ST32000542AS________________________________________5XW205BS: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

The key being “/disks/t10.ATA_____ST32000542AS________________________________________5XW205BS” . That is how ESXi uniquely identifies your drive. I believe it breaks down to something along the lines of /disks/controler.interface_Drive-model_Drive-serial-number, but I’m mostly making that up as I go.

Now that we have the drive identified, we need to take that information and create a drive map file. cd over to a datastore that is NOT a raw mapping (i.e. I have a 500 gig drive in my box that hosts most of my vms called SATA-500 in /vmfs/volumes/SATA-500, so I would cd into there) and run the following, replacing with your own information as needed. Note that I use buslogic adapters on my VMs, if you use lsilogic, replace buslogic with lsilogic. The name “RAW-2TB.vmdk” is free form, but I would try to make it descriptive and it must be a .vmdk extension.

vmkfstools -z /vmfs/devices/disks/t10.ATA_____ST32000542AS________________________________________5XW205BS RAW-2TB.vmdk -a buslogic

Two files get created after you run that command. The RAW-2TB.vmdk file that we specified and a RAW-2TB-rdmp.vmdk . You can cat the RAW-2TB.vmdk file to get a look at how vmware saw your drive and what kind of information it’s passing back to the hypervisor on how to use it, but I would not mess with that rdmp file. Think of it as the actual disk, doing a cat on that file is going to give you nothing but a messy terminal if the console will allow you to do it at all.

At this point, switch into your vcenter client and go to the VM that you want to assign these drives to. Go into Edit Settings, Add a Hard Disk, Use an Existing Virtual Disk and browse to the RAW-2TB.vmdk file that was just created. When you select that file, ESXi reads all of the information in it and grants that VM access to that drive without any additional vmware specific info ever getting written to the disk. You can pop that disk into a physical machine and it would read it just as if it came out of a physical machine.