cyborgworkshop.org » VMWare

My ESXi Whitebox

jason — Mon, 17 Jan 2011 03:38:18 +0000

Now that I have the LAN almost 100% virtual, I thought I would put a little post together on my ESXi 4.1 whitebox. Most of you won’t care, but ESXi whitebox info is kinda hard to come by, so maybe someone on google will find this useful someday. First, the VM Load. I’m currently running 7 VMs.

Boomer ubuntu 10.04 utility box that lives in my DMZ
Hybrid pfSense firewall with 4 physical NICs and 1 virtual attached
GLaDOS OpenFiler NAS with 3 2TB drives setup for raw sata access
Vault ubuntu 10.10 server with 3 1TB drives setup for raw sata access for backups of GLaDOS
Serenity    XP VM that I use when I’m oncall
Sleven    Windows 7 that I use for nefarious deeds
DC1          Windows 2008 Server setup as a DC to experiment with.

The ESXi server itself (Jane, Ender fan anyone?) is a quad core Xeon X3220 with 16 gigs of DDR3 RAM on a gigabyte motherboard that has an ICH10 chipset allowing for local SATA storage. I’ve had to add in an additional SIL3114 SATA 1 PCI controller to support the additional drives that are attached to the vault. SATA 1 drive performance hasn’t really been an issue since backups are mostly occurring over a WAN link anyhow. I was given an Intel quad port gigabit PCIEx4 adapter, so that’s my primary NIC and then I have a few old Intel dual 10/100 cards installed just in case. The server itself boots off of a USB thumbdrive and everything is stored in a 4U case with 7 Kingwin trayless racks holding the drives in place. Here are the actual parts I’m using

GIGABYTE GA-P43T-ES3G motherboard
Intel X3220 Quad core Xeon CPU
GSkill RipJaws DDR3 1333 4GB DIMMS x 4 (16GB)
Seagate 2TB SATA drive x3
Seagate 1TB SATA Drive x3
Vantec 6 port SATA II 150 PCI SATA card
Intel EXPI9404PTG2L20 quad port ethernet adapter (freebie)
Intel Dual port 10/100 ethernet adapter (junk parts store)
Kingwin SATA hotswap rack x7
generic 4U 7 5.25 bay rack mount enclosure

The whole system was put together piece by piece over time, but if I had to buy it all again, I would anticipate around $900 if you had no parts at all. My CPU load is almost always below 25% and RAM is around 50%. The amazing part though is that this setup has replaced at least 7 physical machines and very likely more around 10 when you figure in the “I want to try out solaris again” whims. I literally have a 7 foot rack in my basement that has a single server in it. Looks kinda sad really. I have only 1 upgrade planned for the year and that is to replace the USB boot drive with an SSD that I have, but otherwise this server is setup to carry my needs for a good long time.

Install ESXi 4.1 onto a thumbdrive

jason — Thu, 13 Jan 2011 20:53:08 +0000

This is the easiest esxi thumb install yet. Just download the iso file, grab a thumbdrive and dd a single file onto the drive. Assuming that your thumbdrive is /dev/sdd, just run the following.
mount -t iso9660 -o loop VMware-VMvisor-Installer-4.1.0-260247.x86_64.iso /mnt/cdrom/ bzip2 -d /mnt/cdrom/imagedd.bz2 -c | dd of=/dev/sdd

Done, boot it up!

Enabling raw SATA access in ESXi free

jason — Sun, 09 Jan 2011 02:56:55 +0000

I have recently moved us from having two boxes, the ESXi server and a file server, down to just the ESXi host. This is a great thing because what used to be 14 servers in a rack in the basement, all with howling fans and clunking harddrives, has been reduced to a single quad core chassis sitting lonely in a 7 foot rack. Where I’m not as comfortable though is with running a vm as my file server and having all of my data tied up in vmdks without a solid, enterprise storage array behind it. If something goes wrong I want the peace of mind of being able to rip the drives out of my server, pop them into another box and be back on my feet in a matter of minutes. I’m not certain I can do that if I have all of my data spread across 9 or 10 vmdk files on 3 drives, but I know I can pop a linux based, 3 disk RAID-5 set into any old linux machine and be online. My solution is to create a linux file server in vmware, but give it raw access to some attached SATA drives so the drives end up looking and working just like they had come out of a physical machine. No vmware vmdk or other info anywhere in sight. VMWare will allow you to setup raw disk maps to storage that is located on a SAN using the VI client, but not to local SATA storage. Fortunately it’s a very simple process to hack and works like a charm.

You’ll need ssh access to your esxi host, a vmfs that you are not doing raw access to in order to store a mapping file and drives to map.

First, ssh into your esxi server and run fdisk -l to get a listing of all of the drives that you esxi host sees. The format will look something like this

~ # fdisk -l

Disk /dev/disks/t10.ATA_____ST32000542AS________________________________________5XW205BS: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

The key being “/disks/t10.ATA_____ST32000542AS________________________________________5XW205BS” . That is how ESXi uniquely identifies your drive. I believe it breaks down to something along the lines of /disks/controler.interface_Drive-model_Drive-serial-number, but I’m mostly making that up as I go.

Now that we have the drive identified, we need to take that information and create a drive map file. cd over to a datastore that is NOT a raw mapping (i.e. I have a 500 gig drive in my box that hosts most of my vms called SATA-500 in /vmfs/volumes/SATA-500, so I would cd into there) and run the following, replacing with your own information as needed. Note that I use buslogic adapters on my VMs, if you use lsilogic, replace buslogic with lsilogic. The name “RAW-2TB.vmdk” is free form, but I would try to make it descriptive and it must be a .vmdk extension.

vmkfstools -z /vmfs/devices/disks/t10.ATA_____ST32000542AS________________________________________5XW205BS RAW-2TB.vmdk -a buslogic

Two files get created after you run that command. The RAW-2TB.vmdk file that we specified and a RAW-2TB-rdmp.vmdk . You can cat the RAW-2TB.vmdk file to get a look at how vmware saw your drive and what kind of information it’s passing back to the hypervisor on how to use it, but I would not mess with that rdmp file. Think of it as the actual disk, doing a cat on that file is going to give you nothing but a messy terminal if the console will allow you to do it at all.

At this point, switch into your vcenter client and go to the VM that you want to assign these drives to. Go into Edit Settings, Add a Hard Disk, Use an Existing Virtual Disk and browse to the RAW-2TB.vmdk file that was just created. When you select that file, ESXi reads all of the information in it and grants that VM access to that drive without any additional vmware specific info ever getting written to the disk. You can pop that disk into a physical machine and it would read it just as if it came out of a physical machine.

Enabling SNMP on an ESXi 4 free edition

jason — Sun, 01 Aug 2010 18:05:05 +0000

ESXi doesn’t include a lot of the management functions that the licensed product carries (see my previous post on doing backups in ESXi) but most of those functions are already installed, just disabled. Here is how to enable SNMP queries and traps on an ESXi server.

Enable ssh on your server
I’m assuming you have already licensed your server, we’re going to need to back that out so ssh to the server as root
run the following commands

mv /etc/vmware/vmware.lic /etc/vmware/vmware.lic.orig mv /etc/vmware/license.cfg /etc/vmware/license.cfg.orig services.sh restart

That will temporarily put you back into an eval license, you will be disconnected from your vsphere client while the services restart
Now you need access to the ESX remote command line. You can do that by either downloading and installing the appliance or you can install it locally on your machine.
Configure SNMP for your environment. -c is the snmp community you want to set for queries and -t is the trap destination

vicfg-snmp.pl –server 192.168.1.1 -c public -t 192.168.1.2@162/public

enable snmp

vicfg-snmp.pl –server 192.168.1.1 -E

Test your new snmp config

vicfg-snmp –server 192.168.1.1 -T

and lastly, download the snmp MIBs for your management platform
Now move your license keys back

mv /etc/vmware/vmware.lic.orig /etc/vmware/vmware.lic mv /etc/vmware/license.cfg.orig /etc/vmware/license.cfg services.sh restart


And that's it, you have snmp queries and traps on your ESXi server now.



Backing up an ESXi 4 server
jason — Sat, 31 Jul 2010 15:21:17 +0000
ESXi is one of my favorite products on the market and is really excellent for home use.  It’s missing out on a few enterprise features like SNMP (I’ll show you how to get around that in a later post) but is otherwise very complete.  I had previously been using a separate storage server using iscsi for my vmfs, but the dependency of the two boxes kept me from shutting down either server as needed. So, I converted everything back to local SATA storage (another post) and went on a hunt for how to backup my VMDKs.  My iSCSI SAN had dual power supplies, 2 hot standby drives and I could do snapshot based backups whenever I wanted. My ESXi server has two SATA drives, one power supply and no easy way to backup. Enter GhettoVCB. GhettoVCB is a brilliant little shell script that runs on the ESXi server and performs backups of your vms using snapshots that can be stored on local, attached or network storage.  Getting GhettoVCB up and running isn’t difficult at all, you just need to make sure you have a few things in place first.

A place to store the backups (NFS, iSCSI, Local storage and potentially DAS)
Decide how many backups you want to have available
Decide How you want to store the files (thin, thick, etc) This consideration will impact how many backups you have available because of the different space requirements.
And lastly when you want your backups to run.

I’ve decided to take thin copies of my vmdks at midnight every night and store them on an NFS share with 5 backups available before the older ones get replaced.  I’m not going to try to replicate the excellent instructions that already exist for GhettoVCB,  but here is how I did it.

Create an NFS mount on my NFS server with the following options
/mnt/raid5/VMWare       10.0.0.0/24(rw,async,all_squash,anonuid=99,anongid=99)
Mount the NFS share in vSphere and call it Backups
enable ssh on your ESXi server
scp ghettovcb.tar.gz to your ESXi server
untar ghettovcb.tar.gz to /usr/ghettovcb
edit ghettoVCB.sh and change VM_BACKUP_VOLUME to be VM_BACKUP_VOLUME=/vmfs/volumes/Backups
edit DISK_BACKUP_FORMAT to be DISK_BACKUP_FORMAT=thin
edit VM_BACKUP_ROTATION_COUNT to VM_BACKUP_ROTATION_COUNT=5
edit EMAIL_LOG to EMAIL_LOG=0  unless you want to setup logs to be emailed to you.
edit cron by running this command
echo "0 6 * * * /usr/ghettoVCB/ghettoVCB.sh -a > /var/log/ghettoVCB-backup-$(date +\%s).log" >> /var/spool/cron/crontabs/root

kill $(cat /var/run/crond.pid)

busybox crond 
My cron is set for 6am because my clock is set to UTC, to 6am is midnight for me
Wait until midnight for the cron to run or run it by hand yourself.

Logs will be stored in /var/log/  .  I’m backing up 8 VMs in roughly an hour using about 60GB of space (thin provisioned) and have tested restores successfully. Good luck and let me know if you have any issues!