Tag: vSphere

Using iSCSI storage with vSphere

To tap into some of VMware vSphere’s advanced features such as VMotion, fault tolerance, high availability and the VMware Distributed Resource Scheduler, you need to have shared storage for all of your hosts. vSphere’s proprietary VMFS file system uses a special locking mechanism to allow multiple hosts to connect to the same shared storage volumes and the virtual machines (VMs) on them. Traditionally, this meant you had to implement an expensive Fibre Channel SAN infrastructure, but iSCSI and NFS network storage are now more affordable alternatives.

Focusing on iSCSI, we’ll describe how to set it up and configure it properly for vSphere hosts, as well as provide some tips and best practices for using iSCSI storage with vSphere. In addition, we’ve included the results of a performance benchmarking test for the iSCSI/vSphere pairing, with performance comparisons of the various configurations.

Read the full article in the August 2010 issue of Storage Magazine at searchstorage.com…

cover_vol9_iss7

Share This:

Update on vSphere 4.1 power settings from my Tidbits post

I had a comment from a VMware engineer on my recent vSphere 4.1 Tidbits post that talked about the new power monitoring feature that clarified some things and I thought I would share it with everyone. There isn’t much documentation on this stuff yet so every little bit of information helps. From Tim Mann at VMware:

Let me clarify your paragraph on displaying host and VM power usage.  It muddles those two features together a bit.

(1) The feature of displaying the host power consumption is not
experimental and is always on, but will display 0 watts if the host
is not supported or does not have a power meter.  Hopefully most people
should not have to edit /usr/share/sensors/vmware to get support for
their host, but if you do, the instructions in the paragraph are OK.
Here are some more detailed instructions and additional lines that
are going into esx4.1u1:

#
# This file contains a list of power sensors that are known to VMware, Inc.
#
# OEMs: to add support for new machines, do not modify this file
# directly, but place a new file in this directory instead.
#
# Supported format:
#
#  EntryType:SensorType:Manufacturer:Product:Sensor1[,Sensor2…]:Units
#
# EntryType must be “default”, SensorType must be “power”, and Units
# must be “WATTS” (all without quotation marks).
#
# Manufacturer and Product are compared against the system’s DMI (also
# known as SMBIOS) information from its System Information (Type 1)
# record.  Manufacturer and Product are both case-insensitive and will
# match even if the actual name is longer; for example, “
Dell” would
# match “DELL, INC.”.  Product may be “*” to match all products from
# the specified Manufacturer.
#
# Sensor names are case-sensitive and must match exactly.  If multiple
# Sensors are listed on a line (up to 4), sensord reads them all, sums
# them, and reports the total as the system power.  It is acceptable
# for not all of the sensors listed on a line to be present; sensord
# will skip any that are missing as long as at least one is present.
#
default:power:FUJITSU:*:Pwr Mon:WATTS
default:power:FUJITSU:*:Total Power:WATTS
default:power:FUJITSU:*:SYSTEM:WATTS
default:power:FUJITSU:*:PSU1 Power,PSU2 Power:WATTS
default:power:Dell:*:System Level:WATTS
default:power:HP:*:Power Supply 1,Power Supply 2:WATTS
default:power:Hewlett-Packard:*:Power Meter:WATTS
default:power:Hewlett-Packard:*:Power Supply 1,Power Supply 2:WATTS
default:power:NEC:*:POWER:WATTS
default:power:NEC:*:Power:WATTS
default:power:NEC:*:Input_Power:WATTS
default:power:NEC:*:System Power:WATTS
default:power:MITSUBISHI:*:POWER:WATTS
default:power:MITSUBISHI:*:Power:WATTS
default:power:TOSHIBA:*:POWER:WATTS
default:power:TOSHIBA:*:Power:WATTS
default:power:BULL:*:POWER:WATTS

(2) The feature of displaying per-VM power consumption is experimental
and off by default.  It can be turned on with an advanced config option
as the paragraph describes.  The per-VM power consumption feature is
dependent on the host power consumption feature.

Share This:

Train Signal’s new Pro Series video – my experience

Train Signal approached me a few months ago about doing some video training for them in their new Pro Series line of videos. They asked me for topic suggestions and I presented several ideas to them and they ultimately chose the one on vSphere advanced features. I really enjoy exploring new features, big and small, and I’m not content with just figuring out how to use them but also need to know how they work in-depth. As a result I spend a lot of time researching and playing around to find out all the in’s-n-out’s of a feature. I had plenty of experience with the features but the actual recording process was new to me. As a result I approached it as a producer might do for a movie. I first jotted down an outline to use for creating Powerpoint slides, next I created the slides themselves, once I had the slides finalized I created a script for each slide complete with all the text and any demonstration sections. The reason I created the script was because I didn’t want to leave anything out and I wanted everything to be perfect especially if I had to do multiple takes when recording the videos.

Once I had all that down I started creating the videos, I basically created an AVI file for each slide which the editors at Train Signal stitched together to create the final production. In some cases I recorded some slides several times as I don’t like even minor speaking mistakes to be present in the recordings. It was quite a long tedious process and the more I got into it the more comfortable I became with doing it. I think the end result was pretty good, I tend to be highly critical of myself and as a result try to make sure everything is perfect. I have a few work ethics I live by:

  1. Don’t do anything half-ass
  2. Go big or go home
  3. Over-commit and over-deliver.

In a nutshell I try to deliver high-quality work all the time and always exceed expectations. So I think the videos turned out great and I hope you enjoy them, my advanced features videos covered the following features:

  • VMCI
  • VMDirectPath
  • Dynamic Voltage and Frequency Scaling (DVFS)
  • DPM
  • Fault Tolerance
  • pvSCSI adapters
  • VMFS volume grow and VM disk hot-add (and other disk related stuff)
  • VMotion and Storage VMotion
  • Thin Provisioning

Combined with the other great content from David Davis, Sean Clark and Hal Rottenberg the vSphere Pro Series Volume 2 should be a great training resource for both beginning and experienced vSphere administrators. Overall I had a good experience creating the videos and I look forward to doing more of them in the future. Enjoy!

Share This:

New vSphere security feature that you can’t really use yet

According to the original vSphere feature list there is a new security feature called “VMkernel Protection” that uses a technology called Trusted Platform Module (TPM) to add a layer of protection to the VMkernel. The VMkernel (hypervisor) is the most critical component of a virtual host because if it is compromised the VM’s running on it can easily be compromised. Therefore VMware introduced a new protection mechanism in vSphere to ensure the integrity of the VMkernel both on disk and in memory. Here is how it is described by VMware:

VMkernel Protection – As part of ongoing efforts to protect the hypervisor from common attacks and exploits, mechanisms were introduced to assure the integrity of the VMkernel and loaded modules as they reside on disk and in memory. Disk-integrity techniques protect the boot-up of the hypervisor using the Trusted Platform Module (TPM), a hardware device embedded in servers. To ensure the authenticity and integrity of dynamically loaded code, VMkernel modules are digitally signed and validated during load-time. These disk integrity mechanisms protect against malware, which might attempt to overwrite or modify VMkernel as it persists on disk. VMkernel also uses memory integrity techniques at load-time coupled with microprocessor capabilities to protect itself from common buffer-overflow attacks that are used to exploit running code. These techniques create a stronger barrier of protection around the hypervisor. See the ESX Configuration Guide and the ESXi Configuration Guide.

Having a strong interest in security I was curious about this feature and wanted to try it out so I did some research on it. TPM is a security specification developed by Trusted Computing Group (TCG) that uses cryptographic keys to protect information. It relies on a TPM chip which has a unique RSA key burned into it and is capable of performing platform authentication and can be used to verify that software has not been changed. vSphere can use TPM to digitally sign VMkernel modules and validate them when the host is starting up to protect against malware that might overwrite them. This feature is similar to the Windows File Protection feature that Microsoft has built-in to Windows to prevent critical system files from being modified or overwritten.

TPM is integrated into processors and chipsets so just like every other technology Intel has their version of it and AMD their own. Intel’s is called Trusted Execution Technology (TXT) which has been available for some time and AMD’s is called Secure Execution Mode (AMD has very little information on this) and is not widely available. For TPM to work you must have both a CPU with the necessary processor extensions for TPM and a chipset that supports TPM. TPM uses Platform Configuration Registers (PCRs) that are like containers that can hold 160-bit values in them in the following manner:

  • At boot PCRs are all initialized to a known value (either 0 or -1)
  • An application can then measure things by computing its hash value
  • The resulting measurement is inserted into a PCR, this process is called “extending the PCR”
  • PCRs can be extended multiple times until a final value is calculated
  • Each code segment is measured and validated and control passes from one code segment to the next
  • PCRs represent an accumulated measurement of the history of the executed code beginning with power-up
  • TPM signing keys can be used to sign the values of PCRs
  • The system state can then be verified from the hashes that get stored into the PCRs

The technology behind TPM is a bit complex and if you wish to read more there are some great resources at the end of this post that you can check out. As I wanted to see this technology in action I ordered a TPM chip for one of our servers so I could try it out. The chips are fairly cheap, for HP servers they are about $39. They consist of a small little circuit board that plugs into a TPM slot located on the motherboard of the server.

tpm4-new1tpm5-new1

There is also a pin that secures it so if it is ever removed you will know it has been tampered with.

tpm6-new1

Once the chip is inserted some new security options will appear in the server BIOS to configure the TPM chip as shown below.

tpm3-new1

Once I received the chip and put it in the server I turned to the vSphere documentation to set it up. The problem there was there was no documentation on how to do this despite it being advertised as a new vSphere security feature. The ESXi configuration guide had one little paragraph on TPM which didn’t tell how to set it up and use it:

This module is a hardware element that represents the core of trust for a platform and enables attestation of the boot process, as well as cryptographic key storage and protection. As part of the boot process, ESXi measures the VMkernel by the TPM, and changes to the VMkernel are logged from one boot to the next. Measurement values are propagated to vCenter Server, and can be retrieved by third-party agents using the vSphere API.

Frustrated I reached out to VMware to figure out how to use this feature, some of the information I was able to get is below:

  • TPM is only supported with ESXi.
  • You need a TCG compliant BIOS, TXT needs to be enabled from the BIOS. Once it is enabled, you need to enable use of tboot from the UI Advanced configuration option for the ESXi host (the host has to be added to VC to be able to do this).
  • There are some logs in serial log which can be used to monitor TPM. A 3rd party VC API is provided to fetch the TPM PCRs. If TXT was successful, then VMkernel fingerprint is reported in PCR19 otherwise, if the host has TPM but TXT was not used, then it will show in PCR8, otherwise PCRs should be NULL.
  • There might not be any production server platforms out there ‘today’ which can support TXT.

I never did find the “tboot” advanced parameter that was supposed to be enabled. I checked all through the VMkernel advanced settings and didn’t see anything that was even close. It seems like while TPM provides some additional great protection for the VMkernel it is not yet ready to be used. The building blocks are currently there in vSphere but none of the necessary support features to be able to use it effectively exist yet. For example there is no way to monitor the feature so even if you could enable it there would be not much value to it. I expect both 3rd party vendors and VMware will develop the missing pieces in a future release (note the ESX & ESXi 4.1/4.5 version #’s in the videos) and look forward to being able to fully utilize this new security feature.

Share This:

The importance of the Hardware Compatibility List

VMware publishes a list of all server hardware that is supported with vSphere which includes servers, I/O adapters and storage and SAN devices. This list is continually updated and is most commonly referred to as the Hardware Compatibility List (HCL),  VMware changed the name for it a while back to the VMware Compatibility Guide as it is now referred to. The guide used to be published as PDF files only that you could read through to see if your hardware was listed but is now available as an online interactive webpage that is searchable and filterable as well. The guide lists hardware that is supported by vSphere but if hardware is not listed it does not mean it will not work with vSphere. In many cases the hardware will still work but because it is not listed VMware may not provide you support if you have problems with it. Servers and storage devices are two areas that are very common where they work with vSphere despite not being listed in the guide. I/O devices like network adapters and storage controllers though are less likely to work if they are not listed because they rely on drivers loaded in the VMkernel to work properly. If the driver is not one of the limited ones load in the VMkernel than your device is not going to work.

I don’t go through the guide that often as I assume the new mainstream hardware from large vendors like HP will always work and be supported. However last week that assumption bit me. We had just received some new hardware from HP which included a DL385 G6 server and an MSA G2 iSCSI storage array. I wanted to use hardware iSCSI initiators with TCP/IP Offload Engines (TOE) due to high CPU and I/O demands of the applications running on that server so I initially ordered HP’s only TOE card that was available which was the NC380T, however after we placed the order we were informed that the card was no longer available and was replaced by the newer NC382T. After receiving everything and assembling it and loading ESX on it I found that the NC382T was not listed under Storage Adapters in the vSphere Client as TOE cards should be as they are not treated as network adapters in vSphere. Only my SATA adapter, P410 smart array controller for local storage, Fibre Channel adapters and iSCSI software iniator were showing up under Storage Adapters.

iscsi1

The NC382T was showing up under Network Adapters instead and could not be used as a hardware initiator.

iscsi2So this new TOE card from HP that we bought to use as a hardware initiator could not be used for that purpose. After discovering this I checked VMware’s hardware guide to see what HP iSCSI adapters were listed. Much to my surprise there was only one HP model listed and after looking up that adapter on HP’s website I found that it was for blade systems only and was therefore no use to my server.

iscsi3Now HP OEM’s many of their storage adapters as many of  them are made by QLogic and Emulex. Most of the other vendors listed in the guide for iSCSI adapters had many QLogic adapters re-branded under their name but not HP.

iscsi4After confirming with HP that the blade adapter was the only one that they OEM’d I was forced to get the QLogic branded adapter instead. The card that seemed to be the most popular was the QLE4062C adapter, so now we have one of those on order and may end up just using the NC382T as a regular NIC instead.

So the moral of this story is always check the Compatibility Guide especially when ordering a I/O adapters so you don’t end up with hardware that you can’t use with vSphere. If you want to know more about the guide such as how hardware gets added/removed from it and VMware’s support policy check out this blog post that I did a while ago on it.

Share This:

Distributed Power Management

Share This:

New white paper on 5 Ways VMware vSphere Improves Backup and Recovery

Veeam approached me recently and asked if I was interested in writing a white paper for them to coincide with their launch of latest version of Veeam Backup & Recovery (4.0). The timeframe for writing it was short but I accepted it because I was interested in learning more about the vStorage APIs and also Veeam Backup & Recovery. While doing research for it I did learn a lot about the new vStorage APIs that I had not known before. Typically unless you’re a vendor or developer you don’t deal much with APIs, but I’m the curious type though and like to dig deep and find out how things work. There are many important new features in the vStorage APIs and other storage-related APIs in vSphere that are real game-changers for vendors if they choose to take advantage of them. Even if you’re not a developer you should know a bit about them so you have a better understanding of how things work in vSphere.

I learned two additional things while writing the white paper for Veeam, the first is their new 4.0 version of Veeam Backup & Recovery is the first of the many disk to disk backup applications to take full advantage of the new APIs in vSphere, the second is that there are some pretty smart folks at Veeam that are very passionate about their products. I’d like to thank Doug Hazelman & Anton Gostev for answering my many questions about how things work behind the scenes with their product. So go checkout the white paper and more importantly Veeam’s new 4.0 version which I’ll guarantee you’ll be impressed by. Also look for an upcoming tip on searchdatabackup.com that I did that compares how disk to disk backup vendors are using the vStorage APIs and where they are at with their product releases.

Share This:

New white paper: VMware vSphere 4: The CPU Scheduler in VMware ESX 4

VMware just released a new 21 page white paper entitled VMware vSphere 4: The CPU Scheduler in VMware ESX 4 that is all about how the mysterious CPU scheduler functions. This is a highly recommended read and the scheduler is one of the most important components of the hypervisor and therefore you should understand how it functions. The white paper details how the scheduler works, changes made to it in ESX 4, verifies the effectiveness of CPU resource controls including Shares/Reservations/Limits, compares different co-scheduling algorithms and evaluates the performance impact of CPU scheduler changes in ESX 4. Be sure and go check it out.

Share This: