Welcome to vSphere-land!

Storage is the slowest and most complex host resource, and when bottlenecks occur, they can bring your virtual machines (VMs) to a crawl. In a VMware environment, Storage I/O Control provides much needed control of storage I/O and should be used to ensure that the performance of your critical VMs are not affected by VMs from other hosts when there is contention for I/O resources.

Storage I/O Control was introduced in vSphere 4.1, taking storage resource controls built into vSphere to a much broader level. In vSphere 5, Storage I/O Control has been enhanced with support for NFS data stores and clusterwide I/O shares.

Prior to vSphere 4.1, storage resource controls could be set on each host at the VM level using shares that provided priority access to storage resources. While this worked OK for individual hosts, it is common for many hosts to share data stores, and since each host worked individually to control VM access to disk resources, VMs on one host could limit the amount of disk resources on other hosts.

The following example illustrates the problem:

• Host A has a number of noncritical VMs on Data Store 1, with disk shares set to Normal
• Host B runs a critical SQL Server VM that is also located on Data Store 1, with disk shares set to High
• A noncritical VM on Host A starts generating intense disk I/O due to a job that was kicked off; since Host A has no resource contention, the VM is given all the storage I/O resources it needs
• Data Store 1 starts experiencing a lot of demand for I/O resources from the VM on Host A
• Storage performance for the critical SQL VM on Host B starts to suffer as a result

How Storage I/O Control works

Storage I/O Control solves this problem by enforcing storage resource controls at the data store level so all hosts and VMs in a cluster accessing a data store are taken into account when prioritizing VM access to storage resources. Therefore, a VM with Low or Normal shares will be throttled if higher-priority VMs on other hosts need more storage resources. Storage I/O Control can be enabled on each data store and, once enabled, uses a congestion threshold that measures latency in the storage subsystem. Once the threshold is reached, Storage I/O Control begins enforcing storage priorities on each host accessing the data store to ensure VMs with higher priority have the resources they need.

Read the full article at searchvirtualstorage.com…

Today I wanted to highlight another white paper that I wrote for SolarWinds that is titled “Storage I/O Bottlenecks in a Virtual Environment”. I enjoyed writing this one the most as it digs really deep into the technical aspects of storage I/O bottlenecks. This white paper covers topics such as the effects of storage I/O bottlenecks, common causes, how to identify them and how to solve them. Below is an excerpt from this white paper, you can register and read the full paper over at SolarWinds website.

There are several key statistics that should be monitored on your storage subsystem related to bottlenecks but perhaps the most important is latency. Disk latency is defined as the time it takes for the selected disk sector to be positioned under the drive head so it can be read or written to. Once a VM makes a read or write to its virtual disk that request must follow a path to make its way from the guest OS to the physical storage device. A bottleneck can occur at different points along that path, there are different statistics that can be used to help pinpoint where the bottleneck is occurring in the path. The below figure illustrates the path that data takes to get from the VM to the storage device.

The storage I/O goes through the operating system as it normally would and makes its way to the device driver for the virtual storage adapter. From there it goes through the Virtual Machine Monitor (VMM) of the hypervisor which emulates the virtual storage adapter that the guest sees. It travels through the VMkernel and through a series of queues before it gets to the device driver for the physical storage adapter that is in the host. For shared storage it continues out the host on the storage network and makes its way to its final destination which is the physical storage device. Total guest latency is measured at the point where the storage I/O enters the VMkernel up to the point where it arrives at the physical storage device.

The total guest latency (GAVG/cmd as it is referred to in the esxtop utility) is measured in milliseconds and consists of the combined values of kernel latency (KAVG/cmd) plus device latency (DAVG/cmd). The kernel latency includes all the time that I/O spends in the VMkernel before it exits to the destination storage device. Queue latency (QAVG/cmd) is a part of the kernel latency but also measured independently. The device latency is the total amount of time that I/O spends in the VMkernel physical driver code and the physical storage device. So when I/O leaves the VMkernel and goes to the storage device this is the amount of time that it takes to get there and return. A guest latency value that is too high is a pretty clear indication that you have a storage I/O bottleneck that can cause severe performance issues. Once total guest latency exceeds 20ms you will notice the performance of your VMs suffer, as it approaches 50ms your VMs will become unresponsive.

Full paper including information on the key statistics related to storage I/O bottlenecks available here…

The iPad is becoming more and more popular in the enterprise, and not just for mobile workers. There is also a slew of iPad applications for VMware admins.

Many IT vendors see the iPad’s potential and are developing iPad apps that can manage their traditional hardware and software products. Xsigo Systems, for instance, has a very nice app called Xsigo XMS , which manages virtual I/O through the company’s XMS servers. There is also an iPad application called SiMU Pro that manages Cisco Systems Inc.’s Unified Computing System.

In addition, there are several iPad applications that can supplement the traditional VMware admin toolkit, including the vSphere Client and Secure Shell (SSH) applications. With the right iPad applications, VMware admins will reach a new level of management flexibility that’s not possible with traditional desktops and laptops.

TABLE OF CONTENTS

I. Top iPad applications for VMware management

II. Applications for remotely connecting to hosts and workstations

III. Top iPad applications for VMware networking

IV. General purpose iPad apps for VMware admins

I. TOP IPAD APPLICATIONS FOR VMWARE MONITORING AND MAINTENANCE
With the top iPad applications for infrastructure management, VMware admins can control basic functions, such as powering virtual machines (VMs) on and off and using vMotion.

These iPad apps mimic some of the functionality of the vSphere Client and service console, but they aren’t a full-fledge replacement. Even so, these iPad applications allow VMware admins to perform key virtualization tasks without a full-scale computer.

Read the full article at searchvmware.com…

Came across this session in the VMworld 2011 lineup - (BCO2874 vSphere High Availability 5.0 and SMP Fault Tolerance - Technical Overview), according to the description the Fault Tolerance feature which previously only supported VM’s with a single vCPU will soon support vSMP. Providing this support has been a major development challenge for VMware as it is not an easy task to keep multiple CPUs in sync via lockstep. In fact I talked to VMware about it at the vSphere 5 launch and they said they were working with the CPU vendors to see if they could make it easier for them to do this. Well looks like they finally found a way, it will be interesting to see what they come up with, sign up for the VMworld session to find out more including a live demo. Since it’s not currently in vSphere 5.0 I’m guessing it may make it into a point release like vSphere 5.1.

In a virtual server environment, the interaction between hypervisor and the storage hardware that supports it is complicated. In an effort to simplify that interaction and make it more efficient, VMware developed the vStorage APIs for Array Integration (VAAI). The APIs create a separation of duty between the hypervisor and its storage devices, enabling each to focus on what it does best: virtualization-related tasks for the hypervisor and storage-related tasks for the storage arrays.

With VAAI, storage array vendors can directly integrate their storage hardware and applications with vSphere. VAAI enables certain storage tasks, such as cloning, to be offloaded to the storage array, which can complete them more efficiently than the host can. Rather than use host resources to perform the work (which was required prior to VAAI), the host can simply pass the task onto the storage array, which will perform it while the host monitors the progress of the task. The storage array is purposely built to perform storage tasks and can complete requests much faster than the host can.

Read the full article at searchvirtualstorage.com…

ESXi 4.1 brought changes to the patching process. Previously, the Host Update utility — an application included with the vSphere Client — could patch ESXi 4.0 hosts. VMware removed Host Update from ESXi 4.1, presumably to encourage users to upgrade to paid versions that are managed and patched with vCenter Server’s Update Manager. As a result, the only method left to patch the free version of ESXi is with the vihostupdate command-line utility, which is included in the vSphere Command-Line Interface (CLI).

Before using this method, it’s important to understand how the patches work and where to find them.

Read the full article at searchvmware.com…

VMware announced that ESXi will be the exclusive hypervisor of vSphere 5. As such, we will likely see a greater adoption of VMware’s smaller hypervisor.

ESXi can be either embedded on a server (boot from flash) or installed on existing servers, using the Installable version. The free version of ESXi, the VMware vSphere Hypervisor, includes support for virtual symmetric multiprocessing (vSMP) and thin provisioning. No additional features are included, which means the free version of ESXi cannot be managed by vCenter Server, because it does not include a vCenter Server agent. To gain additional features and a vCenter Server agent, you need to upgrade your ESXi license.

The ESXi installation uses about 5 GB of space. Any remaining space on the drive is automatically formatted as a Virtual Machine File System (VMFS) partition. The hypervisor needs roughly 32 MB; the additional space is used for VMware Tools as well as swap and core dump partitions.

If you already have existing licenses for ESX, you can also choose to deploy ESXi in place of ESX on a server. Simply download ESXi installable. Install it and then license it with vCenter Server, as you would a traditional ESX server. Follow the steps below to install and configure ESXi.

Read the full article at searchvmware.com…

… Mr. T will come whoop your butt.

Seriously though, if you’re involved in virtualization in any way, VMware’s special event on 7/12 is one you really can’t afford to miss. VMware is being vague on what the event is about but you should be able to put two and three together and come up with five. The reason VMware is being purposely vague is because their lawyers told to them be (yeah it sucks being a public company sometimes). But just because you came up with five doesn’t tell the whole story, you’re probably expecting to hear about something related to five but their is much more to it than that. VMware will be making more announcements than you will be expecting and you definitely want to hear what they have to say.

I am one of four bloggers invited to attend the event live, it is not being held in Palo Alto like the vSphere announcement was but instead in San Francisco much like an Apple event. The event starts at 9:00am PST and will go for until about 12:00 PST, below is the schedule for everything:

• 9:00-9:45 Paul and Steve present - live online streaming
• 10:00-12:00 three tracks of deep dive breakout sessions
• 10:00-12:00 live Q&A with VMware cloud and virtualization experts

After the event David Davis & I are going to try and get some Q&A time with Steve Herrod and record a live vChat, so if you have any questions for him send them to us. So register now, seriously you don’t want to miss this one, their are some exciting times ahead for VMware!

With applications becoming increasingly memory-hungry, memory always seems to be the resource in the shortest supply on a host. In a virtualized server environment, the lack of sufficient physical memory can have a real impact on project ROI, since it will limit the density that you could otherwise achieve.

That’s because, while CPU sockets can support an ever-increasing number of cores, for maximum efficiency, a host has to have a balance of resources available to virtual machines (VMs). If a resource shortage occurs in any one area on a host — such as RAM — the number of VMs that the host can run will be restricted despite plentiful resources in other areas.

The technique of memory overcommitment in a virtualized environment can help mitigate the problem, and using SSD to handle memory overcommitment can produce better results at a lower cost than memory overcommitment on mechanical disk.

Let’s examine the memory limitations of server hardware, how memory overcommitment works, and how solid-state drive (SSD) technology can address the problem.

Read the full article at searchvirtualstorage.com…

Your customer’s vSphere environment may seem healthy enough at first glance, but external appearances can be deceiving. Hidden problems may lurk under the cover that can eventually cause the environment to become unhealthy.

Not all problems that can occur are obvious to VARs because virtualization is much more complicated than traditional computing environments. There are many more moving parts that can lead to more complications. Seemingly trivial things such as simple configuration settings can have a ripple effect on the environment if they aren’t set correctly. Virtualization is all about sharing a limited set of resources amongst many virtual machines (VMs) and ensuring the optimum performance and availability you need to ensure the environment stays healthy.

A vSphere environment health check is one of the most valuable services that you can offer to customers because vSphere needs constant maintenance to keep it operating efficiently and problem-free. A good health check should be done on a periodic basis and is similar to a home inspection in that it documents the environment and checks the many different vSphere components to identify any existing or potential problems that can occur. Health check results can also be used to identify potential optimization and performance improvement opportunities that VARs can turn into value-adds.

Read the full article at searchsystemschannel.com…

I just launched my new vBookshelf section of vSphere-land which can be found under the vInfo drop-down menu. I’ve gathered together over 30 books related to VMware and virtualization and have links and information on them. I think I’ve put together a pretty complete selection of good books that are available but if I’ve missed any please let me know. I’d also like to highlight 4 good books that have been recently released.

Click here to access the vBookshelf section of vSphere-land.com

Title: Visible Ops Private Cloud: From Virtualization to Private Cloud in 4 Practical Steps

Authors: Andi Mann, Kurt Milne, Jeanne Morain

Publish Date: April 8, 2011

Title: VMware vSphere Design

Authors: Scott Lowe, Maish Saidel-Keesing, Forbes Guthrie

Publish Date: March 8, 2011

Title: VMware vSphere PowerCLI Reference: Automating vSphere Administration

Authors: Luc Dekens, Alan Renouf, Glenn Sizemore, Arnim van Lieshout, Jonathan Medd

Publish Date: April 12, 2011

Title: VMware ESX and ESXi in the Enterprise: Planning Deployment of Virtualization Servers (2nd Edition)

Authors: Edward Haletky

Publish Date: February 18, 2011

This one is long overdue, I had a section created on my website a year ago that correlated to a link in my book, Maximum vSphere that listed all the many free tools that would be useful to a VMware administrator. But I had forgotten about it until recently so I scoured the internet for free tools and put together a big list of them. The table currently has over 80 tools and is sortable by category, name and vendor to help you go through them. If there is anything missing be sure and let me know.

Click here to access the Free Tools section of vSphere-land.com

Desktop virtualization, or virtual desktop infrastructure (VDI), can bring many benefits to an IT organization, including easier system management and centralized security and data protection. But the storage environment that supports VDI requires some careful planning to avoid the problem of VDI “boot storms” — big slowdowns that can occur when a lot of users log into the system at the same time. There are a few options for addressing the problem, but the one that makes the most sense relies on tactical placement of solid-state drives (SSDs).

The problem of VDI boot storms is a fairly straightforward one. Virtual desktop workloads are predictable; they’re based on the work hours of desktop users, which typically run from about 8 a.m. to 5 p.m. each workday. The overall storage I/O that an average virtual desktop generates is quite low compared with that of a server workload, and so the density of desktop virtual machines on a host is typically much greater than with server virtualization. Conversely, the initial startup of a desktop is very resource-intensive, where the operating system and applications do a large amount of reading from disk while loading and executing.

A boot storm occurs when many virtual desktops all boot up during a short window of time (for example, between 8 a.m. and 9 a.m.), which causes intense concentrated storage I/O that can easily overwhelm a storage subsystem. If the storage subsystem isn’t designed to handle the heavy I/O load, you can effectively end up with a denial-of-service attack on your storage subsystem.

Read the full article at searchvirtualstorage.com…

This chapter excerpt on Upgrading to vSphere (download PDF) is taken from the book Maximum vSphere: Tips, How-Tos, and Best Practices for Working with VMware vSphere 4. Solution providers can use this book to learn about vSphere 4 storage, networking, performance monitoring and advanced features such as high availability, distributed resource, distributed power management and Vmotion.

If you have an existing VI3 environment, at some point you’ll probably want to upgrade it to vSphere. Before jumping right into the upgrade process, though, there are many considerations and requirements that you should be aware of. Once you are aware of everything you need to know, you should then put together a plan for how you are going to proceed. Upgrading to vSphere is fairly straightforward, but there are many gotchas that can make the process more difficult. To avoid surprises during the upgrade, you should properly prepare and know all the steps so that your upgrade is trouble-free and uneventful. In this chapter, we will cover considerations and steps for upgrading your existing virtual environment to vSphere.

Compatibility comparisons
There are many things to consider when upgrading your VI3 environment to vSphere, such as hardware and software compatibility and upgrade methods. You should spend some time researching this to ensure that you have all your bases covered beforehand. Finding out after you upgrade that some of your management tools are not compatible with vSphere can make things very difficult. Upgrading is a much simpler process than downgrading, so make sure you consider everything before beginning your upgrade.

Hardware compatibility
Your server and storage hardware may be supported in VI3, but don’t assume that it’s supported in vSphere. Check VMware’s online Hardware Compatibility Guide to make sure all your hardware components are supported in vSphere. This includes servers, I/O adapters, and storage devices. You may be able to get away with using servers that are not listed in the guide, but it’s critical that your I/O adapters and storage are listed. Refer to the Importance of the Hardware Compatibility Guide section in Chapter 11 for more information on this. The other consideration that you need to be aware of in regard to hardware is the requirement for 64-bit hardware. See the section Selecting Physical Host Hardware to Use with vSphere in Chapter 2 for more information on this.

Read the full chapter excerpt at searchsystemschannel.com…

For better or worse, administrators usually accept the default VMware hypervisor security settings.

VSphere is fairly secure, but VMware security breaches can still occur. Careless mistakes and questionable administrative decisions can weaken infrastructure security — especially if IT pros are more concerned about management convenience than about hardening the hypervisor, hosts and virtual machines (VMs).

To help prevent snafus, here are five ways to maximize VMware hypervisor security.

Firewalls prevent VMware hypervisor security from getting burned
Physical firewalls protect servers and devices directly connected to physical networks, but they aren’t always effective at protecting VMs connected to virtual networks. So use virtual firewalls in conjunction with physical firewalls to ensure that network traffic is secure at every level and nothing slips through the cracks.

Sometimes, virtual machine network traffic doesn’t leave the host or travel over a physical network. Traffic between VMs on the same vSwitch and port group remains inside the host. It travels in the host’s memory, through the virtual network — rather than over the physical network. As such, it’s outside the physical firewall’s protection zone.

Read the full article at searchvmware.com…