November 2011 archive

Nov 28 2011

vSphere Storage I/O Control: What it does and how to configure it

Storage is the slowest and most complex host resource, and when bottlenecks occur, they can bring your virtual machines (VMs) to a crawl. In a VMware environment, Storage I/O Control provides much needed control of storage I/O and should be used to ensure that the performance of your critical VMs are not affected by VMs from other hosts when there is contention for I/O resources.

Storage I/O Control was introduced in vSphere 4.1, taking storage resource controls built into vSphere to a much broader level. In vSphere 5, Storage I/O Control has been enhanced with support for NFS data stores and clusterwide I/O shares.

Prior to vSphere 4.1, storage resource controls could be set on each host at the VM level using shares that provided priority access to storage resources. While this worked OK for individual hosts, it is common for many hosts to share data stores, and since each host worked individually to control VM access to disk resources, VMs on one host could limit the amount of disk resources on other hosts.

The following example illustrates the problem:

  • Host A has a number of noncritical VMs on Data Store 1, with disk shares set to Normal
  • Host B runs a critical SQL Server VM that is also located on Data Store 1, with disk shares set to High
  • A noncritical VM on Host A starts generating intense disk I/O due to a job that was kicked off; since Host A has no resource contention, the VM is given all the storage I/O resources it needs
  • Data Store 1 starts experiencing a lot of demand for I/O resources from the VM on Host A
  • Storage performance for the critical SQL VM on Host B starts to suffer as a result

How Storage I/O Control works

Storage I/O Control solves this problem by enforcing storage resource controls at the data store level so all hosts and VMs in a cluster accessing a data store are taken into account when prioritizing VM access to storage resources. Therefore, a VM with Low or Normal shares will be throttled if higher-priority VMs on other hosts need more storage resources. Storage I/O Control can be enabled on each data store and, once enabled, uses a congestion threshold that measures latency in the storage subsystem. Once the threshold is reached, Storage I/O Control begins enforcing storage priorities on each host accessing the data store to ensure VMs with higher priority have the resources they need.

Read the full article at searchvirtualstorage.com…

Share This:

Nov 24 2011

Easy way to transfer files to a VM from the outside

I find myself frequently needing to copy files such as application installation files from my workstation to the guest OS running inside a VM. Sure you could do this using UNC paths and shares by connecting to the VM at the guest OS layer and authenticating with the VM and then copying files with Windows Explorer but this can be a pain sometimes. It also means that those files are taking up space on the VM and if its only temporary and you’re using thin provisioning you can’t easily reclaim that space. I also find myself installing the same applications on many VMs and to have to copy the files to each VM can be time consuming.

So instead of messing with copying files through the guest OS layer using a copy utility, I prefer to copy them through the virtualization layer using an ISO file. By using an ISO file I can easily mount it on the VMs CD/DVD drive and then access the files without having to copy them to the VM’s file system. This is the same method by which VMware Tools is installed on a VM, the VMware Tools install binaries are mounted from a special mount point on the host disk partition to the CD/DVD drive of a VM so they can be installed. You’ll notice if you create a VM without a CD/DVD drive that you won’t be able to install VMware Tools and will receive an error if you try.

2011-11-24_161603

So you might think, creating an ISO file, isn’t that a pain in the butt as well. It’s actually very quick and easy and once you create it you can use it over and over on any VM to install applications with and copy files to VMs if needed. You can keep a collection of them on your workstation and mount them from your local disk or you can copy them to a host datastore and mount them from there. I sometimes create toolkit ISO images that contain many of the typical applications (i.e. Sysinternals, disk tools, etc.) that I use when troubleshooting problems within a VM. Windows still does not have the native ability to read or write ISO files but there are a number of free tools that you can create ISO files with such as ImgBurn, CDBurnerXP (works on Win7) and ISO Recorder. I usually use ImgBurn but ISO Recorder is even easier as you can select the files/folders that you want to include in the ISO file and launch it via the Windows Explorer menus like below.

2011-11-24_164311

Next you simply select a directory and a name to save your ISO file in, click Next and your ISO file will be created.

2011-11-24_164619

You can now mount it inside the VM using the virtual CD/DVD drive of the VM by browsing to the ISO file you just created so you can access all the files inside the ISO file from within the VM.

2011-11-24_164952

That’s it, quick and easy, takes less than a minute to complete and much easier then having to copy files through the guest OS of the VM.

Share This:

Nov 19 2011

New ESXi 5.0 build to fix Software iSCSI Initiator issue

VMware has recently released a new build of ESXi to fix a bug that causes ESXi to hang for a long period of time while it tries to connect to all iSCSI targets. I’ve personally seen this happen in my lab and it can take quite a long time for ESXi to boot as it will try 9 times to connect to each iSCSI target. VMware sees this as a serious enough issue that not only have they released a patch to fix the problem but they’ve also released a special patch express release of ESXi. So when you go to download ESXi 5.0 now you will see two options for the ESXi ISO: one for systems without software iSCSI configured and one for systems with software iSCSI configured. If you are already using software iSCSI or plan on it at some point  you should choose the ISO image for systems with software iSCSI. You can read more about this issue in this VMware KB article. Here is the details on the two ESXi builds:

  • Original release: Version 5.0.0 – Release Date 8/24/11 – Build 469512
  • iSCSI patch release: Version 5.0.0 – Release Date 11/10/11 – Build 504890

2011-11-19_075315

Share This:

Nov 18 2011

Capacity Planning in Virtual Environments

This is my final post highlighting the white papers that I did for SolarWinds. This one focuses on a white paper titled “Capacity Planning in Virtual Environments” which is a topic that is often not very well understood or executed in virtual environments. Capacity planning can be a real challenge in virtual environments and there is a lot more to it than meets the eye, below is an excerpt from this white paper, you can register and read the full paper over at SolarWinds website.

Virtualization is all about the sharing of resources. You have to plan with the big picture in mind and take into account your virtual environment as a whole. A balance of resources is critical in a virtual environment since the server hardware used for virtualization is bigger and more expensive than traditional server hardware as it has to support many virtual machines (VMs) running on it. If resources are unbalanced on a host, it can lead to wasted resources, and since the whole point of virtualization is to make the most efficient use of all resources, this goes against the reasons that we virtualize in the first place. For example, if a host runs out of physical memory, it limits the number of VMs that can run on that host despite having plenty of other resources available to it. Sure, you can use memory over commitment, but performance severely degrades once your VMs start swapping to disk to make up for the lack of physical host memory. As a result, the lack of having enough physical memory available for VMs means that you are wasting resources and money.

Trying to keep your resources balanced isn’t all that simple; you need to look at historical resource trends and usage to determine what that balance point is. Trying to calculate this manually is almost impossible. You need tools that can analyze your historical data and report how your environment has grown over time, how it stands today and how it will look in the future. Another area that further complicates resource calculations is spare capacity. If you are using High Availability in your environment, which most companies do, you have to maintain sufficient spare capacity to handle the load when host failures occur. So, you need a pool of resources that is unused at all times so you have enough resource capacity to handle VMs from failed hosts. Trying to factor spare capacity into your resource calculations can quickly get complicated, having a tool that can do this for you can make it a much easier exercise.

Full paper available here…

Share This:

Nov 17 2011

Storage I/O Bottlenecks in a Virtual Environment

Today I wanted to highlight another white paper that I wrote for SolarWinds that is titled “Storage I/O Bottlenecks in a Virtual Environment”. I enjoyed writing this one the most as it digs really deep into the technical aspects of storage I/O bottlenecks. This white paper covers topics such as the effects of storage I/O bottlenecks, common causes, how to identify them and how to solve them. Below is an excerpt from this white paper, you can register and read the full paper over at SolarWinds website.

There are several key statistics that should be monitored on your storage subsystem related to bottlenecks but perhaps the most important is latency. Disk latency is defined as the time it takes for the selected disk sector to be positioned under the drive head so it can be read or written to. Once a VM makes a read or write to its virtual disk that request must follow a path to make its way from the guest OS to the physical storage device. A bottleneck can occur at different points along that path, there are different statistics that can be used to help pinpoint where the bottleneck is occurring in the path. The below figure illustrates the path that data takes to get from the VM to the storage device.

latency3

The storage I/O goes through the operating system as it normally would and makes its way to the device driver for the virtual storage adapter. From there it goes through the Virtual Machine Monitor (VMM) of the hypervisor which emulates the virtual storage adapter that the guest sees. It travels through the VMkernel and through a series of queues before it gets to the device driver for the physical storage adapter that is in the host. For shared storage it continues out the host on the storage network and makes its way to its final destination which is the physical storage device. Total guest latency is measured at the point where the storage I/O enters the VMkernel up to the point where it arrives at the physical storage device.

The total guest latency (GAVG/cmd as it is referred to in the esxtop utility) is measured in milliseconds and consists of the combined values of kernel latency (KAVG/cmd) plus device latency (DAVG/cmd). The kernel latency includes all the time that I/O spends in the VMkernel before it exits to the destination storage device. Queue latency (QAVG/cmd) is a part of the kernel latency but also measured independently. The device latency is the total amount of time that I/O spends in the VMkernel physical driver code and the physical storage device. So when I/O leaves the VMkernel and goes to the storage device this is the amount of time that it takes to get there and return. A guest latency value that is too high is a pretty clear indication that you have a storage I/O bottleneck that can cause severe performance issues. Once total guest latency exceeds 20ms you will notice the performance of your VMs suffer, as it approaches 50ms your VMs will become unresponsive.

Full paper including information on the key statistics related to storage I/O bottlenecks available here…

Share This:

Nov 16 2011

Performance Management in a Virtual Environment

Continuing from my post on Monday which covered the white paper that I did for SolarWinds on the top 5 management challenges with virtualized environments, I wanted to highlight another of the white papers that focuses on one of those specific management challenges. This white paper is titled “Performance Management in a Virtual Environment” and covers how performance management differs from traditional physical environments, how to get started with performance management and knowing where to look and how to interpret the many statistics that are unique to virtual environments. The white paper also includes a table that details 13 key statistics that you need to pay attention to in virtual environments. Below is an excerpt from the first paper, you can register and read the full paper over at SolarWinds website.

So you’ve implemented virtualization and don’t know where to start when it comes to monitoring the performance of your virtual environment. In a traditional non-virtual environment you monitor performance through the guest operating system which is installed directly on the server hardware. Typically a centralized monitoring system relies on an agent installed on the guest OS or built-in components like Windows WMI to read performance statistics from the server. With virtualization this type of performance monitoring is no longer effective; the reason is the guest operating system is no longer seeing the physical hardware of the host. Instead it is seeing virtual hardware that is emulated by the hypervisor so performance statistics that are measured inside the guest OS are not an accurate reflection of the physical hardware of the host. As a result you need a monitoring application that is aware of the virtualization layer and can also measure the statistics that are unique to virtual environments.

Virtualization built-in performance monitoring tools like VMware’s vCenter Server can provide raw performance statistics for the virtual environment but doesn’t help you interpret them. The information returned by vCenter Server can be overwhelming and knowing what to look for and what the numbers mean can be difficult. Additionally vCenter Server is designed to mainly monitor and report at the virtualization layer and doesn’t extend to far into the guest OS layer so it does not provide a complete monitoring solution. There are hundreds of performance statistics that are generated by ESX/ESXi and vCenter Server that cover many different areas. Not all of these statistics are useful in most cases and if you tried to monitor them all you would be quickly overwhelmed. Some statistics are only useful in certain situations such as troubleshooting a resource bottleneck but there are others that can provide key indicators to the overall health of your vSphere environment and should be constantly monitored. Some statistics are specific to hosts and others only apply to virtual machines, the below table lists some of the more important statistics that you should focus on when monitoring vSphere.

Full paper including the table detailing key performance metrics available here…

Share This:

Nov 14 2011

Top 5 Things You Need in a Virtualization Management Solution

I recently completed a series of technical papers for SolarWinds that highlight some of the challenges associated with virtualization. The first paper was titled “Top 5 Things You Need in a Virtualization Management Solution” and served as an introductory paper to the subsequent papers that cover the 5 things in more detail. Here are the 5 things that are covered:

  • Performance Management
  • Capacity Planning
  • VM Sprawl
  • Chargeback and Showback
  • Storage I/O Bottlenecks

You might be used to vendor white papers that are focused on marketing and sales information and not much technical content. That’s not my writing style, my goal when writing white papers is to have at least 80% of the content be useful and educational content while still highlighting the vendors products in a more subtle manner.  Since my name is on the paper I want the reader to come away with both a thorough understanding of the topic from a technical perspective and also an understanding on how the vendor’s product relates to the topic. Below is an excerpt from the first paper, you can register and read the full paper over at SolarWinds website.

5-areas-3

For any size data center, having a proper management solution is critical for ensuring an orderly, smooth running and problem free environment. Without one, your data center can quickly turn into a wild jungle with servers growing out of control without any regard for discipline or limits. A data center should be like a well-run city, with all parts of it communicating and interacting with each other. The mayor of that city should know about everything that happens within it so he can make the right decisions when needed to ensure everything runs smoothly. It shouldn’t be like a group of isolated villages that never speak to each other and have no idea what everyone else is doing. Having visibility into all aspects of your server environment is critical, but the amount of information that is monitored can easily be overwhelming, and not knowing how to interpret the information can make dealing with it difficult. Therefore, you need a management solution that can highlight the important information for you and also provide you with dashboards to make interpreting the information as easy as possible.

Virtualization technology has many benefits and most companies are now implementing it or planning to at some point. But the management of virtualized environments can be quite different from traditional physical environments, and the solutions designed to manage physical environments will not be as effective at managing a virtual environment. Therefore, implementing a management solution that is designed specifically for virtual environments is critical to ensure that you are monitoring the metrics and issues that are unique to virtual environments, and are able to keep up with the higher rate-of-change and can scale as your virtual environment grows.
In this white paper, we will look at the top 5 areas that you should be looking at when choosing a management solution for your virtual environment, and also cover the reasons why you need to use management solutions that are designed specifically for virtualization.

Full paper available here…

Share This:

Nov 07 2011

Top blog voting coming up soon!

The polls will open next week for the annual VMware/virtualization top blog voting, so if you want to make sure your site is included make sure I have your blog listed on my vLaunchPad. This year will be a bit different, instead of just a top 25 we’ll also have categories kind of like they do at many awards shows. Some of the categories will be Best New Blog, Best Storage Blog, Blogger You Most Want To Meet, etc. I’m still playing with the categories so sound off in the comments if you have any ideas.

Share This: