If you’re going to VMworld this year be sure and check out my session STO5545 – The Top 10 Things You MUST Know About Storage for vSphere which will be on Tuesday, Aug. 27th from 5:00-6:00 pm. The session was showing full last week but they must have moved it to a larger room as it is currently showing 89 seats available. This session is crammed full of storage tips, best practices, design considerations and lots of other information related to storage. So sign up know before it fills up again and I look forward to seeing you there!
Nov 28 2011
Storage is the slowest and most complex host resource, and when bottlenecks occur, they can bring your virtual machines (VMs) to a crawl. In a VMware environment, Storage I/O Control provides much needed control of storage I/O and should be used to ensure that the performance of your critical VMs are not affected by VMs from other hosts when there is contention for I/O resources.
Storage I/O Control was introduced in vSphere 4.1, taking storage resource controls built into vSphere to a much broader level. In vSphere 5, Storage I/O Control has been enhanced with support for NFS data stores and clusterwide I/O shares.
Prior to vSphere 4.1, storage resource controls could be set on each host at the VM level using shares that provided priority access to storage resources. While this worked OK for individual hosts, it is common for many hosts to share data stores, and since each host worked individually to control VM access to disk resources, VMs on one host could limit the amount of disk resources on other hosts.
The following example illustrates the problem:
- Host A has a number of noncritical VMs on Data Store 1, with disk shares set to Normal
- Host B runs a critical SQL Server VM that is also located on Data Store 1, with disk shares set to High
- A noncritical VM on Host A starts generating intense disk I/O due to a job that was kicked off; since Host A has no resource contention, the VM is given all the storage I/O resources it needs
- Data Store 1 starts experiencing a lot of demand for I/O resources from the VM on Host A
- Storage performance for the critical SQL VM on Host B starts to suffer as a result
How Storage I/O Control works
Storage I/O Control solves this problem by enforcing storage resource controls at the data store level so all hosts and VMs in a cluster accessing a data store are taken into account when prioritizing VM access to storage resources. Therefore, a VM with Low or Normal shares will be throttled if higher-priority VMs on other hosts need more storage resources. Storage I/O Control can be enabled on each data store and, once enabled, uses a congestion threshold that measures latency in the storage subsystem. Once the threshold is reached, Storage I/O Control begins enforcing storage priorities on each host accessing the data store to ensure VMs with higher priority have the resources they need.
Read the full article at searchvirtualstorage.com…
Nov 17 2011
Today I wanted to highlight another white paper that I wrote for SolarWinds that is titled “Storage I/O Bottlenecks in a Virtual Environment”. I enjoyed writing this one the most as it digs really deep into the technical aspects of storage I/O bottlenecks. This white paper covers topics such as the effects of storage I/O bottlenecks, common causes, how to identify them and how to solve them. Below is an excerpt from this white paper, you can register and read the full paper over at SolarWinds website.
There are several key statistics that should be monitored on your storage subsystem related to bottlenecks but perhaps the most important is latency. Disk latency is defined as the time it takes for the selected disk sector to be positioned under the drive head so it can be read or written to. Once a VM makes a read or write to its virtual disk that request must follow a path to make its way from the guest OS to the physical storage device. A bottleneck can occur at different points along that path, there are different statistics that can be used to help pinpoint where the bottleneck is occurring in the path. The below figure illustrates the path that data takes to get from the VM to the storage device.
The storage I/O goes through the operating system as it normally would and makes its way to the device driver for the virtual storage adapter. From there it goes through the Virtual Machine Monitor (VMM) of the hypervisor which emulates the virtual storage adapter that the guest sees. It travels through the VMkernel and through a series of queues before it gets to the device driver for the physical storage adapter that is in the host. For shared storage it continues out the host on the storage network and makes its way to its final destination which is the physical storage device. Total guest latency is measured at the point where the storage I/O enters the VMkernel up to the point where it arrives at the physical storage device.
The total guest latency (GAVG/cmd as it is referred to in the esxtop utility) is measured in milliseconds and consists of the combined values of kernel latency (KAVG/cmd) plus device latency (DAVG/cmd). The kernel latency includes all the time that I/O spends in the VMkernel before it exits to the destination storage device. Queue latency (QAVG/cmd) is a part of the kernel latency but also measured independently. The device latency is the total amount of time that I/O spends in the VMkernel physical driver code and the physical storage device. So when I/O leaves the VMkernel and goes to the storage device this is the amount of time that it takes to get there and return. A guest latency value that is too high is a pretty clear indication that you have a storage I/O bottleneck that can cause severe performance issues. Once total guest latency exceeds 20ms you will notice the performance of your VMs suffer, as it approaches 50ms your VMs will become unresponsive.
Full paper including information on the key statistics related to storage I/O bottlenecks available here…
Mar 21 2011
Implementing a virtual desktop infrastructure (VDI) involves many critical considerations, but storage may be the most vital. User experience can often determine the success of a VDI implementation, and storage is perhaps the one area that has the most impact on the user experience. If you don’t design, implement and manage your VDI storage properly, you’re asking for trouble.
VDI’s impact on storage
The biggest challenge for storage in VDI environments is accommodating the periods of peak usage when storage I/O is at its highest. The most common event that can cause an I/O spike is the “boot storm” that occurs when a large group of users boots up and loads applications simultaneously. Initial startup of a desktop is a very resource-intensive activity with the operating system and applications doing a lot of reading from disk. Multiplied by hundreds of desktops, the amount of storage I/O generated can easily bring a storage array to its knees. Boot storms aren’t just momentary occurrences — they can last from 30 minutes to two hours and can have significant impact.
After users boot up, log in and load applications, storage I/O typically settles down; however, events like patching desktops, antivirus updates/scans and the end-of-day user log off can also cause high I/O. Having a data storage infrastructure that can handle these peak periods is therefore critical.
Cost is another concern. The ROI with VDI isn’t the same as server virtualization, so getting adequate funding can be a challenge. A proper storage infrastructure for VDI can be very costly, and to get the required I/O operations per second (IOPS) you may have to purchase more data storage capacity than you’ll need.
Expect to spend more time on administration, too. Hundreds or thousands of virtual disks for the virtual desktops will have to be created and maintained, which can be a difficult and time-consuming task.
Read the full article in the March 2011 issue of Storage Magazine…
Oct 05 2010
Server and desktop virtualization have provided relatively easy ways to consolidate and conserve, allowing a reduction in physical systems. But these technologies have also introduced problems for data storage managers who need to effectively configure their storage resources to meet the needs of a consolidated infrastructure.
Server virtualization typically concentrates the workloads of many servers onto a few shared storage devices, often creating bottlenecks as many virtual machines (VMs) compete for storage resources. With desktop virtualization this concentration becomes even denser as many more desktops are typically running on a single host. As a result, managing storage in a virtual environment is an ongoing challenge that usually requires the combined efforts of desktop, server, virtualization and storage administrators to ensure that virtualized servers and desktops perform well. Here are 10 tips to help you better manage your storage in virtual environments.
Read the full article at searchstorage.com…
Aug 09 2010
You can use VMware vSphere without a shared storage device, but it limits the amount of advanced features that you can use with it. Certain features in vSphere require that a virtual machine (VM) reside on a shared storage device that is accessible by multiple hosts concurrently. These features include high availability (HA), Distributed Resource Scheduler (DRS), Fault Tolerance (FT) and VMotion, which provide high/continuous availability as well as workload load balancing and live migration of virtual machines. For some storage administrators, these features may only be nice to have, but they are also essential for many IT environments that cannot afford to have VMs down for an extended amount of time.
A few years ago, VMware shared storage typically meant using a Fibre Channel (FC) SAN, which was expensive, required specialized equipment and was complicated to manage. In recent years, other shared storage options that utilize standard network components to connect to storage devices have become popular and make for affordable, easy-to-use shared storage solutions. The protocols used for this are iSCSI and NFS, both of which are natively supported in vSphere. The performance of NFS and iSCSI are similar, but both can vary depending on a variety of factors including the data storage device characteristics, network speed/latency and host server resources. Since both protocols use software built into vSphere to manage the storage connections over the network there is some minimal CPU resource usage on the host server as a result.
Read the full article at searchsmbstorage.com…
Feb 10 2010
In the last week I’ve gotten some new storage devices, both at work and at home. Unfortunately I’ve experienced problems with both and its not been as fun of a week as I would of liked. The new work storage device is a HP MSA-2312i which is the iSCSI version of their Modular Storage Array line.
The new home storage device is an Iomega ix4-200d 4TB which is a relatively low cost network storage device that supports iSCSI & NFS and much more.
The MSA problems have all been firmware related, basically it kept getting stuck in a firmware upgrade loop, if you own one or plan on buying one I would say don’t upgrade the firmware unless you have a reason to and if you do make sure you schedule downtime. I’ll be sharing some tips for upgrading the firmware on that unit later on.
The Iomega problems are from a flaky hard drive presumably, not long after I plugged the unit in and started configuring it I received the message that drive 3 was missing. After talking to support and rebuilding the RAID group the problem briefly went away and then came right back. They graciously waived the $25 replacement fee (it’s brand new, they better!) and refused to expedite the shipping unless I paid $40 (again, its brand new, you would think they would want to make a new customer happy). Having a flaky drive in a brand new unit doesn’t exactly inspire confidence in storing critical data on the device so I’ll have to see how it goes once the drive is replaced.
So look forward to some upcoming posts on using and configuring both devices. The MSA will be used as part of a Domino virtualization project and I’ll be doing performance testing on it in various configurations. The Iomega I’ll be using with VMware Workstation 7 on my home computer as both iSCSI & NFS datastores.