Storage is the slowest and most complex host resource, and when bottlenecks occur, they can bring your virtual machines (VMs) to a crawl. In a VMware environment, Storage I/O Control provides much needed control of storage I/O and should be used to ensure that the performance of your critical VMs are not affected by VMs from other hosts when there is contention for I/O resources.
Storage I/O Control was introduced in vSphere 4.1, taking storage resource controls built into vSphere to a much broader level. In vSphere 5, Storage I/O Control has been enhanced with support for NFS data stores and clusterwide I/O shares.
Prior to vSphere 4.1, storage resource controls could be set on each host at the VM level using shares that provided priority access to storage resources. While this worked OK for individual hosts, it is common for many hosts to share data stores, and since each host worked individually to control VM access to disk resources, VMs on one host could limit the amount of disk resources on other hosts.
The following example illustrates the problem:
- Host A has a number of noncritical VMs on Data Store 1, with disk shares set to Normal
- Host B runs a critical SQL Server VM that is also located on Data Store 1, with disk shares set to High
- A noncritical VM on Host A starts generating intense disk I/O due to a job that was kicked off; since Host A has no resource contention, the VM is given all the storage I/O resources it needs
- Data Store 1 starts experiencing a lot of demand for I/O resources from the VM on Host A
- Storage performance for the critical SQL VM on Host B starts to suffer as a result
How Storage I/O Control works
Storage I/O Control solves this problem by enforcing storage resource controls at the data store level so all hosts and VMs in a cluster accessing a data store are taken into account when prioritizing VM access to storage resources. Therefore, a VM with Low or Normal shares will be throttled if higher-priority VMs on other hosts need more storage resources. Storage I/O Control can be enabled on each data store and, once enabled, uses a congestion threshold that measures latency in the storage subsystem. Once the threshold is reached, Storage I/O Control begins enforcing storage priorities on each host accessing the data store to ensure VMs with higher priority have the resources they need.
Read the full article at searchvirtualstorage.com…
Today I wanted to highlight another white paper that I wrote for SolarWinds that is titled “Storage I/O Bottlenecks in a Virtual Environment”. I enjoyed writing this one the most as it digs really deep into the technical aspects of storage I/O bottlenecks. This white paper covers topics such as the effects of storage I/O bottlenecks, common causes, how to identify them and how to solve them. Below is an excerpt from this white paper, you can register and read the full paper over at SolarWinds website.
There are several key statistics that should be monitored on your storage subsystem related to bottlenecks but perhaps the most important is latency. Disk latency is defined as the time it takes for the selected disk sector to be positioned under the drive head so it can be read or written to. Once a VM makes a read or write to its virtual disk that request must follow a path to make its way from the guest OS to the physical storage device. A bottleneck can occur at different points along that path, there are different statistics that can be used to help pinpoint where the bottleneck is occurring in the path. The below figure illustrates the path that data takes to get from the VM to the storage device.
The storage I/O goes through the operating system as it normally would and makes its way to the device driver for the virtual storage adapter. From there it goes through the Virtual Machine Monitor (VMM) of the hypervisor which emulates the virtual storage adapter that the guest sees. It travels through the VMkernel and through a series of queues before it gets to the device driver for the physical storage adapter that is in the host. For shared storage it continues out the host on the storage network and makes its way to its final destination which is the physical storage device. Total guest latency is measured at the point where the storage I/O enters the VMkernel up to the point where it arrives at the physical storage device.
The total guest latency (GAVG/cmd as it is referred to in the esxtop utility) is measured in milliseconds and consists of the combined values of kernel latency (KAVG/cmd) plus device latency (DAVG/cmd). The kernel latency includes all the time that I/O spends in the VMkernel before it exits to the destination storage device. Queue latency (QAVG/cmd) is a part of the kernel latency but also measured independently. The device latency is the total amount of time that I/O spends in the VMkernel physical driver code and the physical storage device. So when I/O leaves the VMkernel and goes to the storage device this is the amount of time that it takes to get there and return. A guest latency value that is too high is a pretty clear indication that you have a storage I/O bottleneck that can cause severe performance issues. Once total guest latency exceeds 20ms you will notice the performance of your VMs suffer, as it approaches 50ms your VMs will become unresponsive.
Full paper including information on the key statistics related to storage I/O bottlenecks available here…
Implementing a virtual desktop infrastructure (VDI) involves many critical considerations, but storage may be the most vital. User experience can often determine the success of a VDI implementation, and storage is perhaps the one area that has the most impact on the user experience. If you don’t design, implement and manage your VDI storage properly, you’re asking for trouble.
VDI’s impact on storage
The biggest challenge for storage in VDI environments is accommodating the periods of peak usage when storage I/O is at its highest. The most common event that can cause an I/O spike is the “boot storm” that occurs when a large group of users boots up and loads applications simultaneously. Initial startup of a desktop is a very resource-intensive activity with the operating system and applications doing a lot of reading from disk. Multiplied by hundreds of desktops, the amount of storage I/O generated can easily bring a storage array to its knees. Boot storms aren’t just momentary occurrences — they can last from 30 minutes to two hours and can have significant impact.
After users boot up, log in and load applications, storage I/O typically settles down; however, events like patching desktops, antivirus updates/scans and the end-of-day user log off can also cause high I/O. Having a data storage infrastructure that can handle these peak periods is therefore critical.
Cost is another concern. The ROI with VDI isn’t the same as server virtualization, so getting adequate funding can be a challenge. A proper storage infrastructure for VDI can be very costly, and to get the required I/O operations per second (IOPS) you may have to purchase more data storage capacity than you’ll need.
Expect to spend more time on administration, too. Hundreds or thousands of virtual disks for the virtual desktops will have to be created and maintained, which can be a difficult and time-consuming task.
Read the full article in the March 2011 issue of Storage Magazine…
Server and desktop virtualization have provided relatively easy ways to consolidate and conserve, allowing a reduction in physical systems. But these technologies have also introduced problems for data storage managers who need to effectively configure their storage resources to meet the needs of a consolidated infrastructure.
Server virtualization typically concentrates the workloads of many servers onto a few shared storage devices, often creating bottlenecks as many virtual machines (VMs) compete for storage resources. With desktop virtualization this concentration becomes even denser as many more desktops are typically running on a single host. As a result, managing storage in a virtual environment is an ongoing challenge that usually requires the combined efforts of desktop, server, virtualization and storage administrators to ensure that virtualized servers and desktops perform well. Here are 10 tips to help you better manage your storage in virtual environments.
Read the full article at searchstorage.com…
You can use VMware vSphere without a shared storage device, but it limits the amount of advanced features that you can use with it. Certain features in vSphere require that a virtual machine (VM) reside on a shared storage device that is accessible by multiple hosts concurrently. These features include high availability (HA), Distributed Resource Scheduler (DRS), Fault Tolerance (FT) and VMotion, which provide high/continuous availability as well as workload load balancing and live migration of virtual machines. For some storage administrators, these features may only be nice to have, but they are also essential for many IT environments that cannot afford to have VMs down for an extended amount of time.
A few years ago, VMware shared storage typically meant using a Fibre Channel (FC) SAN, which was expensive, required specialized equipment and was complicated to manage. In recent years, other shared storage options that utilize standard network components to connect to storage devices have become popular and make for affordable, easy-to-use shared storage solutions. The protocols used for this are iSCSI and NFS, both of which are natively supported in vSphere. The performance of NFS and iSCSI are similar, but both can vary depending on a variety of factors including the data storage device characteristics, network speed/latency and host server resources. Since both protocols use software built into vSphere to manage the storage connections over the network there is some minimal CPU resource usage on the host server as a result.
Read the full article at searchsmbstorage.com…
In the last week I’ve gotten some new storage devices, both at work and at home. Unfortunately I’ve experienced problems with both and its not been as fun of a week as I would of liked. The new work storage device is a HP MSA-2312i which is the iSCSI version of their Modular Storage Array line.
The new home storage device is an Iomega ix4-200d 4TB which is a relatively low cost network storage device that supports iSCSI & NFS and much more.
The MSA problems have all been firmware related, basically it kept getting stuck in a firmware upgrade loop, if you own one or plan on buying one I would say don’t upgrade the firmware unless you have a reason to and if you do make sure you schedule downtime. I’ll be sharing some tips for upgrading the firmware on that unit later on.
The Iomega problems are from a flaky hard drive presumably, not long after I plugged the unit in and started configuring it I received the message that drive 3 was missing. After talking to support and rebuilding the RAID group the problem briefly went away and then came right back. They graciously waived the $25 replacement fee (it’s brand new, they better!) and refused to expedite the shipping unless I paid $40 (again, its brand new, you would think they would want to make a new customer happy). Having a flaky drive in a brand new unit doesn’t exactly inspire confidence in storing critical data on the device so I’ll have to see how it goes once the drive is replaced.
So look forward to some upcoming posts on using and configuring both devices. The MSA will be used as part of a Domino virtualization project and I’ll be doing performance testing on it in various configurations. The Iomega I’ll be using with VMware Workstation 7 on my home computer as both iSCSI & NFS datastores.
General
Everything you need to know about vSphere and data storage (Storage Magazine)
10 tips for managing storage for virtual servers and virtual desktops (Storage Magazine)
What’s New in VMware vSphere 4.1 - Storage (VMware)
Storage IO Control Technical Overview & Considerations for Deployment (VMware)
vSphere Storage: Features and Enhancements (Professional VMware)
Storage Basics - Part I: An Introduction (VM Today)
Storage Basics - Part II: IOPS (VM Today)
Storage Basics - Part III: RAID (VM Today)
Storage Basics - Part IV: Interface (VM Today)
Storage Basics - Part V: Controllers, Cache and Coalescing (VM Today)
Storage Basics - Part VI: Storage Workload Characterization (VM Today)
Storage Basics - Part VII: Storage Alignment (VM Today)
vSphere and 2TB LUNs - changes from VI3.x (Virtual Geek)
A look at the ESX I/O stack (NetApp)
vSphere on NetApp Storage Best Practices (Netapp)
Storage Protocol Choices & Storage Best Practices for vSphere (EMC)
Improve Storage Efficiency and Management with VMware vSphere 4 (VMware)
What Is New in VMware vSphere 4: Storage (VMware)
VMware vSphere 4: Exchange Server on NFS, iSCSI, and Fibre Channel (VMware)
Data Compression, Deduplication, & Single Instance Storage (Virtual Storage Guy)
vSphere Introduces the Plug-n-Play SAN (Virtual Storage Guy)
A look at the ESX I/O stack (NetApp)
Storage Oversubscription Technologies (Xtravirt)
The Debate-Why NFS vs Block Access for OS/Applications (vTexan)
Fiber Channel over Ethernet (FCoE)
“FCoE vs. iSCSI - Making the Choice” from Interop Las Vegas 2011 (Stephen Foskett)
How FCoE and iSCSI Fit into Your Storage Strategy (NetApp Tech OnTap)
Fibre Channel over Ethernet in the Data Center: An Introduction (Cisco)
Solid-State Disks (SSDs)
Making the case for solid-state storage (Storage Magazine)
vStorage APIs
VAAI
VAAI sweetness (Yellow Bricks)
vStorage APIs for Array Integration aka VAAI (Yellow Bricks)
VMware VAAI pros and cons and the hidden fourth primitive (SearchVMware.com)
vStorage APIs for Array Integration (VAAI) (VMTN)
vSphere 4.1 - What do the “vStorage APIs for Array Integration” mean to you? (Virtual Geek)
vSphere 4.1 and vStorage APIs for Array Integration (VAAI) (Virtual Storage Guy)
VMware vSphere VAAI Demo with NetApp (Virtual Storage Guy)
VAAI and Deployment - a Practical Example (NTPro.nl)
Exploring the performance benefits of VAAI (The Lower Case w)
VMware vSphere 4.1 vStorage APIs for Array Integration (VAAI) understanding (GeekSilver)
vStorage APIs for Array Integration FAQ (VMware KB)
3PAR and VMware go back to Storage’s Roots to Enhance the Future of Virtualized Data Centers (DCIG)
Using VMware vStorage APIs for Array Integration with EMC Symmetrix VMAX (EMC)
Chad’s Choice - vStorage API for Array Integration: What is it; What to expect. (EMC Webcast)
VMware VAAI Storage Array Support in Plain English (Stephen Foskett)
VAMP
VMware PSP and SATP in Plain English (Stephen Foskett)
Configure VMware ESX(i) Round Robin on EMC Storage (boche.net)
What’s that ALUA exactly? (Yellow Bricks)
Pluggable Storage Architecture, exploring the next version of ESX/vCenter (Yellow Bricks)
A couple important (ALUA and SRM) notes (Virtual Geek)
Understanding more about NMP RR and iooperationslimit=1 (Virtual Geek)
vSphere Introduces the Plug-n-Play SAN (Virtual Storage Guy)
VMware PSA, MPP, NMP, PSP, MRU, … And Tutti Quanti! (DeinosCloud)
Best practices for HP EVA, vSphere 4 and Round Robin multi-pathing (Ivobeerens.nl)
vSphere Round Robin MultiPathing (Phil the Virtualizer)
Multipathing policies in ESX 4.x (VMware KB)
iSCSI
A “Multivendor Post” on using iSCSI with VMware vSphere (Virtual Geek)
A “Multivendor Post” to help our mutual iSCSI customers using VMware (Virtual Geek)
Using iSCSI storage with vSphere (Storage Magazine)
Configuring VMware vSphere Software iSCSI with Dell Equallogic PS Series Storage (Equallogic)
How to Configure Openfiler iSCSI Storage for VMware ESX 4 (Xtravirt)
Putting your storage to the test - Part 1 iSCSI on Iomega IX4-200D (Gabe’s Virtual World)
How-to connect ESX4, vSphere to Openfiler iSCSI NAS (Vladan.fr)
EMC Virtual Infrastructure for Exchange 2007 using vSphere 4.0 and iSCSI (EMC)
How to setup basic software iSCSI for VMware vSphere (video)
iSCSI Design Considerations and Deployment Guide (VMware)
Converged Storage Infrastructure for VMware vSphere 4.1 (Broadcom)
NFS
A “Multivendor Post” to help our mutual NFS customers using VMware (Virtual Geek)
Using NAS for virtual machines (Storage Magazine)
Putting your storage to the test Part 2 NFS on Iomega IX4-200D (Gabe’s Virtual World)
Best Practices for Running vSphere on NFS Storage (VMware)
Republished: Dispelling Some VMware over NFS Myths (Scott Lowe)
Reasons For Using NFS With VMware Virtual Infrastructure (VM/ETC)
SAN/Fiber Channel
SAN System Design and Deployment Guide (VMware)
Configuring and Troubleshooting N-Port ID Virtualization (VMware)
Thin Provisioning
Dynamic Storage Provisioning (VMware)
VMware ESX 4 Reclaiming Thin Provisioned disk Unused Space (Virtualization Team)
Performance
vStorage: Troubleshooting Performance (Professional VMware)
Calculate IOPS in a storage array (Tech Republic)
IOPs? (Yellow Bricks)
Storage System Performance Analysis with Iometer (VMware)
Benchmarking Storage for VMware (Peacon)
Performance Troubleshooting VMware vSphere - Storage (Virtual Insanity)
NetApp TR-3808 - VMware vSphere and ESX 3.5 Multiprotocol Performance Comparison Using FC, iSCSI, and NFS (NetApp)
VMTN Storage Performance Thread and the EMC Celerra NS-120 (Virtualization Evangelist)
Comparing the I/O Performance of 2 or more Virtual Machines SSD, SATA & IOmeter (Vinf.net)
Poor performance and high disk latency with some storage configurations (VMware KB)
pvSCSI
More Bang for Your Buck with PVSCSI (Part 1) (Virtual Insanity)
More Bang for your Buck with PVSCSI (Part 2) (Virtual Insanity)
Boot from Paravirtualized SCSI Adapter (Xtravirt)
Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters (VMware KB)
VDI
Managing storage for virtual desktops (Storage Magazine)
VDI and Storage: Deep Impact (Virtuall.eu)
VDI User Sizing Example (EMC)
Vendor Specific
HP
Configuration best practices for HP StorageWorks Enterprise Virtual Array (EVA) family and VMware vSphere 4 (HP)
