Category: News

One year ago vSphere 6.0 was released and along with it came the new Virtual Volumes (VVols) storage architecture for shared storage. Many years in the making VVols revolutionized how storage arrays interact with VMs going from the archaic LUN-centric model to a new VM-centric model. VMware just published a blog post on this 1 year milestone for VVols and I thought I would comment and provide my perspective as well.

In a recent blog post VMware published some stats on VVol deployment that are supposedly based on real-life customer adoption but I don’t feel their numbers provide much meaningful information on how customers are using VVols. To be honest their numbers don’t make sense to me, they stated the following in their blog post:

“The median number of datastores deployed by a customer using VVols is two, serving on average just over 30TB of capacity with about half of that consumed by Virtual Machines.”

VVols doesn’t use datastores, it uses Storage Containers which is a purely logical entity as their is no physical storage assigned to a Storage Container. In most cases you are only going to use a single Storage Container as their isn’t too much benefit to using multiple storage containers besides logically separating VMs on a storage array. So when they say 2 datastores that’s confusing, again there are no datastores, are they referring to Storage Containers, if so I would think more people are using one then 2.

Next on capacity, with VVols their is no over-provisioning like you do with VMFS so their is no wasted space, you are using exactly the amount of space that your VMs need. So to say that you have 30TB of capacity allocated to VVols with only half that used by VMs doesn’t add up. Your VVol capacity should exactly equal the capacity used by your VMs on VVol storage, again this is confusing.

One metric that they don’t provide which I feel would be the most useful is the average numbers of VVols associated with a VM. This would be a key metric that would help determine how many VMs an array could support based on vendor max VVol limitations. This will vary by customer largely based on snapshot usage but I’m always curious to see what customers are averaging with this.

They mention VVol use cases and not being sure how customers are using them, I think at this point it is more dev/test, low hanging fruit apps that aren’t mission critical as most people are still testing the VVol waters right now before going all in. They also mention the benefits and highlight snapshots, I agree this is a big one along with space reclamation and no more over-allocating space. The new snapshot mechanism that VVols uses will have a big impact on backup efficiency as I highlighted in this post.

They talk about the partner ecosystem which is still a work in progress, they show a lot of partner logos but only 14 of them support VVols after one year, I did an update on that a few weeks ago. It’s nice to see VVols hit the one year milestone and start to mature and to see customer interest and adoption in VVols grow. I think it will take another year and another big vSphere release though before it gains good momentum much the same way that VSAN did (it turned 2 years old this month). For more on my perspective on customer adoption of VVols you can read this post I did a few months back.

[important]Update below from Ben Meadowcroft at VMware that clarifies this, author of the blog post and product manager, thanks Ben!:[/important]

I appreciate the write up and commentary on my blog post. There are a couple of items I’d like to clarify and add some context to if I may:

> VVols doesn’t use datastores, it uses Storage Containers

Storage Containers are the array side construct but within vSphere the storage container is exposed and consumed as a datastore. When you want to expose a storage container to vSphere you are in fact adding a new datastore to the inventory.

> In most cases you are only going to use a single Storage Container as their isn’t too much benefit to using multiple storage containers besides logically separating VMs on a storage array. …if so I would think more people are using one then 2.

I agree that the primary reason why you would want to do this are logical separation, perhaps taking advantage of vSphere permissions to restrict access to specific datastores to different users for example. This is why it is important to note that a storage container is exposed as a datastore within vSphere and so you get all the permissions capabilities that you previously had with the new storage model as well. Another (and probably more likely) explanation would simply be that some customers are following old habits, perhaps trying VVols on a couple of clusters and deciding to carve it out as a container per cluster say.

> Next on capacity, with VVols their is no over-provisioning like you do with VMFS so their is no wasted space, you are using exactly the amount of space that your VMs need. So to say that you have 30TB of capacity allocated to VVols with only half that used by VMs doesn’t add up. Your VVol capacity should exactly equal the capacity used by your VMs on VVol storage, again this is confusing.

This is a good point, there is a difference between the consumption on the array with VMFS and VVols that is an important consideration with VVols. The numbers from my blog are taken from what is reported by the datastore metrics. While the storage container is a logical entity it does have an associated capacity value (that is reported via VASA and reported against the datastore metrics received by VMware). It is this capacity that is picked up in the reports in the post. You are correct to highlight that the VVol storage container is not consumed at configuration time as a LUN with VMFS would be. I consider the capacity associated with the storage container to be more like a logical limit on the capacity that can be consumed by the container (and as it is logical it can be adjusted). I apologize that this was confusing, I can see how it could have been communicated more clearly.

Thanks again for your write-up, your article are a great resource and always an interesting read.

I came across this one a few weeks ago and thought it was interesting. In a nutshell one the biggest contributors to the Linux kernel (Christoph Hellwig) is trying to sue VMware over GPL (General Public License) violations from VMware taking the Linux kernel and modifying it for use with their own ESXi vmkernel. As you may know VMware has longed used the Linux operating system in both the ESX & ESXi hypervisors. ESX used a almost full-blown Linux install as part of the Service Console and ESXi used a smaller and streamlined Linux shell called BusyBox that serves as the console for ESXi. VMware essentially took the Linux code and modified it into their own module that they call vmklinux.

Well one Linux developer didn’t like what VMware was doing with the Linux code that they were using in ESXi and tried to get them to comply with the terms of the GPL licensing for Linux. VMware attempted to comply in some areas and refused in others which prompted the lawsuit. This diagram below from the Software Freedom Conservancy is a good depiction of how VMware blended Linux with their own vmkernel:

From reading through the documents that are published it’s not completely clear what they are wanting from VMware. It looks like they are trying to get VMware to publish the complete source code for vmklinux or quit using it entirely. Apparently VMware has requested all court documents not be released to the public as it contains many examples of their source code. It appears that they have a pretty strong case against VMware but I can see this one dragging on for many years and by that time VMware may have dumped the Linux code for something else that they developed. You can read about the latest court updates that occurred a few weeks ago in this blog post from someone who attended it. You can read the whole FAQ on the case from the SF Conservancy that provides a good overview of the case and is an interesting read.

VMware has been in the spotlight a lot recently in regards to some business level news about the state of the company. On one hand you see them declared one of the best companies to work for, but on the other hand you see that they are laying off people and there seems to be a leadership revolving door. In this post I wanted to give my thoughts and perspective on this news about VMware.

Before I begin I want to point out how many years ago many people were stating that VMware might be the next Novell who’s Netware product  was the dominate player in the user networking/file sharing realm. I personally worked with a lot of Netware installations back in the day. The other players in that biz were Linux and OS/2, and of course Microsoft Windows (NT/Server). The major threat to Netware was Windows and by 2000, Novell was in major decline and eventually faded into tech obscurity.

The similarities between VMware and Novell are mainly that Microsoft is the biggest threat to their business and they were both market leaders by a large margin, but that’s where the similarities end as VMware has done an excellent job adapting and evolving and not becoming a one trick pony like Novell largely was. Despite the claims that Hyper-V would eventually crush vSphere as it was embedded into Windows, that hasn’t happened and VMware still enjoys a very big slice of the hypervisor market share and I don’t see that declining very much in the future.

Let’s start with VMware being named one of the best places to work by Fortune magazine. Not sure what the process is for picking that is, VMware is listed as #40 (Google is #1), I have never worked for VMware but from what I’ve seen and heard I think that is well deserved. They seem to treat their employees the right way, have a wonderful campus and people I know genuinely seem like they love working there. It’s nice to be recognized on this list but it really doesn’t have much bearing on how successful their business is. On a side note I was actually pictured in Fortune magazine back in 1995 when they did best states to work in before they picked the best companies to work at. That’s me front and center below.

So if VMware is such a great company to work for why are so many executives abandoning ship? I’m sure they all had their own reasons but one of the biggest influences is undoubtedly the upcoming EMC-Dell merger. When companies merge or are bought out, management is usually the first thing that gets scrutinized and cut. If I was on the management team at EMC or VMware I would definitely be nervous and maybe proactively looking for greener pastures with better job security. It also doesn’t help that EMC has been a constant nagging parent to VMware which makes VMware management constantly prone to scrutiny and second guessing.

So management is a bit unstable at VMware, is that such a big deal though, what’s most important as John Troyer point’s out in his latest Tech Reckoning Dispatch is that the smart techies that are the worker bees at VMware are staying put. That’s whats most important to a company, their intellectual property, the management ladder will eventually sort itself out, the VMware ship is heading in the right direction and their are still plenty of good management types there. Personally I think the management revolving door is more FUD and less a legitimate concern. However I think VMware would of thrived a lot more had they remained a truly independent company without being owned by any other company.

So has VMware jumped the shark? Maybe, maybe not, I think their glory days are definitely over and it’s reality check and tighten the belts time at VMware. VMware recently announced their first ever large scale lay-offs, to me that’s not a big concern, it’s just them right-sizing the business as they prepare for tougher times. Microsoft still remains a legitimate threat along with other big players and of course the whole OpenStack movement has gained momentum. VMware definitely has it a lot tougher today then they did 5 years ago and one big reason for that is that almost all of their once valued technology partners are now mainly competitors to VMware. Back when VMware only sold a hypervisor product their partners built a very strong ecosystem around them which made for a powerful combination. Now that VMware has extended their line-up into just about every area including networking, storage, management, data protection, cloud, automation, etc. there is a lot more friction between VMware and it’s partners and of course competition.

VMware may have some tough days ahead of it but they have proven very resilient and will persevere and remain a dominate player. VMware is a software company, there overhead is mainly employees which is also their best asset. Once the acquisition process is over and management solidifies I think you’ll see VMware stay strong and remain focused without any chance of becoming a footnote in tech history like once dominate companies such as Novell, NetScape and Atari eventually became. Where VMware once swam in a mostly empty sea, they now swim in a sea full of sharks and only time will truly tell but my money is on VMware staying afloat and remaining strong and relevant.

I’ve seen Craigslist used for many different things but this is the first time I’ve ever seen someone try and get VSAN tech support on Craigslist. I can only imagine the responses they will get. While you’re there it’s always fun to read the Craigslist Best Of posts…

I was browsing through the latest docs in the VMware KB this week and noticed one describing the end of availability of vSphere Enterprise, vSphere with Operations Manager (vSOM) Standard and Enterprise edition. That got me to browsing through VMware’s product lifecycle matrix which provides dates for every VMware products on GA, end of availability, end of support and end of technical guidance. Going through the huge list (VMware has so many products past and present), these are the key ones listed that are ending support in 2016:

• VMware App Volumes 2.5, 2.6, 2.7 – 2016/12/09
• VMware Data Recovery 2.0 – 2016/08/24
• VMware ESXi 5.0 and 5.1 – 2016/08/24
• VMware EVO:RAIL 1.0 and 1.1 – 2016/09/09
• VMware EVO:RAIL 1.2 – 2016/12/10
• VMware Fusion 7.x – 2016/03/03
• VMware NSX for vSphere 6.1 – 2016/10/15
• VMware vCenter Log Insight 2.0 – 2016/06/10
• VMware vCenter Operations 5.8.5 – 2016/12/31
• VMware vCenter Server 5.0 and 5.1 – 2016/08/24
• VMware vCenter Site Recovery Manager 5.0 and 5.1 – 2016/08/24
• VMware vCenter Update Manager 5.0 and 5.1 – 2016/08/24
• VMware vCloud Automation Center 5.2 – 2016/12/10
• VMware vCloud Networking and Security 5.5 – 2016/09/19
• VMware vRealize Operations 6.0 and 6.1 – No earlier than 2016/12/09
• VMware vSphere Data Protection 5.1 – 2016/08/24
• VMware vSphere Replication 5.1 – 2016/08/24
• VMware Workstation 11.x – 2016/06/02

Again these are the end of support dates for these products which means if you have SnS, VMware offers maintenance updates and upgrades, bug/security fixes and technical assistance until these dates. After these dates they have typically have a technical guidance phase which lastes for a fixed duration and provides support through their self-help portal only (no telephone support), they do not offer new hardware support, server/client/guest OS updates, new security patches or bug fixes during this phase. You can read more about these different product support phases here. So for those of you still on vSphere 5.0 & 5.1, the clock is ticking, better start planning your upgrades.

Date anomalies such as Y2K, Leap Years, and the infamous1970 bug can be disruptive if software code is not designed to handle them properly. You might remember the infamous VMware time bomb bug a few years back which made them look bad. In 2016 we once again have a Leap Year and VMware has gone out and tested the impact of that date across their entire product line to ensure nothing is impacted by this once every four year date occurrence. VMware has published the results of their testing in a KB article which appears to cover all of their products and which are unaffected by Leap Year. I expect they didn’t test every past version of these products but as Leap Year isn’t really new and occurs every four years I don’t expect you’ll see any issues regardless of which version of a VMware product you are on. So with this reassurance in mind hopefully you will sleep better Sunday night and not get woken up by any surprises.

VMware’s new Virtual Volumes storage architecture required some changes to the SCSI specifications for block arrays to support the new sub-LUN model that VVols uses. Because of this it required I/O vendors to update their software to be able to support the new Protocol Endpoint component in VVols that is used as the data path between hosts and VMs (VVols) residing on storage arrays. Early on after VVols was released many I/O vendors were in catch-up mode to introduce support for this into their I/O adapter firmware. As a result there is a good chance that the I/O adapter that you are using may not work with VVols or may require a firmware upgrade to support it.

Let’s first look at what changed with VVols that required changes to be made to I/O adapter firmware to support VVols. A traditional block I/O adapter connects to a LUN on a storage array where your VMFS datastore is located. To connect to the LUN it simply needs the data path information (WWN) which includes HBA #, controller and of course the LUN ID of the volume associated with the VMFS datastore. You’ll see this in the vSphere client when selecting an I/O adapter with a syntax similar to vmhbaAdapter:CChannel:TTarget:LLUN (i.e. vmhba1:C0:T3:L1).

VVols introduces the concept of Secondary LUN IDs which is essentially an additional layer of LUN numbering that supports a lot more sub-LUNs than a array traditionally supports. The way this works is that a host will connect to a special Administrative LUN on the storage array via the Protocol Endpoint. This Admin LUN has no storage allocated to it and serves as a gateway to the sub-LUNs beneath it, it’s LUN ID is usually greater than 255 to identify it as a non-data LUN. A host cannot connect directly to a sub-LUN and must go through the Admin LUN to get to it. These Secondary LUN ID’s are provided to a host via the VASA Provider, so you can see why it is an important component. You can also see why direct to SAN backup is not supported with VVols as you cannot connect to a sub LUN without going through a ESXi host. The relationship between these components is outlined in the figure below:

The architecture is a bit complex and introduces additional identifiers beyond the LUN ID to connect to VVols on the array. These additional identifiers include secondary level IDs (SLLID), VVol IDs and Reference counts, the PE LUN still has a traditional WWN associated with it. Now while both block and NFS arrays utilize protocol endpoints and SLLIDs, the special Admin (PE) LUN only applies to block storage arrays, if you are using a NFS array with VVols this doesn’t apply as their is no special PE LUN.  With block arrays PEs are discovered via an in-band path using the standard SCSI command, REPORT_LUNS which reports the WWN of the PE, with NFS PEs are discovered via an out-of-band path using an API which returns the IP address and mount point of the PE. PE LUNs on block storage are recognized differently from traditional data LUNs as they have a special conglomerate bit set (LU_CONG).

VMware has patented this new storage architecture that they refer to as “Computer system accessing object storage system“, here are a few diagrams from the patent, if you want to deep-dive into this new architecture give the patent a read.

This new sub LUN architecture required VMware to submit update proposals to the T-10 committee for SCSI specifications to support it and the end result of all this was I/O adapters had to be updated to support this as well.

So know we know the why, let’s look at the how, as in how do I know if my I/O adapter will support VVols? Perhaps the quickest and easiest way is to simply look it up on the VMware HCL. If you go the VMware HCL and select I/O Adapters you will notice a new Feature there that you can select that is called “Secondary LUNID”. You can simply select that feature and select your I/O Adapter brand name, optionally the device type (i.e. FC) and then search as shown below.

In the results you will see the firmware versions needed to support VVols and links to any special drivers that might be needed for ESXi to support it as shown below.

From what I’ve seen there are some I/O Adapters that will work using the standard ESXi image, some require custom server ESXI images (i.e. HPE, Dell) and other that require you to download and install a driver into ESXi. You can see many of these in the vSphere Drivers/Tools download page.

One thing to note is if your I/O Adapter does not support VVols or does not have the firmware to support it you will see this error in your vmkernel logs:

Sanity check failed for path vmhbaX:Y:Z. The path is to a VVol PE, but it goes out of adapter vmhbaX which is not PE capable. Path dropped.

To check if your HBA is VVol capable you can run this command on your ESXi hosts: esxcli storage core adapter list , you should see Second Level Lun ID (SLLID) listed under the Capabilities column if the I/O Adapter supports VVols as shown below, VMware has a KB on this as well.

Once you are sure your I/O Adapter has the required firmware and the proper ESXi I/O driver is present you can check from the host to see if it is able to recognize the Protocol Endpoint (Admin LUN). You’ll need to make sure that your VASA Provider is enabled and setup before you can do this though. Once that is setup you can use the esxcli command run from each host to see if the PE is visible to it, the syntax for the command is esxcli storage core device list –pe-only as shown below.

In the output that is produced if you see a value of “true” for “Is VVOL PE” that confirms that the host is able to connect successfully to the array PE to access VVols. Often times this will be false if your I/O adapter does not support VVols or VVols is not setup correctly.

So there you have it, it may seem a bit complicated but there is a good chance that your I/O Adapter already supports VVols and you really won’t have to do anything to start using it. It’s always a good thing to check though and make sure your I/O Adapters and storage arrays have the required firmware level to support VVols. Hopefully this gives you a better understanding of how VVols works under the hood and the relationships between hosts, protocol endpoints and VVols.