One thing I noticed that was missing from the vSphere 6.5 release which is expected at some point is support for an in-band VVol bind process that a host must perform when a VM is powered on. The current VVol implementation is an out-of-band process which involves the VASA Provider which acts as the control path between vCenter, ESXi hosts and a storage array. In this post I’m going to go into detail about the bind process and why it would be beneficial to be an in-band process.
So first the basics, the concept of binding a VM to it’s disk files is unique to VVols. With VMFS/NFS we didn’t need this process as you have a file system in place that can handle file look-ups and locking. With VVols we don’t have a file system written to the storage array, VMs are written natively to the array as VVol objects which are essentially sub LUNS (block) or files (NFS). There still is a mini VMFS file system with VVols but it is contained within the Config VVol of a VM. So because we don’t have a file system we need a way to connect a VM to it’s VVols when it is powered on.
When a VM is powered on using VVols the bind operation is performed which connects a host to all of the VVols associated with that VM. When a VM is powered off those VVols are then unbound from the host as there is no need for them to be connected and a host can only bind to a limited number of VVols (4,096). To perform the bind though the host needs to know where the VVols of a VM reside and for that it needs to lookup the sub LUN ID’s for each VVol so they can be bound to.
Sub LUN’s are also a new concept with VVols, VMware had to request modifications to the SCSI T-10 standards to support the concept of sub LUNs. A sub LUN is basically a secondary level of LUNs, a host cannot directly see sub LUNs and they will not be visible during any SCSI scans that a host performs. To connect to a sub LUN a host must first connect to an administrative LUN which is visible to the host but it’s a special LUN with no provisioned storage and a bit set identifying it as part of a conglomerate. This administrative LUN is known as the Protocol Endpoint.
So to look up the sub LUN IDs for a VVol, a host has to ask someone for that information. The storage array knows the sub LUN IDs but a host can’t ask it directly, it must instead make the request to the VASA Provider which looks up the sub LUN information on the storage array and passes it on to the host so it can proceed with the bind. Because this process involves an intermediary and is not direct it is considered out-of-band. So essentially the host is asking for directions to get to it’s destination and once the VASA Provider provides it with an address (sub LUN ID) it can directly connect to those VVols via it’s direct in-band data path.
Now the VASA Provider component can be implemented either within a storage array or as an external component such as a VM appliance, it’s up to each vendor to choose how they want to implement it. No matter which way it is implemented the connection to the VASA Provider is via standard network protocols (TCP/IP) and not via a storage protocol (Fiber Channel/NFS/iSCSI). The connection to the VASA Provider is typically over your network via a management port on the storage array, if it’s external you have to connect to whatever VM/server is running the VASA Provider and then on to the storage array. Because the VASA Provider is connected over your standard network it is an out of band operation, to be an in-band operation a host would have to be able to perform this lookup via the data path using whatever storage protocol it is using to connect to the storage array. Below is a diagram which illustrates the components and paths that are used with VVols:
So now that you understand the difference between in-band and out-of-band binding, why does it matter which way you do it, either way you are getting the job done. Well performing the bind operation in-band has several advantages related to performance and availability compared to an out-of-band bind operation. An in-band operation will be performed quicker and more efficiently as a result of having a smaller number of network hops and by using faster in-band network connections. In addition a storage controller will typically higher processing power available to perform the operation faster. There is also no overhead imposed by the entire SOAP based VASA communication that is used when doing an out-of-bind operation.
Doing an in-band bind operation also provides overall better availability and reliability as you are not relying on a 3rd party to perform the operation which if it failed would prevent binding from occurring. This is especially true of external VASA Providers, if they crash, go down or something happens to them for whatever reason you cannot power on VMs until it is back up. In addition it also simplifies the bind process and makes it less likely that a mis-configuration could impact it.
So it sounds like in-band binding is definitely the way to go, why isn’t VVols using it yet? It appears VMware needs more time to implement this, I have seen mentions of in-band binding in both the VASA 2.0 & VASA 3.0 developer documentation. It looks like much of the framework to support in-band bind is already part of the VASA 3.0 spec so I expect to see the capability become available eventually in an upcoming release. Here’s what the developer guide says about this:
The vSphere 6.0 and 6.5 releases did not implement in-band bind. Bind is one of a few VVol management operations that are required for VM power-on. Failure to bind results in a “Data Unavailable” event that reflects poorly on storage system reliability. As a member of the T10 SCSI standard committee, VMware has been working to add in-band BIND and UNBIND commands to the SCSI standard. The intent was to define an in-band alternative to the corresponding VASA calls and enable use of the data path between host and storage array to power-on a VM when the network path between host and VASA provider is unavailable. When the T10 standard is finalized the specifications will be added to the VASA specification.
So for now we’ll just have to wait, once VMware completes what it needs to for in-band bind to be enabled in the VASA spec it will then be up on the storage vendors to make changes on their side to make the switch over to in-band bindings. Regardless of whether a bind operation is in-band or out-of-band it doesn’t impact the many benefits that VVols provides right now. The benefits of in-band binding will be under the covers and provide an extra lit bit of efficiency and reliability. So why wait, get started using VVols today and if you want to read more on VVols be sure and check out my huge VVols link collection and also read about what’s new with VVols in vSphere 6.5.