For all those that are using vVols and looking to upgrade to vSphere 6.7 Update 3 take note of an important change that VMware quietly made in that release which could potentially cause issues with vVols by not allowing self-signed certificates to be used by default. As you know a certificate is required for communication between vCenter, ESXi hosts and the VASA Provider of a storage array, this certificate is setup when you register a VASA Provider in vCenter Server. There are 2 different kinds of certificates that you can use for this, a server/self-signed certificate and a Certificate Authority (CA) signed certificate, let’s talk about the differences between the two.
A self-signed certificate is the cheap and easy route so many people go this route but it’s also less secure. Anyone can create a self-signed certificate which is basically equivalent to creating your own ID card that isn’t verified by the government. You can use self signed certificates both internally on your network and externally on the public internet. You’ll know if you are accessing a website that uses self-signed certificates as most browsers will display a warning and not load a https:// page by default and you have to choose an option to proceed at your own risk.
As it relates to the VASA Provider many users will create a self-signed certificate on a storage array that is then used to register with vCenter Server to secure communication to the VASA Provider. This is quick and easy to do and for internal use within a data center it is often done this way as many people foresee minimal risk within their internal networks. For example on 3PAR you simply use the ‘createcert vasa -selfsigned’ command to create the certificate and then when you register the VASA Provider in vCenter and it connects to the array using the URL provided you will get a security alert that you can verify the certificate fingerprint and you will have to say Yes to be able to proceed acknowledging you want to connect anyway.
A Certificate Authority (CA) signed certificate is more complicated as it requires setting up a trusted entity that can act as a authority to verify that a certificate is legitimate, this is equivalent of having a government verified ID card. A CA can be either privately deployed and managed in your data center or using one of the public CA’s like VeriSign or GoDaddy. In a private data center you would typically deploy your own CA such as the one built into Windows Server/Active Directory that would serve as the trusted root CA for your entire data center.
As it relates to the VASA Provider, VMware provides a CA service with their VMware Certificate Authority (VMCA) which is part of their Platform Services Controller (PSC). The VMCA acts as a CA for certificate management across the entire vSphere environment which includes vCenter Server, ESXi hosts and also the VASA Provider. So instead of using a certificate created and signed by the array you have to instead create a Certificate Signing Request (CSR) which is then used by the CA to generate a certificate which can be used by the storage array. Think of this as filling out an application for an ID card that is submitted to a government agency which approves it and issues you an ID card to use.
Setting up PSC and VMCA has historically been a complicated process since it was introduced in vSphere 6.0, VMware has tried to make it easier in subsequent releases but it can still be a headache. I know I personally have always dreaded certificate management and I think many admins are also intimidated by certificate management. That’s why for those instances when you need to use a certificate, creating a self-signed one is much simpler to do especially if you do not have a CA in your environment already.
So onto the issue as it relates to vVols and vSphere 6.7 U3, VMware has been continually trying to make vSphere more secure and the change they made in 6.7 U3 is to not allow self-signed certificates by default. We were told by VMware to expect this change in an upcoming 6.7 release and also in vSphere.Next and have been pushing back on them to give us time to allow us to test, document and communicate this to customers. The change made it into the upcoming vSphere.Next build but we were able to get them to back it out and wait for U1 but it looks like they went ahead and already implemented it in 6.7 U3.
So if you are using any self-signed certificates they will no longer work if you upgrade to 6.7 U3. So this means the communication with the VASA Provider will also no longer work which will impact certain operations such as powering on VMs. We think the majority of our customers using vVols are using self-signed certificates so this could have a big impact, I have already been involved with one customer that upgraded and experienced this issue. The quick workaround is to add a host advanced setting on every host to allow self-signed certificates, once you add this setting they will work again. See the below note in the vSphere 6.7 U3 release notes:
- You might be unable to add a self-signed certificate to the ESXi trust store and fail to add an ESXi host to the vCenter Server system The ESXi trust store contains a list of Certificate Authority (CA) certificates that are used to build the chain of trust when an ESXi host is the client in a TLS channel communication. The certificates in the trust store must be with a CA bit set:
X509v3 Basic Constraints: CA: TRUE. If a certificate without this bit set is passed to the trust store, for example, a self-signed certificate, the certificate is rejected. As a result, you might fail to add an ESXi host to the vCenter Server system.This issue is resolved in this release. The fix adds the advanced optionConfig.HostAgent.ssl.keyStore.allowSelfSigned. If you already face the issue, set this option toTRUEto add a self-signed server certificate to the ESXi trust store.
While this works for now, longer term you will need to switch to using VMCA as your certificate authority and not use self-signed certificates for your VASA Provider. We are working with VMware to see if they can back this out of 6.7 for now but that most likely will not happen. Hopefully VMware will communicate and document this better instead of just a footnote in their release notes as it can have a big impact on customer environments not just with vVols but anything that uses a self-signed certificate. We are also looking to publish some guidance to help make it easier for customers to migrate from self-signed certificates to VMCA certificates for their VASA Provider. So be careful if you plan on upgrading to 6.7 U3 and definitely plan on using VMCA at some point. As I get more information on this change I’ll pass it along.
