February 2020 archive

With the continued spread of the coronavirus more and more large event conferences are being canceled. Facebook has already canceled their F8 conference in May which was going to be held in San Francisco and attracts around 5,000 attendees. Mobile World Congress a large telecom event which was being held in Barcelona was also recently canceled. Other conferences like the Game Developers Conference (GDC) which attracts over 28,000 attendees and is scheduled to be held in March in San Francisco is still on for the moment but key vendors have already pulled out and they are closely watching for new developments that may cause them to cancel the event.

The situation with coronavirus is expected to get worse before it gets better and the CDC warns that it’s not a matter of if the coronavirus will spread in the US but more a question of when it will happen. In addition San Francisco has recently declared a state of emergency to help prepare for the spread of the virus within the community. With that in mind I expect we will see more and more tech conferences get cancelled this year. VMworld is still 6 months out and it might be too early to tell what the situation will be like then but if I had to speculate I’m betting there is a good chance that there won’t be a VMworld this year.

If the conference does get canceled I expect the event will go on instead as a virtual event instead of a physical in-person event which is what Facebook is doing with their F8 conference. There is already a framework in place to do this with the virtual VMUG events that are held periodically throughout the year. Virtual VMUGs do have keynotes, breakout sessions, vendor booths, chat and more but it definitely won’t be the same as an in-person event as you just can’t have the same quality of interaction with others that you get at a physical event.

I’m sure VMware is monitoring the situation closely and will plan accordingly based on how events unfold in the US. VMware’s big announcement for this year will occur next month at an online event but I’m sure they are excited to be able to use VMworld to show off everything they have announced. Other big tech conferences such as Dell World is coming up in May and HPE Discover in June and it will be interesting to see what happens with those events.

Until then we’ll have to wait and see, I’m still planning to attend, this will be my 13th time attending VMworld and I’ll be dis-appointed if the event does not happen this year. I’m hopeful that the emergency response within the US stops coronavirus from spreading through mass population centers but nothing is more important then your health and if the event cannot go on I’ll more than understand.

VMware has announced an upcoming event where they will reveal “new product details across VMware’s complete modern applications portfolio”. You can probably guess what this is about, the long awaited next major version of vSphere featuring the native Kubernetes support that they announced as Project Pacific back at VMworld. This has been the longest time between VMware major releases, almost 2 years since vSphere 6.7 was released in April 2018, so a release is way overdue. Part of the reason for the delay I believe is the extensive engineering VMware had to do to embed Kubernetes support directly into vSphere. So go sign-up for the event, in usual fashion the event is just the announcement for the new products and the release is usually a few weeks afterwards.

For all those that are using vVols and looking to upgrade to vSphere 6.7 Update 3 take note of an important change that VMware quietly made in that release which could potentially cause issues with vVols by not allowing self-signed certificates to be used by default. As you know a certificate is required for communication between vCenter, ESXi hosts and the VASA Provider of a storage array, this certificate is setup when you register a VASA Provider in vCenter Server. There are 2 different kinds of certificates that you can use for this, a server/self-signed certificate and a Certificate Authority (CA) signed certificate, let’s talk about the differences between the two.

A self-signed certificate is the cheap and easy route so many people go this route but it’s also less secure. Anyone can create a self-signed certificate which is basically equivalent to creating your own ID card that isn’t verified by the government. You can use self signed certificates both internally on your network and externally on the public internet. You’ll know if you are accessing a website that uses self-signed certificates as most browsers will display a warning and not load a https:// page by default and you have to choose an option to proceed at your own risk.

As it relates to the VASA Provider many users will create a self-signed certificate on a storage array that is then used to register with vCenter Server to secure communication to the VASA Provider. This is quick and easy to do and for internal use within a data center it is often done this way as many people foresee minimal risk within their internal networks. For example on 3PAR you simply use the ‘createcert vasa -selfsigned’ command to create the certificate and then when you register the VASA Provider in vCenter and it connects to the array using the URL provided you will get a security alert that you can verify the certificate fingerprint and you will have to say Yes to be able to proceed acknowledging you want to connect anyway.

A Certificate Authority (CA) signed certificate is more complicated as it requires setting up a trusted entity that can act as a authority to verify that a certificate is legitimate, this is equivalent of having a government verified ID card. A CA can be either privately deployed and managed in your data center or using one of the public CA’s like VeriSign or GoDaddy. In a private data center you would typically deploy your own CA such as the one built into Windows Server/Active Directory that would serve as the trusted root CA for your entire data center.

As it relates to the VASA Provider, VMware provides a CA service with their VMware Certificate Authority (VMCA) which is part of their Platform Services Controller (PSC). The VMCA acts as a CA for certificate management across the entire vSphere environment which includes vCenter Server, ESXi hosts and also the VASA Provider. So instead of using a certificate created and signed by the array you have to instead create a Certificate Signing Request (CSR) which is then used by the CA to generate a certificate which can be used by the storage array. Think of this as filling out an application for an ID card that is submitted to a government agency which approves it and issues you an ID card to use.

Setting up PSC and VMCA has historically been a complicated process since it was introduced in vSphere 6.0, VMware has tried to make it easier in subsequent releases but it can still be a headache. I know I personally have always dreaded certificate management and I think many admins are also intimidated by certificate management. That’s why for those instances when you need to use a certificate, creating a self-signed one is much simpler to do especially if you do not have a CA in your environment already.

So onto the issue as it relates to vVols and vSphere 6.7 U3, VMware has been continually trying to make vSphere more secure and the change they made in 6.7 U3 is to not allow self-signed certificates by default. We were told by VMware to expect this change in an upcoming 6.7 release and also in vSphere.Next and have been pushing back on them to give us time to allow us to test, document and communicate this to customers. The change made it into the upcoming vSphere.Next build but we were able to get them to back it out and wait for U1 but it looks like they went ahead and already implemented it in 6.7 U3.

So if you are using any self-signed certificates they will no longer work if you upgrade to 6.7 U3. So this means the communication with the VASA Provider will also no longer work which will impact certain operations such as powering on VMs. We think the majority of our customers using vVols are using self-signed certificates so this could have a big impact, I have already been involved with one customer that upgraded and experienced this issue. The quick workaround is to add a host advanced setting on every host to allow self-signed certificates, once you add this setting they will work again. See the below note in the vSphere 6.7 U3 release notes:

• You might be unable to add a self-signed certificate to the ESXi trust store and fail to add an ESXi host to the vCenter Server system The ESXi trust store contains a list of Certificate Authority (CA) certificates that are used to build the chain of trust when an ESXi host is the client in a TLS channel communication. The certificates in the trust store must be with a CA bit set: X509v3 Basic Constraints: CA: TRUE. If a certificate without this bit set is passed to the trust store, for example, a self-signed certificate, the certificate is rejected. As a result, you might fail to add an ESXi host to the vCenter Server system.This issue is resolved in this release. The fix adds the advanced option Config.HostAgent.ssl.keyStore.allowSelfSigned. If you already face the issue, set this option to TRUE to add a self-signed server certificate to the ESXi trust store.

While this works for now, longer term you will need to switch to using VMCA as your certificate authority and not use self-signed certificates for your VASA Provider. We are working with VMware to see if they can back this out of 6.7 for now but that most likely will not happen. Hopefully VMware will communicate and document this better instead of just a footnote in their release notes as it can have a big impact on customer environments not just with vVols but anything that uses a self-signed certificate. We are also looking to publish some guidance to help make it easier for customers to migrate from self-signed certificates to VMCA certificates for their VASA Provider. So be careful if you plan on upgrading to 6.7 U3 and definitely plan on using VMCA at some point. As I get more information on this change I’ll pass it along.