Nov 18 2016

New PowerCLI cmdlets to support VVol replication in vSphere 6.5

With the release of vSphere 6.5 and support for VVol replication there was also a need to be able to manage and automate certain replication related tasks. Out of the box there is no integration with VMware’s automation products such as SRM and vRealize Orchestrator and Automation. So to be able to actually use the VVol replication capabilities in vSphere 6.5 VMware had to develop new cmdlets for it. There are only a few scenarios associated with VVol replication that you can perform actions on. That includes doing either a planned or unplanned failover, a test failover, reversing replication and forcing a sync replication.

VMware posted PowerCLI 6.5R1 yesterday and below are the cmdlets related to VVol replication along with a brief description on what they do. Customers will have to create their own scripts for now until VMware adds support for VVol replication tasks into their automation products. Also note right now there is no cmdlet for doing a test failover which essentially clones the replicated VVols at the target site and exposes them to vSphere for testing. So if you want to do that (who wouldn’t) you’ll have to use one of their other programming interfaces such as Java.

  • Get-SpbmFaultDomain – This cmdlet retrieves fault domains based on name or ID filter. The fault domain ID is globally unique.
  • Get-SpbmReplicationGroup – This cmdlet retrieves replication groups. The replication groups can be of type source or target.
  • Get-SpbmReplicationPair – This cmdlet retrieves the relation of replication groups in a pair of source and target replication group.
  • Start-SpbmReplicationFailover – This cmdlet performs a failover of the devices in the specified replication groups. This cmdlet should be called at the replication target location. After the operation succeeds, the devices will be ready to be registered by using the virtual machine file path.
  • Start-SpbmReplicationPrepareFailover – This cmdlet prepares the specified replication groups to fail over.
  • Start-SpbmReplicationReverse – This cmdlet initiates reverse replication, by making the currently failed over replication group the source and its peer replication group the target. The devices in the replication group will start getting replicated to this new target site, which was the source before the failover.
  • Sync-SpbmReplicationGroup – This cmdlet synchronizes the data between source and replica for the specified replication group. The replicas of the devices in the replication group are updated and a new point in time replica is created. This function should be called at the replication target location
Share This:

Nov 18 2016

Which storage vendors support VVols 2.0 in vSphere 6.5?

Answer: Not many at all. The vSphere 6.5 release introduces the next iteration of VVols that is built on the updated VASA 3.0 specification that brings support for array based replication which was not supported with VVols in vSphere 6. If you take a look at the VVols HCL for vSphere 6.5 there are only 4 vendors listed there for supporting VVols right  now. This is pretty much in line with what happened with the first release of VVols in vSphere 6, only 4 vendors supported VVols on day 1 of the vSphere 6 launch as well.

vvols20-day1The 4 vendors listed in the HCL for supporting VVols in vSphere 6.5 are Fujitisu, Hitachi (HDS), HPE & Huawei. One thing that is not clear though is of those 4 vendors, which of them support replication with VVols. I had heard that there would be an indication of some sort in the HCL listing if a vendor supported VVol replication, presumably this would be listed in the Feature section. If this is the case it looks like there are no vendors that support VVol replication yet. Supporting VVols in vSphere 6.5 doesn’t automatically mean that you support replication as well. I know with HPE specifically, 3PAR does not yet support replication despite supporting VVols in vSphere 6.5, replication support will be coming in an upcoming 3PAR OS release.

I’m not really surprised by the lack of support on day 1 as implementing replication of VVols is complicated and requires a lot of development work. I’ve seen this first hand as I sit in on weekly meetings with the 3PAR VVol development team. To be fair, I don’t expect most vendors to rush their VVol support out the door as there are not really a lot of customers that are upgrading to vSphere 6.5 on day 1. The VVol design partners and reference platforms (HPE, Dell, NetApp) have an edge as they have been working longer at it and also working close with VMware to develop and test VVols through the vSphere 6.5 development lifecycle so I would expect them to be quicker to market with VVols 2.0 support.

So if you don’t see your vendor on the HCL today, have patience as they will likely introduce support for VVols when they are are ready. Remember that VVols is a specification that VMware dictates and it’s up to each vendor to develop their own capabilities within that specification however that want to. I suspect if you check the HCL again in 3 months it will start to grow larger, I’m sure Dell, NetApp and Nimble may be there soon. So stay tuned, if you’re really antsy to replicate VVols check with your storage vendor and ask them when they will deliver it. VVols 2.0 delivers both maturity in the VASA specification as well as replication support so it’s going to be worth the wait.

Share This:

Nov 17 2016

A comparison of VMFS5 & VMFS6 in vSphere 6.5

vSphere 6.5 has introduced a new VMFS version 6 and there are a few changes in it compared to VMFS version 5 that you should be aware of. You have to love VMware’s crazy out of sync versioning across their product lines, now naturally you would think vSphere 6.0 would have VMFS6 in it but VMware kept it at VMFS5 with an incremental version and VMFS6 is new with vSphere 6.5. The table below highlights the difference between the two that you should be aware of but I also wanted to make you aware of some additional info you should know when upgrading to vSphere 6.5 or operating in a mixed vSphere version environment.

The first is once again you can’t upgrade in place existing VMFS5 volumes to VMFS6. That royally sucks and you have to plan migrations by creating new VMFS6 datastores, migrating VMs to them with Storage vMotion and then deleting the VMFS5 datastores when you are done to get back your disk space. Depending on your environment and how much space you have available on your array this can be a long and painful migration. [BEGIN VVols Plug] With VVols you don’t have to deal with any of that BS as you aren’t using VMFS and don’t have to constantly upgrade a file system [END VVols Plug]

Now you might think, screw that, a file system is just a file system and I’ll stick with VMFS5, but you miss out on the automatic space reclamation that is finally back in vSphere 6.5. Be aware that the new 512e drive support in vSphere 6.5 is supported on either VMFS6 or VMFS5 as long as the host is running ESXi 6.5. Beyond that there isn’t too much difference between the two, they also both now support 512 LUNs/VMFS datastores per host as well (note vSphere 6.5 storage doc incorrectly states 1024). So you may end up sticking with VMFS5 but I think the automatic reclamation does make for a compelling use case to upgrade to VMFS6.

So it’s up to you to decide, if you want to learn more be sure and look through the vSphere 6.5 storage documentation, and if you are fed up with VMFS upgrades and want something way cooler give VVols a serious look.

Feature & FunctionalityVMFS6VMFS5
Can vSphere 6.5 host access?YesYes
Can vSphere 6.0 and earlier hosts access?NoYes
VMFS Datastores per Host512*512*
512n storage device supportYesYes
512e storage device supportYesYes (Not on local 512e devices)
Automatic space reclamation (UNMAP)YesNo
Manual space reclamation (esxcli)YesYes
Space reclamation within guest OSYesLimited
GPT storage device partitioningYesYes
MBR storage device partitioningNoYes
Block size1MB1MB
Default snapshot typeSEsparseVMFSsparse (virtual disks < 2 TB SEsparse (virtual disks > 2 TB)
Virtual disk emulation type512n512n
Support of small files of 1 KBYesYes

Share This:

Nov 16 2016

How the Top vBlogs are performing (or how to optimize your WordPress site)

My site has recently been plagued by performance issues and I spent a lot of time trying to figure out the cause as well as optimizing it so it would perform much better. It was pretty bad and pages were taking 8-12 seconds to load on average. As a former server admin my natural instinct was to immediately blame the hardware platform that my site was running on. In my case I run WordPress hosted by InMotion Hosting so I started by calling them and blaming them for my slow hosting.

They had me run some external tests and from those the underlying causes were uncovered (it wasn’t a server issue). Having eliminated the hosting platform as the cause I did a bunch of research and tried a lot of optimization plug-ins and methods in an effort to speed up the site. When I was done the combination of everything I had done had sped up the site considerably to the point that I am now happy with my overall performance.

Now that I am past that I thought I would pass on some lessons learned to help other who sites might not be optimized properly which can cause your blog visitor experience to suffer not to mention impact how google searches your site. Google has stated that PageSpeed, which is a measurement that they use when ranking search results will directly impact how your site is ranked when user search for content. They have measurements for both mobile and desktop page load times, the slower your site takes to load, the worse you end up in their rankings. So having a site that is well optimized is very beneficial to both your visitors and to your visibility on the internet.

So let’s start on how to analyze your site, I mainly used one online tool for that which is a site called GTmetrix. All you do is go to their site, put in your blog URL and they run a variety of tests on it and generate a report.that has detailed scores, benchmarks, timings and recommendations. They run your site through both Google PageSpeed testing and YSlow testing so your site is analyzed from 2 different perspectives. The results can point you in the right direction of what to fix and the recommendations they provide can tell you how to do it. When you run a report make sure you click on the Waterfall tab to see the load times for all your website elements.

My site was initially scoring as an F grade by both PageSpeed and YSlow, so I had a lot of work to do, below are the various things I did to bring that score up an A & B grade.

gtmetrix1Implement a caching plug-in

I went with W3 Total Cache which has excellent reviews, it’s fairly easy to install but there are a lot of knobs you have to turn to configure it optimally, here is a guide to get it set properly although every site may vary.

Implement CDN caching

Having a content delivery network (CDN) cache your content for you can help your site perform considerably faster as they deliver content from CDN servers across the world instead of from your site. CDN servers are typically closer geographically to visitors and can serve content more efficiently. W3 Total Cache can work together with a CDN to provide the best possible site performance. I signed up for a free CloudFlare CDN account, updated my DNS nameservers to point to the CloudFlare nameservers and then configured the CloudFlare extension inside the W3 Total Cache Plugin. All in in it wasn’t that difficult and you can read how to do it here.

Optimize your database

Your WordPress MySQL database can accumulate a lot of remnants and junk over time, so the best way to clean it up is to use a plug-in that will scan all your tables and make clean-up recommendations that you can execute. I first disabled revisions on my site as the more you edit a doc (which I do frequently on my link pages) you will create more revisions of it that fill up your database. I used the Advanced Database Cleaner plug-in which did the job nicely, be sure and backup your database first using a plug-in or through your provider.

Modify ajax heartbeats

Disabling or limiting ajax heartbeats is recommended to improve performance, here is a good article on how to do that with a plug-in. In my case the admin-ajax.php was causing a lot of delay in load times which was mostly caused by a plug-in (still trying to fix that). I identified that in the Waterfall tab in the GTMetrix report, there was a GET to admin-ajax.php that was taking 6-7 seconds to complete.

Optimize your images

I tend to use a lot of images in my content and unless you optimize them properly it can really slow your site down as non-optimized images can take much longer time to load. I initially looked at the WP Smush plug-in but ended up going with the EWWW Image Optimizer plug-in instead. It will run though all your existing images and optimize them and reduce their total size which will reduce their load time.

Force browser caching

I inserted some code into my .htaccess file to force browser caching, you can read how to do that here.

Investigate your plug-ins

Plug-ins can frequently be the cause of slow performance but unfortunately there isn’t any easy way to determine which one is at fault. There is a plug-in called P3 Performance Profiler that GoDaddy created for WordPress to analyze plug-in performance but unfortunately it hasn’t been updated in over 2 years and does not work with the latest versions of WordPress so avoid it. If you look the GTMetrix Waterfall results you might be able to identify a slow plug-in but if you suspect a plug-in is causing performance issues the easiest way to identify which one is to disable them one at a time and run a new GTMetrix scan afterwards to see if your site performance improves. Once you know which plug-in is causing it go talk to the developer or switch to a different plug-in that offers the same functionality.

How did the Top vBlogs score?

So once I had my site optimized I was curious as to how other Top vBlogs were performing so I ran the top 10 through GTMetrix and below are the results. Hats off to Chris Wahl and Scott Lowe whose sites are blazing fast, Alan Renouf’s site was generating 404 errors in the analysis so I skipped it and included Chad Sakac instead. I encourage other bloggers to test out their site as well, see how you score and let me know in the comments, also let me know what you did to improve it and hopefully what I posted here helped you out.

Site URLPageSpeed ScoreYSlow ScorePage Load TimeTotal Page SizeRequests (79%)E (51%)3.5s.97MB57 (84%)D (62%)6.5s879KB77 (83%)D (61%)3.4s940KB79 (75%)D (66%)4.7s1.05MB58 A (96%)B (86%)1.7s368KB40 C (75%)E (57%)5.2s1.53MB130 A (96%)B (87%)0.8s190KB11 (35%)E (58%)3.6s1.28MB35 (77%)E (54%)8.4s1.14MB124 (63%)D (67%)6.6s6.35MB70

Share This:

Nov 15 2016

Configuration Maximum changes from vSphere 6.0 to vSphere 6.5

vSphere 6.5 is now available and with every release VMware makes changes to the configuration maximums for vSphere. Since VMware never highlights what has changed between releases in their official Configuration Maximum 6.5 documentation, I thought I would compare the document with the 6.0 version and list the changes between the versions here.

ConfigurationvSphere 6.5vSphere 6.0
Virtual Machine Maximums
RAM per VM6128GB4080GB
Virtual NVMe adapters per VM4N/A
Virtual NVMe targets per virtual SCSI adapter15N/A
Virtual NVMe targets per VM60N/A
Virtual RDMA Adapters per VM1N/A
Video memory per VM2GB512MB
ESXi Host Maximums
Logical CPUs per host576480
RAM per host12TB6TB *some exceptions
LUNs per server512256
Number of total paths on a server20481024
FC LUNs per host512256
LUN ID0 to 163830 to 1023
VMFS Volumes per host512256
FT virtual machines per cluster12898
vCenter Maximums
Hosts per vCenter Server20001000
Powered-on VMs per vCenter Server2500010000
Registered VMs per vCenter Server3500015000
Number of host per datacenter2000500
Maximum mixed vSphere Client (HTML5) + vSphere Web
Client simultaneous connections per VC
60 (30 Flex, 30 maximum HTML5)N/A
Maximum supported inventory for vSphere Client
10,000 VMs, 1,000 HostsN/A
Host Profile Datastores256120
Host Profile Created5001200
Host Profile Attached5001000
Platform Services Controller Maximums
Maximum PSCs per vSphere Domain108
vCenter Server Extensions Maximums
[VUM] VMware Tools upgrade per ESXi host3024
[VUM] Virtual machine hardware upgrade per host3024
[VUM] VMware Tools scan per VUM server20090
[VUM] VMware Tools upgrade per VUM server20075
[VUM] Virtual machine hardware scan per VUM server20090
[VUM] Virtual machine hardware upgrade per VUM server20075
[VUM] ESXi host scan per VUM server23275
[VUM] ESXi host patch remediation per VUM server23271
[VUM] ESXi host upgrade per VUM server23271
Virtual SAN Maximums
Virtual machines per cluster60006400
Number of iSCSI LUNs per Cluster1024N/A
Number of iSCSI Targets per Cluster128N/A
Number of iSCSI LUNs per Target256N/A
Max iSCSI LUN size62TBN/A
Number of iSCSI sessions per Node1024N/A
iSCSI IO queue depth per Node4096N/A
Number of outstanding writes per iSCSI LUN128N/A
Number of outstanding IOs per iSCSI LUN256N/A
Number of initiators who register PR key for a iSCSI LUN64N/A
Storage Policy Maximums
Maximum number of VM storage policies1024Not Published
Maximum number of VASA providers1024Not Published
Maximum number of rule sets in VM storage
Maximum capabilities in VM storage policy
rule set
Maximum vSphere tags in virtual machine storage policy128Not Published

Share This:

Nov 15 2016

vSphere 6.5 is now available!

You can download it here.

Also check out the documentation here, the first doc that I always go to is the Configuration Maximums to see how things have grown.

And be sure to read through my Summary of What’s New in vSphere 6.5 post and What’s New in VSAN 6.5 post and also check out my huge vSphere 6.5 Link-O-Rama which is your one source for all vSphere 6.5 related links on the internet.


Share This:

Nov 09 2016

Upcoming vmLIVE webinar on VVols replication

Want to learn more about the new Replication functionality for VVols in vSphere 6.5? If you are a VMware partner you can sign-up for a vmLIVE webinar next week on 11/15 at 7:00am PST. I’ll be co-presenting with Ben Meadowcroft who is the VMware product manager for VVols. We’ll be covering what’s new with VVol’s in vSphere 6.5, much of which is around replication and how HPE 3PAR StoreServ has implemented support for it. So if you are a VMware partner head on over to VMware’s Partner Central website and sign-up. If you are not a partner you won’t be able to sign-up but you can still read my overview on VVols replication and also check out my VMworld 2016 session which covers it as well.


Share This:

Oct 28 2016

vSphere 6.5 with VVols 2.0 does not yet support in-band bind

One thing I noticed that was missing from the vSphere 6.5 release which is expected at some point is support for an in-band VVol bind process that a host must perform when a VM is powered on. The current VVol implementation is an out-of-band process which involves the VASA Provider which acts as the control path between vCenter, ESXi hosts and a storage array. In this post I’m going to go into detail about the bind process and why it would be beneficial to be an in-band process.

So first the basics, the concept of binding a VM to it’s disk files is unique to VVols. With VMFS/NFS we didn’t need this process as you have a file system in place that can handle file look-ups and locking. With VVols we don’t have a file system written to the storage array, VMs are written natively to the array as VVol objects which are essentially sub LUNS (block) or files (NFS). There still is a mini VMFS file system with VVols but it is contained within the Config VVol of a VM. So because we don’t have a file system we need a way to connect a VM to it’s VVols when it is powered on.

When a VM is powered on using VVols the bind operation is performed which connects a host to all of the VVols associated with that VM. When a VM is powered off those VVols are then unbound from the host as there is no need for them to be connected and a host can only bind to a limited number of VVols (4,096). To perform the bind though the host needs to know where the VVols of a VM reside and for that it needs to lookup the sub LUN ID’s for each VVol so they can be bound to.

Sub LUN’s are also a new concept with VVols, VMware had to request modifications to the SCSI T-10 standards to support the concept of sub LUNs. A sub LUN is basically a secondary level of LUNs, a host cannot directly see sub LUNs and they will not be visible during any SCSI scans that a host performs. To connect to a sub LUN a host must first connect to an administrative LUN which is visible to the host but it’s a special LUN with no provisioned storage and a bit set identifying it as part of a conglomerate. This administrative LUN is known as the Protocol Endpoint.

So to look up the sub LUN IDs for a VVol, a host has to ask someone for that information. The storage array knows the sub LUN IDs but a host can’t ask it directly, it must instead make the request to the VASA Provider which looks up the sub LUN information on the storage array and passes it on to the host so it can proceed with the bind. Because this process involves an intermediary and is not direct it is considered out-of-band. So essentially the host is asking for directions to get to it’s destination and once the VASA Provider provides it with an address (sub LUN ID) it can directly connect to those VVols via it’s direct in-band data path.

Now the VASA Provider component can be implemented either within a storage array or as an external component such as a VM appliance, it’s up to each vendor to choose how they want to implement it. No matter which way it is implemented the connection to the VASA Provider is via standard network protocols (TCP/IP) and not via a storage protocol (Fiber Channel/NFS/iSCSI). The connection to the VASA Provider is typically over your network via a management port on the storage array, if it’s external you have to connect to whatever VM/server is running the VASA Provider and then on to the storage array. Because the VASA Provider is connected over your standard network it is an out of band operation, to be an in-band operation a host would have to be able to perform this lookup via the data path using whatever storage protocol it is using to connect to the storage array. Below is a diagram which illustrates the components and paths that are used with VVols:


So now that you understand the difference between in-band and out-of-band binding, why does it matter which way you do it, either way you are getting the job done. Well performing the bind operation in-band has several advantages related to performance and availability compared to an out-of-band bind operation. An in-band operation will be performed quicker and more efficiently as a result of having a smaller number of network hops and by using faster in-band network connections. In addition a storage controller will typically higher processing power available to perform the operation faster. There is also no overhead imposed by the entire SOAP based VASA communication that is used when doing an out-of-bind operation.

Doing an in-band bind operation also provides overall better availability and reliability as you are not relying on a 3rd party to perform the operation which if it failed would prevent binding from occurring. This is especially true of external VASA Providers, if they crash, go down or something happens to them for whatever reason you cannot power on VMs until it is back up. In addition it also simplifies the bind process and makes it less likely that a mis-configuration could impact it.

So it sounds like in-band binding is definitely the way to go, why isn’t VVols using it yet? It appears VMware needs more time to implement this, I have seen mentions of in-band binding in both the VASA 2.0 & VASA 3.0 developer documentation. It looks like much of the framework to support in-band bind is already part of the VASA 3.0 spec so I expect to see the capability become available eventually in an upcoming release. Here’s what the developer guide says about this:

The vSphere 6.0 and 6.5 releases did not implement in-band bind. Bind is one of a few VVol management operations that are required for VM power-on. Failure to bind results in a “Data Unavailable” event that reflects poorly on storage system reliability. As a member of the T10 SCSI standard committee, VMware has been working to add in-band BIND and UNBIND commands to the SCSI standard. The intent was to define an in-band alternative to the corresponding VASA calls and enable use of the data path between host and storage array to power-on a VM when the network path between host and VASA provider is unavailable. When the T10 standard is finalized the specifications will be added to the VASA specification.

So for now we’ll just have to wait, once VMware completes what it needs to for in-band bind to be enabled in the VASA spec it will then be up on the storage vendors to make changes on their side to make the switch over to in-band bindings. Regardless of whether a bind operation is in-band or out-of-band it doesn’t impact the many benefits that VVols provides right now. The benefits of in-band binding will be under the covers and provide an extra lit bit of efficiency and reliability. So why wait, get started using VVols today and if you want to read more on VVols be sure and check out my huge VVols link collection and also read about what’s new with VVols in vSphere 6.5.

Share This:

Older posts «

» Newer posts