February 2016 archive

I was browsing through the latest docs in the VMware KB this week and noticed one describing the end of availability of vSphere Enterprise, vSphere with Operations Manager (vSOM) Standard and Enterprise edition. That got me to browsing through VMware’s product lifecycle matrix which provides dates for every VMware products on GA, end of availability, end of support and end of technical guidance. Going through the huge list (VMware has so many products past and present), these are the key ones listed that are ending support in 2016:

• VMware App Volumes 2.5, 2.6, 2.7 – 2016/12/09
• VMware Data Recovery 2.0 – 2016/08/24
• VMware ESXi 5.0 and 5.1 – 2016/08/24
• VMware EVO:RAIL 1.0 and 1.1 – 2016/09/09
• VMware EVO:RAIL 1.2 – 2016/12/10
• VMware Fusion 7.x – 2016/03/03
• VMware NSX for vSphere 6.1 – 2016/10/15
• VMware vCenter Log Insight 2.0 – 2016/06/10
• VMware vCenter Operations 5.8.5 – 2016/12/31
• VMware vCenter Server 5.0 and 5.1 – 2016/08/24
• VMware vCenter Site Recovery Manager 5.0 and 5.1 – 2016/08/24
• VMware vCenter Update Manager 5.0 and 5.1 – 2016/08/24
• VMware vCloud Automation Center 5.2 – 2016/12/10
• VMware vCloud Networking and Security 5.5 – 2016/09/19
• VMware vRealize Operations 6.0 and 6.1 – No earlier than 2016/12/09
• VMware vSphere Data Protection 5.1 – 2016/08/24
• VMware vSphere Replication 5.1 – 2016/08/24
• VMware Workstation 11.x – 2016/06/02

Again these are the end of support dates for these products which means if you have SnS, VMware offers maintenance updates and upgrades, bug/security fixes and technical assistance until these dates. After these dates they have typically have a technical guidance phase which lastes for a fixed duration and provides support through their self-help portal only (no telephone support), they do not offer new hardware support, server/client/guest OS updates, new security patches or bug fixes during this phase. You can read more about these different product support phases here. So for those of you still on vSphere 5.0 & 5.1, the clock is ticking, better start planning your upgrades.

Date anomalies such as Y2K, Leap Years, and the infamous1970 bug can be disruptive if software code is not designed to handle them properly. You might remember the infamous VMware time bomb bug a few years back which made them look bad. In 2016 we once again have a Leap Year and VMware has gone out and tested the impact of that date across their entire product line to ensure nothing is impacted by this once every four year date occurrence. VMware has published the results of their testing in a KB article which appears to cover all of their products and which are unaffected by Leap Year. I expect they didn’t test every past version of these products but as Leap Year isn’t really new and occurs every four years I don’t expect you’ll see any issues regardless of which version of a VMware product you are on. So with this reassurance in mind hopefully you will sleep better Sunday night and not get woken up by any surprises.

VMware’s new Virtual Volumes storage architecture required some changes to the SCSI specifications for block arrays to support the new sub-LUN model that VVols uses. Because of this it required I/O vendors to update their software to be able to support the new Protocol Endpoint component in VVols that is used as the data path between hosts and VMs (VVols) residing on storage arrays. Early on after VVols was released many I/O vendors were in catch-up mode to introduce support for this into their I/O adapter firmware. As a result there is a good chance that the I/O adapter that you are using may not work with VVols or may require a firmware upgrade to support it.

Let’s first look at what changed with VVols that required changes to be made to I/O adapter firmware to support VVols. A traditional block I/O adapter connects to a LUN on a storage array where your VMFS datastore is located. To connect to the LUN it simply needs the data path information (WWN) which includes HBA #, controller and of course the LUN ID of the volume associated with the VMFS datastore. You’ll see this in the vSphere client when selecting an I/O adapter with a syntax similar to vmhbaAdapter:CChannel:TTarget:LLUN (i.e. vmhba1:C0:T3:L1).

VVols introduces the concept of Secondary LUN IDs which is essentially an additional layer of LUN numbering that supports a lot more sub-LUNs than a array traditionally supports. The way this works is that a host will connect to a special Administrative LUN on the storage array via the Protocol Endpoint. This Admin LUN has no storage allocated to it and serves as a gateway to the sub-LUNs beneath it, it’s LUN ID is usually greater than 255 to identify it as a non-data LUN. A host cannot connect directly to a sub-LUN and must go through the Admin LUN to get to it. These Secondary LUN ID’s are provided to a host via the VASA Provider, so you can see why it is an important component. You can also see why direct to SAN backup is not supported with VVols as you cannot connect to a sub LUN without going through a ESXi host. The relationship between these components is outlined in the figure below:

The architecture is a bit complex and introduces additional identifiers beyond the LUN ID to connect to VVols on the array. These additional identifiers include secondary level IDs (SLLID), VVol IDs and Reference counts, the PE LUN still has a traditional WWN associated with it. Now while both block and NFS arrays utilize protocol endpoints and SLLIDs, the special Admin (PE) LUN only applies to block storage arrays, if you are using a NFS array with VVols this doesn’t apply as their is no special PE LUN.  With block arrays PEs are discovered via an in-band path using the standard SCSI command, REPORT_LUNS which reports the WWN of the PE, with NFS PEs are discovered via an out-of-band path using an API which returns the IP address and mount point of the PE. PE LUNs on block storage are recognized differently from traditional data LUNs as they have a special conglomerate bit set (LU_CONG).

VMware has patented this new storage architecture that they refer to as “Computer system accessing object storage system“, here are a few diagrams from the patent, if you want to deep-dive into this new architecture give the patent a read.

This new sub LUN architecture required VMware to submit update proposals to the T-10 committee for SCSI specifications to support it and the end result of all this was I/O adapters had to be updated to support this as well.

So know we know the why, let’s look at the how, as in how do I know if my I/O adapter will support VVols? Perhaps the quickest and easiest way is to simply look it up on the VMware HCL. If you go the VMware HCL and select I/O Adapters you will notice a new Feature there that you can select that is called “Secondary LUNID”. You can simply select that feature and select your I/O Adapter brand name, optionally the device type (i.e. FC) and then search as shown below.

In the results you will see the firmware versions needed to support VVols and links to any special drivers that might be needed for ESXi to support it as shown below.

From what I’ve seen there are some I/O Adapters that will work using the standard ESXi image, some require custom server ESXI images (i.e. HPE, Dell) and other that require you to download and install a driver into ESXi. You can see many of these in the vSphere Drivers/Tools download page.

One thing to note is if your I/O Adapter does not support VVols or does not have the firmware to support it you will see this error in your vmkernel logs:

Sanity check failed for path vmhbaX:Y:Z. The path is to a VVol PE, but it goes out of adapter vmhbaX which is not PE capable. Path dropped.

To check if your HBA is VVol capable you can run this command on your ESXi hosts: esxcli storage core adapter list , you should see Second Level Lun ID (SLLID) listed under the Capabilities column if the I/O Adapter supports VVols as shown below, VMware has a KB on this as well.

Once you are sure your I/O Adapter has the required firmware and the proper ESXi I/O driver is present you can check from the host to see if it is able to recognize the Protocol Endpoint (Admin LUN). You’ll need to make sure that your VASA Provider is enabled and setup before you can do this though. Once that is setup you can use the esxcli command run from each host to see if the PE is visible to it, the syntax for the command is esxcli storage core device list –pe-only as shown below.

In the output that is produced if you see a value of “true” for “Is VVOL PE” that confirms that the host is able to connect successfully to the array PE to access VVols. Often times this will be false if your I/O adapter does not support VVols or VVols is not setup correctly.

So there you have it, it may seem a bit complicated but there is a good chance that your I/O Adapter already supports VVols and you really won’t have to do anything to start using it. It’s always a good thing to check though and make sure your I/O Adapters and storage arrays have the required firmware level to support VVols. Hopefully this gives you a better understanding of how VVols works under the hood and the relationships between hosts, protocol endpoints and VVols.

This is a follow-up to my previous post on VVols commenting on The Register’s VVol article. I’ve seen this question come up often so I wanted to provide some detailed information on the minimum, average and maximum amount of VVols a VM can have. This has a direct impact on the scalability of VVols as the number of VVols that can be supported on a storage array varies by vendor, I’ve seen some as low as 1,000 and other over 100,000. Let’s first start with the number of VVols that can be bound to a single host, right now that number is 64,000, although I’m pretty sure that number was 4,096 initially and has been subsequently increased in vSphere 6.0 updates. VMware doesn’t version control or have update dates on their vSphere 6 max config guide so you can’t see what has changed in vSphere point releases.

Minimum # of VVols per VM:

The minimum number VVols a powered-off VM with one virtual disk will have is 2 (config, data), once a VM is powered on a swap (vswp) VVol is created which increased that number to 3, the swap VVol is then deleted when a VM is powered off. For any additional virtual disks that a VM has there will be another VVol created. So a powered on VM with 3 virtual disks will have a minimum of 5 VVols associated with it (1-config, 3-data, 1-swap). For any VM snapshot that is created it will add at least one additional VVol per virtual disk (plus an additional if memory state is selected).

Average number of VVols per VM:

This one is a hard one to calculate as every environment varies, the big influencer here is the amount of VM snapshots that you normally maintain. You’ll typically have at least one per VM a night when your backup jobs are running, but that is removed after the backup completes. If you’re one that takes a lot of VM snapshots, calculate the average number per VM that are running concurrently then add 3 (min) to approximate your average number. Based on what I’ve seen and heard so far from customers I’d put the average number of VVols per VM at around 7-10. If you multiply that times that number of VMs that you plan to run on an array, this number should be well under the max VVols supported by the array.

Maximum # of VVols per VM.

This one is easy to calculate by simply doing the math. The maximum number of VVols a powered-on VM could have is around 2,000:

• 1 – config
• 1 – swap (one memory file per VM)
• 60 – data (max # of virtual disks per VM)
• 1,920 – snapshots (60×32 – max # of snapshots a VM can have is 32 times max # of virtual disks)
• 32 – snapshot memory state (based on max # of snapshots a VM can have, one for each)

Now this is the theoretical maximum, I don’t think you’ll see anyone running VMs with 60 virtual disks and 32 active snapshots.

Again the maximum number of VVols supported by a storage array is up to each vendor to decide what they want to support. The maximum number of VVols that vSphere can support is a pretty high number, the only maximum that VMware has published is the amount of VVols that can be bound to a single host which is 64,000. Hosts can see more than 64,000 VVols on a storage array but they can only bind (powered on VM) to a max of 64,000 of them at any one time. Given that a host has a max limit of 1,024 VMs, 64,000 VVols per host is a pretty high limit.

We can calculate the theoretical maximum of VVols in a vCenter cluster by doing the math:

• 60 – max # of virtual disks per VM
• 32 – max number of snapshots per virtual disk
• 10,000 – max number of VMs per vCenter cluster

This make the theoretical maximum in a vCenter cluster around 19 million total VVols, again I don’t think you will see any environments this size but you can see that very large environments will require an array that can support a large number of VVols. If we used 8 VVols per VM which is in my range of averages above, the numbers would come out to for different size environments around:

• 250 VMs – 2,000 VVols
• 500 VMs – 4,000 VVols
• 1,200 VMs – 9,600 VVols
• 3,000 VMs – 24,000 VVols
• 4,800 VMs – 38,400 VVols
• 6,000 VMs – 48,000 VVols

So when planning your migration to VVols keep these numbers in mind, do the math based on your own requirements and make sure your results are in line with your array vendors supported VVol limits.

The Register just published another article on VMware’s new Virtual Volumes (VVols) storage architecture and why I agree with a lot of it I thought I would provide some clarification and analysis of parts of it.

The first part hits on end user adoption of VVols, or rather the lack of, I’ve addressed that before so rather than rehash all those reasons for it you can go read that here, here and here. VVols is barely a year old and frankly I would expect low adoption of any new technology in less than a year, look at where VSAN was a year after it’s one year release.

Next it talks about the complexity of VVols, well of course it’s complex, it’s an entire new storage architecture that was many years in the making and required entire new T-10 specs to support it. You can go read VMware’s patent on VVols if you really want to know more about it or the new T-10 specs they submitted on bind/unbind operations and conglomerate LUNs. There are lots of new components with VVols and I don’t think you will hear any storage vendor say it was easy to implement, we’ve spent over 4 years developing our support at HPE.

In comparison look at just about anything in a VMware environment and you will see design complexity, hypervisor scheduling, virtual switches and memory management certainly aren’t easy to implement. The key distinction though is why it may have been complex for VMware and storage vendors to implement VVOls on the back end, it’s by design made to be less complex for vSphere admins. End users don’t have to look at complex architecture diagrams and component relationships for VVols, all they care about is what they see and do in the vSphere client.

Next it covers the VVol architecture which has a lot of new components, if you want to know more about that check out my VMworld session on VVols. It also talks about the number of VVols that arrays will have to support, I’m not sure all the math is accurate, a VM will always have a minimum of 2 VVols when powered off (config/data) and 3 when powered on (+vswp) and then additional VVols for snapshots as needed, I’ll do a separate post that covers this in more detail, the theoretical max VVols in a vSphere cluster is 19 million and the theoretical max for a single VM is around 2,000. We’ve had feedback from customers that are looking to do large scale implementations of VVols and based on their requirements they estimated about 7 VVols per VM.

On array VVol limits, this is more of a challenge for block arrays that are used to dealing with LUNs in the hundreds instead of sub-LUNs in the thousands. I’ve seen some vendor implementations as low as 1,000, with 3PAR it’s at 128,000. NAS arrays are already used to dealing with large amounts of objects so I have seen some vendors claim support in the millions.

On the array controller side it required changes as well as arrays controller and arrays had to understand protocol endpoints, bindings and special LUNs that have conglomerate status (admin LUN) with bindings to secondary LUN IDs (sub–LUNs). As far as VASA Providers go they most definitely don’t need to be external and Window’s based, it’s up to vendors to choose how to implement this, most have gone external but some have embedded the VASA Provider in the array. VVols does have it’s own HCL for vendors that are certified to support VVols, I just posted an update on this earlier this week.

The bottom line is, yes VVols is a complex architecture, yes vendor implementations vary, yes many vendors are way behind on support, yes it needs to mature more but I think overall the benefits VVols brings will overcome all this. A year from now I predict you will see much higher adoption of VVols as VASA 3.0 will address some current shortcomings, storage vendors will get caught up with VVols development and customers start to embrace it. Like any new technology it takes time for partners and the entire VMware ecosystem to catch up but once they do you will eventually see VVols as the de facto standard for external storage arrays.

If you are running a WordPress blog chances are it will be hacked at some point due to the many vulnerabilities that are constantly being uncovered in both WordPress and plug-ins. I’ve had hacks several times over the years to this blog and recently just came across another. There was nothing obvious to this hack and I probably would never have noticed it except for google search results for my site returning the disclaimer “This site may be hacked”. Google can detect hacks when it crawls a site and does a fetch as if it detects anything potentially malicious it will flag that site in search results.

I’ve become pretty WordPress savvy, I know my way around the core files, themes and the database very well and can typically spot anything that looks hacked. This one was a bit tough and took me at least 4-5 hours to uncover. As I mentioned the site looked normal but if you looked in the page code I could see spam text and links in there. So determined to find the cause I went through my usual troubleshooting process.

1. Get a ftp client like FileZilla and check the obvious files like index.php and .htaccess, I did find a few index.php’s scattered around which looked suspicious so I removed them all except for the one in the root directory but that didn’t fix it.
2. Look for .php files that don’t belong, I know the core WordPress files well so I know what shouldn’t be there, didn’t really find anything.
3. Check your wpconfig.php file, this one contains your database and other config info an dis a commonly hacked file, mine was OK.
4. Check your WordPress tables, I use PHPMyAdmin to browse the db tables, the WP_OPTIONS table is the main config table and is another commonly hacked table. I’ve had malicious rows injected in this table in the past, this time mine was OK. An easy way to look through all your table data is just export it to a .sql file and open it in a text editor.
5. Check your plug-ins, I disabled most of them and tested my site and the problem was still there. So that eliminated the plug-ins as the cause. One key thing to check though is to look for hidden plug-ins in the Active Plugins row in your WP_OPTIONS table.
6. Replace WordPress core files, I downloaded a copy of 4.4.2 and manually ftp’d the files in the wpadmin and wpincludes to the server to overwrite them with fresh copies, also the wp*.php files in the root directory. That didn’t help in my case.
7. Check your theme, I confirmed the theme was the culprit by switching to another theme and the hack disappeared. I didn’t want to replace my theme with a fresh copy as a I did some hacking and customization to it to get it exactly like I wanted it. I did examine all the files looking at date stamps and did notice one way newer then the other, it was a theme-search.php file, when I opened it there was a bunch of obfuscated text in it, definitely looked suspicious. I did have multiple backups so I compared the contents of them and that file was definitely not there before. So I deleted that file but the hack was still there. Next I copied all the theme files from the backup overwriting the current ones and that did the direct. I suspect some of the theme files were altered but in a way that preserved their data/time stamps.

Now that the hack was gone, I went to Google Webmaster Tools and requested a Fetch of my site which basically has the Google bots re-crawl it. A few hours later my hack message in Google search was gone. It’s a good idea to periodically check your blog for vulnerabilities, malicious code and hacks. Here’s some tools to help you with this by checking your site externally:

• Aw Snap – has a good collection of tools and information to both check your blog for malicious code and recover from hacks. The File Viewer will check a website for malicious redirects, malicious scripts and other bad stuff.
• Is It Hacked? -checks to see if your site is cloaked to GoogleBot, has spammy links, funny redirects, or otherwise appears to be hacked. They’ll fetch your site and analyze it for signs of an infection by doing multiple checks, from detecting spam links, hidden text, up to sophisticated cloaking.
• Sucuri SiteCheck – will check the website for known malware, blacklisting status, website errors, and out-of-date software.
• Google WebMaster Tools – add your site as a property and then you can see any security issues that Google has detected when they crawl your site, you can also request a re-crawl (fetch) of your site.

You should also check your site internally as well, external scanning can’t check your files and database so you need a security plug-in to scan internally. Here’s a couple good ones, I wouldn’t recommend having these all active simultaneously but sometimes one scanner will find something that another doesn’t so it’s good to activate and use them one by one and use the one that works best for you:

• Wordfence Security – I liked this one the best, has tons of customization option for scanning and real-time protection. It does vulnerability scanning, user monitoring, anti-virus, firewall, high speed cache and much more. It does a deep server-side scan of your source code comparing it to the Official WordPress repository for core, themes and plugins, it also checks your WordPress database.
• Theme Authenticity Checker (TAC) – searches the source files of every installed theme for signs of malicious code. If such code is found, TAC displays the path to the theme file, the line number, and a small snippet of the suspect code.
• Exploit Scanner – searches the files on your website, and the posts and comments tables of your database for anything suspicious. It also examines your list of active plugins for unusual filenames.
• Sucuri Security – a security suite meant to complement your existing security posture. It offers it’s users four key security features for their website, each designed to have a positive affect on their security posture.
• Anti-Malware Security and Brute-Force Firewall – searches for Malware, Viruses, and other security threats and vulnerabilities on your server and it helps you fix them.
• All In One WP Security & Firewall – will take your website security to a whole new level. this plugin is designed and written by experts and is easy to use and understand.It reduces security risk by checking for vulnerabilities, and by implementing and enforcing the latest recommended WordPress security practices and techniques.

VMware’s new storage architecture, Virtual Volumes (VVols) has been out for almost a year now but vendor support for it has been fairly sluggish. On Day 1 of the vSphere 6 launch only 4 vendors supported it (HPE, IBM, NEC & SanBlaze) with additional vendors slowly starting to support it after that. Most of the other big storage vendors (HDS, Dell, NetApp) supported it within a few months of launch but EMC stood out as the only big storage vendor that has really lagged behind on VVol support. That has changed as EMC finally has limited support for VVols only on their high end VMAX platform. I expect you’ll see it come to the VNX platform next as they already have a virtual appliance (vVNX) that supports VVols. As it stands today the vendors that support VVols are:

• HPE
• HDS
• IBM
• Dell
• NetApp
• EMC
• NexGen
• Tintri
• NEC
• SANBlaze
• Huawei
• Fujitsu
• DataCore

Note the protocols supported and array family/model support varies by vendor, check the VVol HCL for more information on what exactly is supported by each vendor. Notably absent with VVol Support is:

• Pure Storage
• Nexenta
• SolidFire
• Simplivity
• Nimble
• Tegile
• Nutanix

You’ll also note that there are no VSA’s that support VVols today, DataCore has support but they are not really a true VSA. Eventually you’ll see all the vendors come on board with VVol support, it certainly is no easy task to engineer this into arrays as evidenced by the slow trickle of supported vendors. If your vendor does not support it today check with them as I’m sure it’s on their roadmap and also ask about the important details such as which capabilities they will support, how they will scale (# of VVols) and how their VASA provider is implemented. For more information on VVol support in general and specific to each vendor check out my big VVol link collection.