Mar 09 2016

Interesting lawsuit against VMware regarding their use of Linux in ESXi

I came across this one a few weeks ago and thought it was interesting. In a nutshell one the biggest contributors to the Linux kernel (Christoph Hellwig) is trying to sue VMware over GPL (General Public License) violations from VMware taking the Linux kernel and modifying it for use with their own ESXi vmkernel. As you may know VMware has longed used the Linux operating system in both the ESX & ESXi hypervisors. ESX used a almost full-blown Linux install as part of the Service Console and ESXi used a smaller and streamlined Linux shell called BusyBox that serves as the console for ESXi. VMware essentially took the Linux code and modified it into their own module that they call vmklinux.

Well one Linux developer didn’t like what VMware was doing with the Linux code that they were using in ESXi and tried to get them to comply with the terms of the GPL licensing for Linux. VMware attempted to comply in some areas and refused in others which prompted the lawsuit. This diagram below from the Software Freedom Conservancy is a good depiction of how VMware blended Linux with their own vmkernel:

linux-vs-vmkernel_enFrom reading through the documents that are published it’s not completely clear what they are wanting from VMware. It looks like they are trying to get VMware to publish the complete source code for vmklinux or quit using it entirely. Apparently VMware has requested all court documents not be released to the public as it contains many examples of their source code. It appears that they have a pretty strong case against VMware but I can see this one dragging on for many years and by that time VMware may have dumped the Linux code for something else that they developed. You can read about the latest court updates that occurred a few weeks ago in this blog post from someone who attended it. You can read the whole FAQ on the case from the SF Conservancy that provides a good overview of the case and is an interesting read.

Mar 07 2016

Has VMware jumped the shark?

VMware has been in the spotlight a lot recently in regards to some business level news about the state of the company. On one hand you see them declared one of the best companies to work for, but on the other hand you see that they are laying off people and there seems to be a leadership revolving door. In this post I wanted to give my thoughts and perspective on this news about VMware.

Before I begin I want to point out how many years ago many people were stating that VMware might be the next Novell who’s Netware product  was the dominate player in the user networking/file sharing realm. I personally worked with a lot of Netware installations back in the day. The other players in that biz were Linux and OS/2, and of course Microsoft Windows (NT/Server). The major threat to Netware was Windows and by 2000, Novell was in major decline and eventually faded into tech obscurity.

The similarities between VMware and Novell are mainly that Microsoft is the biggest threat to their business and they were both market leaders by a large margin, but that’s where the similarities end as VMware has done an excellent job adapting and evolving and not becoming a one trick pony like Novell largely was. Despite the claims that Hyper-V would eventually crush vSphere as it was embedded into Windows, that hasn’t happened and VMware still enjoys a very big slice of the hypervisor market share and I don’t see that declining very much in the future.

Let’s start with VMware being named one of the best places to work by Fortune magazine. Not sure what the process is for picking that is, VMware is listed as #40 (Google is #1), I have never worked for VMware but from what I’ve seen and heard I think that is well deserved. They seem to treat their employees the right way, have a wonderful campus and people I know genuinely seem like they love working there. It’s nice to be recognized on this list but it really doesn’t have much bearing on how successful their business is. On a side note I was actually pictured in Fortune magazine back in 1995 when they did best states to work in before they picked the best companies to work at. That’s me front and center below.

fortune-me-cropSo if VMware is such a great company to work for why are so many executives abandoning ship? I’m sure they all had their own reasons but one of the biggest influences is undoubtedly the upcoming EMC-Dell merger. When companies merge or are bought out, management is usually the first thing that gets scrutinized and cut. If I was on the management team at EMC or VMware I would definitely be nervous and maybe proactively looking for greener pastures with better job security. It also doesn’t help that EMC has been a constant nagging parent to VMware which makes VMware management constantly prone to scrutiny and second guessing.

So management is a bit unstable at VMware, is that such a big deal though, what’s most important as John Troyer point’s out in his latest Tech Reckoning Dispatch is that the smart techies that are the worker bees at VMware are staying put. That’s whats most important to a company, their intellectual property, the management ladder will eventually sort itself out, the VMware ship is heading in the right direction and their are still plenty of good management types there. Personally I think the management revolving door is more FUD and less a legitimate concern. However I think VMware would of thrived a lot more had they remained a truly independent company without being owned by any other company.

So has VMware jumped the shark? Maybe, maybe not, I think their glory days are definitely over and it’s reality check and tighten the belts time at VMware. VMware recently announced their first ever large scale lay-offs, to me that’s not a big concern, it’s just them right-sizing the business as they prepare for tougher times. Microsoft still remains a legitimate threat along with other big players and of course the whole OpenStack movement has gained momentum. VMware definitely has it a lot tougher today then they did 5 years ago and one big reason for that is that almost all of their once valued technology partners are now mainly competitors to VMware. Back when VMware only sold a hypervisor product their partners built a very strong ecosystem around them which made for a powerful combination. Now that VMware has extended their line-up into just about every area including networking, storage, management, data protection, cloud, automation, etc. there is a lot more friction between VMware and it’s partners and of course competition.

VMware may have some tough days ahead of it but they have proven very resilient and will persevere and remain a dominate player. VMware is a software company, there overhead is mainly employees which is also their best asset. Once the acquisition process is over and management solidifies I think you’ll see VMware stay strong and remain focused without any chance of becoming a footnote in tech history like once dominate companies such as Novell, NetScape and Atari eventually became. Where VMware once swam in a mostly empty sea, they now swim in a sea full of sharks and only time will truly tell but my money is on VMware staying afloat and remaining strong and relevant.

Mar 04 2016

Some Friday vHumor for you…

I’ve seen Craigslist used for many different things but this is the first time I’ve ever seen someone try and get VSAN tech support on Craigslist. I can only imagine the responses they will get. While you’re there it’s always fun to read the Craigslist Best Of posts


Feb 29 2016

It’s the end of the road for these VMware products in 2016

I was browsing through the latest docs in the VMware KB this week and noticed one describing the end of availability of vSphere Enterprise, vSphere with Operations Manager (vSOM) Standard and Enterprise edition. That got me to browsing through VMware’s product lifecycle matrix which provides dates for every VMware products on GA, end of availability, end of support and end of technical guidance. Going through the huge list (VMware has so many products past and present), these are the key ones listed that are ending support in 2016:

  • VMware App Volumes 2.5, 2.6, 2.7 – 2016/12/09
  • VMware Data Recovery 2.0 – 2016/08/24
  • VMware ESXi 5.0 and 5.1 – 2016/08/24
  • VMware EVO:RAIL 1.0 and 1.1 – 2016/09/09
  • VMware EVO:RAIL 1.2 – 2016/12/10
  • VMware Fusion 7.x – 2016/03/03
  • VMware NSX for vSphere 6.1 – 2016/10/15
  • VMware vCenter Log Insight 2.0 – 2016/06/10
  • VMware vCenter Operations 5.8.5 – 2016/12/31
  • VMware vCenter Server 5.0 and 5.1 – 2016/08/24
  • VMware vCenter Site Recovery Manager 5.0 and 5.1 – 2016/08/24
  • VMware vCenter Update Manager 5.0 and 5.1 – 2016/08/24
  • VMware vCloud Automation Center 5.2 – 2016/12/10
  • VMware vCloud Networking and Security 5.5 – 2016/09/19
  • VMware vRealize Operations 6.0 and 6.1 – No earlier than 2016/12/09
  • VMware vSphere Data Protection 5.1 – 2016/08/24
  • VMware vSphere Replication 5.1 – 2016/08/24
  • VMware Workstation 11.x – 2016/06/02

Again these are the end of support dates for these products which means if you have SnS, VMware offers maintenance updates and upgrades, bug/security fixes and technical assistance until these dates. After these dates they have typically have a technical guidance phase which lastes for a fixed duration and provides support through their self-help portal only (no telephone support), they do not offer new hardware support, server/client/guest OS updates, new security patches or bug fixes during this phase. You can read more about these different product support phases here. So for those of you still on vSphere 5.0 & 5.1, the clock is ticking, better start planning your upgrades.

Feb 28 2016

vBlogger Spotlight: Duncan Epping


Top vBlog 2016 is about to kick off so I’m continuing my vBlogger Spotlight series that I started last year to shine the spotlight on several prominent bloggers in the community to give you some insight into their experiences with blogging. Today’s spotlight is on Duncan Epping, the un-disputed king of the bloggers and voted #1 top vBlog for 7 years straight. Duncan sets the bar pretty high for how a virtualization blog should be run and has demonstrated all the characteristics of a great blogger when it comes to longevity, frequency and quality. Over the 7 year history of Top vBlog nobody has come close to knocking him off that #1 spot and as he is showing no signs of slowly down it’s unlikely that anyone ever will. So without further ado enjoy a Q&A session with Duncan Epping, you can read the other vBlogger Spotlight series here.

What year did you start your blog?

[Duncan] I started my blog in December of 2007. Had been playing around with a theme and a logo though for a couple of weeks. First article was on 18 December, 2007.

What inspired you to start a blog?

[Duncan] I’d been active with regards to writing for a long time, but on a completely different topic: hardcore punk / metal core. I was working for a consultancy company in the Netherlands and needed a place to document my finding, share my problems and solutions. I was an avid reader of Mike Laverick and Scott Lowe’s blog and figured that I could do something like that. Considering I would normally write multiple CD reviews a week and do interviews, I figured when I stopped with the online community that this tech blogging would be a nice way to fill that gap. I never expected it to take off like this though.

So what inspired the blog name?

[Duncan] When I started blogging most people had a “vSomething” name. None of them really stood our for obvious reasons. I wanted something different, something that stood out, something that sounded cool and was easy to remember. Simply looked at my fav. bands and song titles, and this name came out. (Based on Old Yellow Bricks by Arctic Monkeys)

Describe your early blogging experiences and how you have evolved over the years?

[Duncan] To be honest, I’ve always blogged about the things that I am passionate about and things I encounter. Whether that was an issue discovered at a customer site, a new product or something cool I learned. I don’t think that has changed. My blog is still my blog and usually reflects what I am working on, or what I am thinking about. The big change over the years probably has been moving from shorter “I had this problem and this is how you fix it” articles to more “educational” pieces where I explain (short or long) how something works. But still, I very much enjoy doing the “problem/solution” articles.

What has kept you blogging over the years and not quitting at it?

[Duncan] Some seem to think that blogging is part of my job, it isn’t. Sure VMware highly appreciates all that I do, but there are no goals or even expectations when it comes to blogging. To be honest, it is who I am. I’ve been writing for such a long time now, I need it to stay sane.

When you are not blogging or working what do you enjoy doing?

[Duncan] I do many different things, but typically: running, watch my kids practice taekwon-do, crossfit/weightlifting, day trips to cities / museums etc. Basically a lot :)

What was your best experience or fondest memory related to blogging?

[Duncan] Don’t really have a fondest memory or “best experience”, but I can tell you that it is an awesome feeling when you walk through the VMware headquarters and you see an engineer reading your blog… Or you see your blog being referenced on an internal engineering wiki page. It is great to see that it doesn’t only help users / architects, but also helps people internally. I guess the biggest compliment though was when the HA and DRS team bought a bunch of copies of the HA/DRS deep dives and gave them to new employees and interns… they could not ask any questions until they finished the book. That was definitely the best compliment ever, and Frank and I smiled from “ear to ear” :)

If you had to choose a theme song for yourself what would it be?

[Duncan] You really had to ask me this question, man there are so many great songs I wouldn’t even know where to start. I also listen to so many different styles of music it is very hard to pick a song. But if I have to pick one, State of Love and Trust by the almighty Pearl Jam. Probably my all-time fav. song. I can listen to that one a million times in a row and it doesn’t bore me and I find myself always singing along the words “And I listen for the voice inside my head Nothin’, I’ll do this one myself…”

Any advice for others who are new to blogging?

[Duncan] Just do it, but if you start… Don’t do it because you want to be a vExpert, or because many others are doing it. Do it because you enjoy writing, you enjoy sharing knowledge / experience, do it because you want to learn. If you do it for other reasons chances are big you won’t last long, as it is a lot of work.

What’s your favorite tech gadget?

[Duncan] Not sure I have one really, I spend a lot of time on my phone checking the different social networks and keep up to date, but not sure it is my favourite as it also causes me to sometimes forget what is happening around me. I try to stay away from my phone as much as possible in the evening, but it is very tempting. Love/Hate relationship I guess.

Feb 27 2016

Worried about Leap Year bugs in 2016, VMware has you covered

bkpam2160482_12355920_1018130441584156_2007192750_oDate anomalies such as Y2K, Leap Years, and the infamous1970 bug can be disruptive if software code is not designed to handle them properly. You might remember the infamous VMware time bomb bug a few years back which made them look bad. In 2016 we once again have a Leap Year and VMware has gone out and tested the impact of that date across their entire product line to ensure nothing is impacted by this once every four year date occurrence. VMware has published the results of their testing in a KB article which appears to cover all of their products and which are unaffected by Leap Year. I expect they didn’t test every past version of these products but as Leap Year isn’t really new and occurs every four years I don’t expect you’ll see any issues regardless of which version of a VMware product you are on. So with this reassurance in mind hopefully you will sleep better Sunday night and not get woken up by any surprises.

Feb 26 2016

Why and how to find out if your I/O Adapter will support Virtual Volumes (VVols)

VMware’s new Virtual Volumes storage architecture required some changes to the SCSI specifications for block arrays to support the new sub-LUN model that VVols uses. Because of this it required I/O vendors to update their software to be able to support the new Protocol Endpoint component in VVols that is used as the data path between hosts and VMs (VVols) residing on storage arrays. Early on after VVols was released many I/O vendors were in catch-up mode to introduce support for this into their I/O adapter firmware. As a result there is a good chance that the I/O adapter that you are using may not work with VVols or may require a firmware upgrade to support it.

Let’s first look at what changed with VVols that required changes to be made to I/O adapter firmware to support VVols. A traditional block I/O adapter connects to a LUN on a storage array where your VMFS datastore is located. To connect to the LUN it simply needs the data path information (WWN) which includes HBA #, controller and of course the LUN ID of the volume associated with the VMFS datastore. You’ll see this in the vSphere client when selecting an I/O adapter with a syntax similar to vmhbaAdapter:CChannel:TTarget:LLUN (i.e. vmhba1:C0:T3:L1).

VVols introduces the concept of Secondary LUN IDs which is essentially an additional layer of LUN numbering that supports a lot more sub-LUNs than a array traditionally supports. The way this works is that a host will connect to a special Administrative LUN on the storage array via the Protocol Endpoint. This Admin LUN has no storage allocated to it and serves as a gateway to the sub-LUNs beneath it, it’s LUN ID is usually greater than 255 to identify it as a non-data LUN. A host cannot connect directly to a sub-LUN and must go through the Admin LUN to get to it. These Secondary LUN ID’s are provided to a host via the VASA Provider, so you can see why it is an important component. You can also see why direct to SAN backup is not supported with VVols as you cannot connect to a sub LUN without going through a ESXi host. The relationship between these components is outlined in the figure below:

VVol-archThe architecture is a bit complex and introduces additional identifiers beyond the LUN ID to connect to VVols on the array. These additional identifiers include secondary level IDs (SLLID), VVol IDs and Reference counts, the PE LUN still has a traditional WWN associated with it. Now while both block and NFS arrays utilize protocol endpoints and SLLIDs, the special Admin (PE) LUN only applies to block storage arrays, if you are using a NFS array with VVols this doesn’t apply as their is no special PE LUN.  With block arrays PEs are discovered via an in-band path using the standard SCSI command, REPORT_LUNS which reports the WWN of the PE, with NFS PEs are discovered via an out-of-band path using an API which returns the IP address and mount point of the PE. PE LUNs on block storage are recognized differently from traditional data LUNs as they have a special conglomerate bit set (LU_CONG).

VMware has patented this new storage architecture that they refer to as “Computer system accessing object storage system“, here are a few diagrams from the patent, if you want to deep-dive into this new architecture give the patent a read.

vvols-patent3-4vvols-patent1-2This new sub LUN architecture required VMware to submit update proposals to the T-10 committee for SCSI specifications to support it and the end result of all this was I/O adapters had to be updated to support this as well.

So know we know the why, let’s look at the how, as in how do I know if my I/O adapter will support VVols? Perhaps the quickest and easiest way is to simply look it up on the VMware HCL. If you go the VMware HCL and select I/O Adapters you will notice a new Feature there that you can select that is called “Secondary LUNID”. You can simply select that feature and select your I/O Adapter brand name, optionally the device type (i.e. FC) and then search as shown below.

VVol-hcl1In the results you will see the firmware versions needed to support VVols and links to any special drivers that might be needed for ESXi to support it as shown below.

VVol-hcl2From what I’ve seen there are some I/O Adapters that will work using the standard ESXi image, some require custom server ESXI images (i.e. HPE, Dell) and other that require you to download and install a driver into ESXi. You can see many of these in the vSphere Drivers/Tools download page.

One thing to note is if your I/O Adapter does not support VVols or does not have the firmware to support it you will see this error in your vmkernel logs:

Sanity check failed for path vmhbaX:Y:Z. The path is to a VVol PE, but it goes out of adapter vmhbaX which is not PE capable. Path dropped.

To check if your HBA is VVol capable you can run this command on your ESXi hosts: esxcli storage core adapter list , you should see Second Level Lun ID (SLLID) listed under the Capabilities column if the I/O Adapter supports VVols as shown below, VMware has a KB on this as well.

VVol-pe-check2Once you are sure your I/O Adapter has the required firmware and the proper ESXi I/O driver is present you can check from the host to see if it is able to recognize the Protocol Endpoint (Admin LUN). You’ll need to make sure that your VASA Provider is enabled and setup before you can do this though. Once that is setup you can use the esxcli command run from each host to see if the PE is visible to it, the syntax for the command is esxcli storage core device list –pe-only as shown below.

VVol-pe-checkIn the output that is produced if you see a value of “true” for “Is VVOL PE” that confirms that the host is able to connect successfully to the array PE to access VVols. Often times this will be false if your I/O adapter does not support VVols or VVols is not setup correctly.

So there you have it, it may seem a bit complicated but there is a good chance that your I/O Adapter already supports VVols and you really won’t have to do anything to start using it. It’s always a good thing to check though and make sure your I/O Adapters and storage arrays have the required firmware level to support VVols. Hopefully this gives you a better understanding of how VVols works under the hood and the relationships between hosts, protocol endpoints and VVols.

Feb 25 2016

Just how many Virtual Volumes (VVols) can a VM have?

This is a follow-up to my previous post on VVols commenting on The Register’s VVol article. I’ve seen this question come up often so I wanted to provide some detailed information on the minimum, average and maximum amount of VVols a VM can have. This has a direct impact on the scalability of VVols as the number of VVols that can be supported on a storage array varies by vendor, I’ve seen some as low as 1,000 and other over 100,000. Let’s first start with the number of VVols that can be bound to a single host, right now that number is 64,000, although I’m pretty sure that number was 4,096 initially and has been subsequently increased in vSphere 6.0 updates. VMware doesn’t version control or have update dates on their vSphere 6 max config guide so you can’t see what has changed in vSphere point releases.

Minimum # of VVols per VM:

The minimum number VVols a powered-off VM with one virtual disk will have is 2 (config, data), once a VM is powered on a swap (vswp) VVol is created which increased that number to 3, the swap VVol is then deleted when a VM is powered off. For any additional virtual disks that a VM has there will be another VVol created. So a powered on VM with 3 virtual disks will have a minimum of 5 VVols associated with it (1-config, 3-data, 1-swap). For any VM snapshot that is created it will add at least one additional VVol per virtual disk (plus an additional if memory state is selected).

Average number of VVols per VM:

This one is a hard one to calculate as every environment varies, the big influencer here is the amount of VM snapshots that you normally maintain. You’ll typically have at least one per VM a night when your backup jobs are running, but that is removed after the backup completes. If you’re one that takes a lot of VM snapshots, calculate the average number per VM that are running concurrently then add 3 (min) to approximate your average number. Based on what I’ve seen and heard so far from customers I’d put the average number of VVols per VM at around 7-10. If you multiply that times that number of VMs that you plan to run on an array, this number should be well under the max VVols supported by the array.

Maximum # of VVols per VM.

This one is easy to calculate by simply doing the math. The maximum number of VVols a powered-on VM could have is around 2,000:

  • 1 – config
  • 1 – swap (one memory file per VM)
  • 60 – data (max # of virtual disks per VM)
  • 1,920 – snapshots (60×32 – max # of snapshots a VM can have is 32 times max # of virtual disks)
  • 32 – snapshot memory state (based on max # of snapshots a VM can have, one for each)

Now this is the theoretical maximum, I don’t think you’ll see anyone running VMs with 60 virtual disks and 32 active snapshots.

Again the maximum number of VVols supported by a storage array is up to each vendor to decide what they want to support. The maximum number of VVols that vSphere can support is a pretty high number, the only maximum that VMware has published is the amount of VVols that can be bound to a single host which is 64,000. Hosts can see more than 64,000 VVols on a storage array but they can only bind (powered on VM) to a max of 64,000 of them at any one time. Given that a host has a max limit of 1,024 VMs, 64,000 VVols per host is a pretty high limit.

We can calculate the theoretical maximum of VVols in a vCenter cluster by doing the math:

  • 60 – max # of virtual disks per VM
  • 32 – max number of snapshots per virtual disk
  • 10,000 – max number of VMs per vCenter cluster

This make the theoretical maximum in a vCenter cluster around 19 million total VVols, again I don’t think you will see any environments this size but you can see that very large environments will require an array that can support a large number of VVols. If we used 8 VVols per VM which is in my range of averages above, the numbers would come out to for different size environments around:

  • 250 VMs – 2,000 VVols
  • 500 VMs – 4,000 VVols
  • 1,200 VMs – 9,600 VVols
  • 3,000 VMs – 24,000 VVols
  • 4,800 VMs – 38,400 VVols
  • 6,000 VMs – 48,000 VVols

So when planning your migration to VVols keep these numbers in mind, do the math based on your own requirements and make sure your results are in line with your array vendors supported VVol limits.

Feb 18 2016

Analyzing The Register’s latest article on VVols

The Register just published another article on VMware’s new Virtual Volumes (VVols) storage architecture and why I agree with a lot of it I thought I would provide some clarification and analysis of parts of it.

The first part hits on end user adoption of VVols, or rather the lack of, I’ve addressed that before so rather than rehash all those reasons for it you can go read that here, here and here. VVols is barely a year old and frankly I would expect low adoption of any new technology in less than a year, look at where VSAN was a year after it’s one year release.

Next it talks about the complexity of VVols, well of course it’s complex, it’s an entire new storage architecture that was many years in the making and required entire new T-10 specs to support it. You can go read VMware’s patent on VVols if you really want to know more about it or the new T-10 specs they submitted on bind/unbind operations and conglomerate LUNs. There are lots of new components with VVols and I don’t think you will hear any storage vendor say it was easy to implement, we’ve spent over 4 years developing our support at HPE.

In comparison look at just about anything in a VMware environment and you will see design complexity, hypervisor scheduling, virtual switches and memory management certainly aren’t easy to implement. The key distinction though is why it may have been complex for VMware and storage vendors to implement VVOls on the back end, it’s by design made to be less complex for vSphere admins. End users don’t have to look at complex architecture diagrams and component relationships for VVols, all they care about is what they see and do in the vSphere client.

Next it covers the VVol architecture which has a lot of new components, if you want to know more about that check out my VMworld session on VVols. It also talks about the number of VVols that arrays will have to support, I’m not sure all the math is accurate, a VM will always have a minimum of 2 VVols when powered off (config/data) and 3 when powered on (+vswp) and then additional VVols for snapshots as needed, I’ll do a separate post that covers this in more detail, the theoretical max VVols in a vSphere cluster is 19 million and the theoretical max for a single VM is around 2,000. We’ve had feedback from customers that are looking to do large scale implementations of VVols and based on their requirements they estimated about 7 VVols per VM.

On array VVol limits, this is more of a challenge for block arrays that are used to dealing with LUNs in the hundreds instead of sub-LUNs in the thousands. I’ve seen some vendor implementations as low as 1,000, with 3PAR it’s at 128,000. NAS arrays are already used to dealing with large amounts of objects so I have seen some vendors claim support in the millions.

On the array controller side it required changes as well as arrays controller and arrays had to understand protocol endpoints, bindings and special LUNs that have conglomerate status (admin LUN) with bindings to secondary LUN IDs (subLUNs). As far as VASA Providers go they most definitely don’t need to be external and Window’s based, it’s up to vendors to choose how to implement this, most have gone external but some have embedded the VASA Provider in the array. VVols does have it’s own HCL for vendors that are certified to support VVols, I just posted an update on this earlier this week.

The bottom line is, yes VVols is a complex architecture, yes vendor implementations vary, yes many vendors are way behind on support, yes it needs to mature more but I think overall the benefits VVols brings will overcome all this. A year from now I predict you will see much higher adoption of VVols as VASA 3.0 will address some current shortcomings, storage vendors will get caught up with VVols development and customers start to embrace it. Like any new technology it takes time for partners and the entire VMware ecosystem to catch up but once they do you will eventually see VVols as the de facto standard for external storage arrays.

Feb 17 2016

Another day, another blog hack – here’s how to scan and protect your WordPress blog

hackedIf you are running a WordPress blog chances are it will be hacked at some point due to the many vulnerabilities that are constantly being uncovered in both WordPress and plug-ins. I’ve had hacks several times over the years to this blog and recently just came across another. There was nothing obvious to this hack and I probably would never have noticed it except for google search results for my site returning the disclaimer “This site may be hacked”. Google can detect hacks when it crawls a site and does a fetch as if it detects anything potentially malicious it will flag that site in search results.

I’ve become pretty WordPress savvy, I know my way around the core files, themes and the database very well and can typically spot anything that looks hacked. This one was a bit tough and took me at least 4-5 hours to uncover. As I mentioned the site looked normal but if you looked in the page code I could see spam text and links in there. So determined to find the cause I went through my usual troubleshooting process.

  1. Get a ftp client like FileZilla and check the obvious files like index.php and .htaccess, I did find a few index.php’s scattered around which looked suspicious so I removed them all except for the one in the root directory but that didn’t fix it.
  2. Look for .php files that don’t belong, I know the core WordPress files well so I know what shouldn’t be there, didn’t really find anything.
  3. Check your wpconfig.php file, this one contains your database and other config info an dis a commonly hacked file, mine was OK.
  4. Check your WordPress tables, I use PHPMyAdmin to browse the db tables, the WP_OPTIONS table is the main config table and is another commonly hacked table. I’ve had malicious rows injected in this table in the past, this time mine was OK. An easy way to look through all your table data is just export it to a .sql file and open it in a text editor.
  5. Check your plug-ins, I disabled most of them and tested my site and the problem was still there. So that eliminated the plug-ins as the cause. One key thing to check though is to look for hidden plug-ins in the Active Plugins row in your WP_OPTIONS table.
  6. Replace WordPress core files, I downloaded a copy of 4.4.2 and manually ftp’d the files in the wpadmin and wpincludes to the server to overwrite them with fresh copies, also the wp*.php files in the root directory. That didn’t help in my case.
  7. Check your theme, I confirmed the theme was the culprit by switching to another theme and the hack disappeared. I didn’t want to replace my theme with a fresh copy as a I did some hacking and customization to it to get it exactly like I wanted it. I did examine all the files looking at date stamps and did notice one way newer then the other, it was a theme-search.php file, when I opened it there was a bunch of obfuscated text in it, definitely looked suspicious. I did have multiple backups so I compared the contents of them and that file was definitely not there before. So I deleted that file but the hack was still there. Next I copied all the theme files from the backup overwriting the current ones and that did the direct. I suspect some of the theme files were altered but in a way that preserved their data/time stamps.

Now that the hack was gone, I went to Google Webmaster Tools and requested a Fetch of my site which basically has the Google bots re-crawl it. A few hours later my hack message in Google search was gone. It’s a good idea to periodically check your blog for vulnerabilities, malicious code and hacks. Here’s some tools to help you with this by checking your site externally:

  • Aw Snap – has a good collection of tools and information to both check your blog for malicious code and recover from hacks. The File Viewer will check a website for malicious redirects, malicious scripts and other bad stuff.
  • Is It Hacked? -checks to see if your site is cloaked to GoogleBot, has spammy links, funny redirects, or otherwise appears to be hacked. They’ll fetch your site and analyze it for signs of an infection by doing multiple checks, from detecting spam links, hidden text, up to sophisticated cloaking.
  • Sucuri SiteCheck – will check the website for known malware, blacklisting status, website errors, and out-of-date software.
  • Google WebMaster Tools – add your site as a property and then you can see any security issues that Google has detected when they crawl your site, you can also request a re-crawl (fetch) of your site.

You should also check your site internally as well, external scanning can’t check your files and database so you need a security plug-in to scan internally. Here’s a couple good ones, I wouldn’t recommend having these all active simultaneously but sometimes one scanner will find something that another doesn’t so it’s good to activate and use them one by one and use the one that works best for you:

  • Wordfence Security – I liked this one the best, has tons of customization option for scanning and real-time protection. It does vulnerability scanning, user monitoring, anti-virus, firewall, high speed cache and much more. It does a deep server-side scan of your source code comparing it to the Official WordPress repository for core, themes and plugins, it also checks your WordPress database.
  • Theme Authenticity Checker (TAC) – searches the source files of every installed theme for signs of malicious code. If such code is found, TAC displays the path to the theme file, the line number, and a small snippet of the suspect code.
  • Exploit Scanner – searches the files on your website, and the posts and comments tables of your database for anything suspicious. It also examines your list of active plugins for unusual filenames.
  • Sucuri Security – a security suite meant to complement your existing security posture. It offers it’s users four key security features for their website, each designed to have a positive affect on their security posture.
  • Anti-Malware Security and Brute-Force Firewall – searches for Malware, Viruses, and other security threats and vulnerabilities on your server and it helps you fix them.
  • All In One WP Security & Firewall – will take your website security to a whole new level. this plugin is designed and written by experts and is easy to use and understand.It reduces security risk by checking for vulnerabilities, and by implementing and enforcing the latest recommended WordPress security practices and techniques.

Feb 16 2016

EMC arrives to the VVols party fashionably late

VMware’s new storage architecture, Virtual Volumes (VVols) has been out for almost a year now but vendor support for it has been fairly sluggish. On Day 1 of the vSphere 6 launch only 4 vendors supported it (HPE, IBM, NEC & SanBlaze) with additional vendors slowly starting to support it after that. Most of the other big storage vendors (HDS, Dell, NetApp) supported it within a few months of launch but EMC stood out as the only big storage vendor that has really lagged behind on VVol support. That has changed as EMC finally has limited support for VVols only on their high end VMAX platform. I expect you’ll see it come to the VNX platform next as they already have a virtual appliance (vVNX) that supports VVols. As it stands today the vendors that support VVols are:

  • HPE
  • HDS
  • IBM
  • Dell
  • NetApp
  • EMC
  • NexGen
  • Tintri
  • NEC
  • SANBlaze
  • Huawei
  • Fujitsu
  • DataCore

Note the protocols supported and array family/model support varies by vendor, check the VVol HCL for more information on what exactly is supported by each vendor. Notably absent with VVol Support is:

  • Pure Storage
  • Nexenta
  • SolidFire
  • Simplivity
  • Nimble
  • Tegile
  • Nutanix

You’ll also note that there are no VSA’s that support VVols today, DataCore has support but they are not really a true VSA. Eventually you’ll see all the vendors come on board with VVol support, it certainly is no easy task to engineer this into arrays as evidenced by the slow trickle of supported vendors. If your vendor does not support it today check with them as I’m sure it’s on their roadmap and also ask about the important details such as which capabilities they will support, how they will scale (# of VVols) and how their VASA provider is implemented. For more information on VVol support in general and specific to each vendor check out my big VVol link collection.

Feb 15 2016

Last call for blog-o-hol before Top vBlog 2016 begins

last-call-sponsors-300x133I’ve added lots of new blogs to my vLaunchpad but I’m sure there are some that I’ve missed. Every year I get emails from bloggers after the voting starts wanting to be added but once it starts its too late as it messes up the ballot. I’ve also archived a bunch of blogs that have not blogged in over a year in a special section, those archived blogs still have good content so I haven’t removed them but since they are not active they will not be on the Top vBlog ballot. In addition for the first time blogs must have at least 10 posts last year to be included on the ballot.

So if you’re not listed on the vLaunchpad, here’s your last chance to get listed. Please use this form and give me your name, blog name, blog URL, twitter handle & RSS URL. So if you haven’t submitted your blog here’s your last chance to do it so you don’t miss out on the recognition and cool commemorative coin that the top 50 blogs will receive. So hurry on up so the voting can begin, the nominations for voting categories will be opening up very soon.

Feb 14 2016

HPE snatches up one of the last remaining small backup companies

There have been a number of small backup companies that were formed years ago to fill the data protection void that was created by virtualization. Over the years many of these companies have been snatched up by larger companies and now one of the last remaining small companies has been bought out. Trilead, the creators of a popular free tool, VM Explorer was recently purchased by Hewlett Packard  Enterprise for an undisclosed amount. Before I go into that let’s take a look at the original players in that SMB backup space and look at where they are at now.

  • Vizioncore vRanger – Founded in 2002 and one of the original big 3 SMB backup products for virtualization, they were acquired by Quest Software in 2008 and then by Dell in 2012.
  • PHD Virtual – Founded in 2005 and another of the original big 3 SMB backup products, they were acquired by Unitrends in 2013.
  • Veeam Backup & Replication – Founded in 2006 and the last member of the big 3 SMB backup products, Veeam was the most successful and has grown so large they rival the big players like Symantec.
  • AppAssure – Founded in 2006 and acquired by Dell in 2012.
  • vSphere Data Protection – VMware’s foray into data protection, started life as vSphere Data Recovery (VDR) in 2009 and was a pretty limited and basic solution. VMware killed it off and replaced it with vSphere Data Protection (VDP) in 2012, VDP is based on EMC Avamar technology to provide a more robust and mature solution then VDR.

And that leaves us with Trilead which was founded in 2007, their VM Explorer product allowed you to do VM file management as well as backup and recovery of VMs. Their product started out fairly basic but has evolved a bit over the years and is very affordable (Pro Edition $790/Enterprise Edition $1460). The licensing cost is per site for use with unlimited hosts and virtual machines. They also offer a limited free edition that has been very popular. Their latest version 6.0 release added automatic backup testing, cloud backup, a new web interface as well as support for Microsoft Hyper-V 3.0.

So why did HPE snatch them up when they already have a backup product in-house, Data Protector. That surprised me at first, but when I thought about it for a while it made sense. while Data Protector is a enterprise level, very robust and highly scalable backup product, it’s a bit too complex and costly for the low-end SMB market. So it seems like rather than trying to bring Data Protector down to the low end they opted to purchase a product that is already there.

As far as I know Trilead was one of the smallest players in the SMB backup space so I’m sure the move was more about getting the Trilead intellectual property than it is about getting their customer base. It will be interesting to see what HPE does with it, I imagine at some point they will roll it into the Data Protector family name. The product should definitely get a development boost which will be good for current customers but I rather doubt that they will keep the price as cheap as it is today. The official word from HPE is below:


As part of our strategic focus and investment in virtualization, HPE has acquired Trilead, a next-generation provider of VM backup software that supports both VMware vSphere and Microsoft Hyper-V. Trilead provides HPE with a virtualized environment backup platform that is highly complementary with HPE’s Data Protector suite. HPE customers can expect the same level of service and support with the enhanced capabilities that Trilead’s technology brings to our portfolio.

Feb 13 2016

Top 12 essential plug-ins every WordPress blog should have

I’ve been using WordPress to host this blog for over 5 years and I’ve found plug-ins to be an invaluable way to enhance and protect my blog. There are literally thousands of plug-ins that you can add to WordPress that cover just about any functionality that you can think of. Trying to sort through them all and find the good ones can be a challenge though. I thought I would share what I use with this blog and get feedback from other bloggers on the plug-ins that they use as well.

  • Wordfence Security – Must have plug-in to secure your WordPress blog and protect it from hackers. A Swiss army knife of plug-ins it does vulnerability scanning, user monitoring, anti-virus, firewall, high speed cache and much more. Wordfence starts by checking if your site is already infected. We do a deep server-side scan of your source code comparing it to the Official WordPress repository for core, themes and plugins. Then Wordfence secures your site and makes it up to 50 times faster
  • UpdraftPlus Backup and Restoration – Another must have plug-in to protect your valuable content via scheduled backups.UpdraftPlus simplifies backups (and restoration). Backup into the cloud (Amazon S3 (or compatible), Dropbox, Google Drive, Rackspace Cloud, DreamObjects, FTP, Openstack Swift, UpdraftPlus Vault and email) and restore with a single click. Backups of files and database can have separate schedules.
  • Ninja Forms – Ninja Forms is the easiest way to build any form you need for your WordPress website. No longer mess with code or worry about made-up limitations. Create the form you want, when you want with a simple drag and drop interface provided by the very powerful Ninja Forms framework.
  • TablePress – TablePress allows you to easily create and manage beautiful tables. You can embed the tables into posts, pages, or text widgets with a simple Shortcode. Table data can be edited in a speadsheet-like interface, so no coding is necessary. Tables can contain any type of data, even formulas that will be evaluated.
  • Page Links To – This plugin allows you to make a WordPress page or post link to a URL of your choosing, instead of its WordPress page or post URL. It also will redirect people who go to the old (or “normal”) URL to the new one you’ve chosen.
  • Contact Form 7 & Contact Form DB – Contact Form 7 can manage multiple contact forms, plus you can customize the form and the mail contents flexibly with simple markup. The form supports Ajax-powered submitting, CAPTCHA, Akismet spam filtering and so on. The “CFDB” plugin saves contact form submissions to your WordPress database and provides and administration page and short codes to view and display the data.
  • Captcha by BestWebSoft – The Captcha plugin adds a captcha form into web pages. This captcha can be used for login, registration, password recovery, comments forms. It protects your website from spammers by means of math logic, easily understandable by human beings. All you need is to do one of the three basic maths actions – add, subtract and multiply.
  • Jetpack by – Jetpack simplifies managing WordPress sites by giving you visitor stats, security services, speeding up images, and helping you get more traffic.
  • WP-Optimize – WP-Optimize is an extensive WordPress database cleanup and optimization tool. It doesn’t require PhpMyAdmin to clean and optimize your database tables.
  • WPTouch – WPtouch is a mobile plugin for WordPress that automatically adds a simple and elegant mobile theme for mobile visitors to your WordPress website. Recommended by Google, it will instantly enable a mobile-friendly version of your website that passes the Google Mobile test, and ensure your SEO rankings do not drop due to not having a mobile-friendly website.
  • W3 Total Cache – W3 Total Cache improves the user experience of your site by increasing server performance, reducing the download times and providing transparent content delivery network (CDN) integration.
  • Yoast SEO – This plugin is written from the ground up to improve your site’s SEO on all needed aspects. While this Yoast SEO plugin goes the extra mile to take care of all the technical optimization, it first and foremost helps you write better content. Yoast SEO forces you to choose a focus keyword when you’re writing your articles, and then makes sure you use that focus keyword everywhere.

Feb 12 2016

New VMware White Paper: What’s New with VMware Virtual SAN 6.2

VMware released a new technical white paper to coincide with the announcement of VSAN 6.2, it goes into more detail on some of the new architecture features such as dedepulication, compression, erasure coding and QoS. The paper is written by Jase McCarty and Jeff Hunter from VMware and provides the missing technical details on VSAN 6.2 that you won’t find in the VMware announcements. Be sure and read my complete overview of VSAN 6.2 first and then give the technical paper a read and you’ll know everything you need to know about VSAN 6.2.

Older posts «

» Newer posts