I’ve spent a lot of time recently updating and refreshing all my sites and I recently migrated my vLaunchpad site to a new hosting provider, new WordPress version and to a new theme. I also took some time and went through the big backlog I had of adding new blogs to it. I’ve added at least 50 new blogs as I get ready for the upcoming top vBlog voting which will be kicking off in December. There are some cool changes in store for top vBlog this year so make sure your blog is listed on the vLaunchpad so it is included in the voting and you don’t miss out. I just updated the blog submission form so make sure you use that as it includes all the key info that I need to add your blog to it.

VMware recently acknowledged a vulnerability with their Transparent Page Sharing (TPS) feature that could potentially allow VM’s to access memory pages of other VMs running on a host. TPS is a host process that leverages the Virtual Machine Monitor (VMM) component of the VMkernel to scan physical host memory to identify redundant VM memory pages. The benefits of TPS are that it allows a host to reduce physical memory usage so you can squeeze more VMs onto a host as memory is often one of the most constrained resources on a host. TPS is basically the equivalent of storage de-duplication for RAM and works at the 4KB block level. You can read a full description on how it works in this classic VI3 memory white paper that refers to it as “Memory Sharing” or this more recent vSphere 5 white paper on memory management. Here’s a brief description of it and a slide on how it works:

[important]When multiple virtual machines are running, some of them may have identical sets of memory content. This presents opportunities for sharing memory across virtual machines (as well as sharing within a single virtual machine). For example, several virtual machines may be running the same guest operating system, have the same applications, or contain the same user data. With page sharing, the hypervisor can reclaim the redundant copies and keep only one copy, which is shared by multiple virtual machines in the host physical memory. As a result, the total virtual machine host memory consumption is reduced and a higher level of memory overcommitment is possible.[/important]

It’s important to note that the TPS feature is nothing new to vSphere having been first introduced with the VI3 release back in 2006, but apparently someone has now finally gotten around to try and successfully exploit it. Why is this a big deal? Because a virtualized architecture demands VM isolation, this is the most important security requirement for virtualization. Each VM guest running on a host must not be allowed in any way to access another VM guest. They must be kept in separate locked rooms with only the hypervisor possessing the keys to access all of them.

To illustrate this let’s use a real world scenario, imagine if you checked into a hotel, you expect privacy and isolation and to not have any other guests be able to come into your room. However your room has an adjoining shared door with another room but neither guest could get through it, only the hotel management can control the door. But somehow a guest has figured out a way to to open that door and get into your room and invade your privacy, that’s a pretty big deal wouldn’t you say.

VMware appears to be down-playing it as it obviously exposes a chink in their virtual armor, they have issued a KB article describing the vulnerability and giving guidance on how customers can disable TPS on their hosts. VMware doesn’t name the specific source that found the vulnerability in the KB article, they simply refer to it as “an academic paper”. The academic paper is entitled “Wait a minute! A fast, Cross-VM attack on AES” and was written by a group of individuals from Worcester Polytechnic Institute in 2014. It was funded by a National Science Foundation grant and it’s a pretty technical, head-spinning read with a lot of mathematical formulas. It’s worth taking a look at though as parts of it are more easily consumable and describe the attack scenarios and results. The overview of the paper is described as:

[important]In this work, we show a novel cache-based side-channel attack on AES that—by employing the Flush+Reload technique—enables, for the first time, a practical full key recovery attack across virtual machine boundaries in a realistic cloud-like server setting. The attack takes advantage of deduplication mechanism called the Transparent Page Sharing which is employed by VMware virtualization engine and is the focus of this work. The attack works well across cores, i.e. it works well in a high-end server with multiple cores scenario that is commonly found in cloud systems. The attack is, compared to [13], minimally invasive, significantly reducing requirements on the adversary: memory accesses are minimal and the accesses do not need to interrupt the victim process’ execution. This also means that the attack is hardly detectable by the victim. Last but not least, the attack is lightning fast: we show that, when running in a realistic scenario where an encryption server is attacked, the whole key is recovered in less than 10 seconds in non-virtualized setting (i.e. using a spy process) even across cores, and in less than a minute in virtualized setting across VM boundaries.[/important]

When I was going through the paper at the end of the paper in the references section one thing I noticed was that the authors of the paper had previously written a paper earlier in 2014 entitled “Fine grain Cross-VM Attacks on Xen and VMware are possible!“. So it appears based on that they worked to apply the theoretical research from that paper into a real-life attack which succeeded and prompted the 2nd paper to be written.

It’s important to note that this is not an easily exploitable vulnerability and the risk is super low so most environments should not really be impacted by it. VMware is being overly cautious though and will disable the feature by default in all upcoming releases of vSphere which includes:

• ESXi 5.5 Update release – Q1 2015
• ESXi 5.1 Update release – Q4 2014
• ESXi 5.0 Update release – Q1 2015
• The next major version of ESXi

In addition they will be issuing patches before that to address it sooner rather than later as follows:

• ESXi 5.5 Patch 3
• ESXi 5.1 patch planned for Q4, 2014
• ESXi 5.0 patch planned for Q4, 2014

All versions of vSphere back to VI3 are vulnerable to the exploit but VMware is only patching the 5.x versions of vSphere as the 4.x versions are no longer officially supported as of May 2014. Note these patches only disable TPS which is currently enable by default, they do nothing to fix the vulnerability, it will most likely take VMware some time to figure out how to make TPS work in a way that cannot be exploited. So if you disable TPS on your own you don’t really need the patch. VMware states in the KB article that “Administrators may revert to the previous behavior if they so wish” so it sounds like they are not too worried about it.

The benefits that TPS provides will vary in each environment depending on VM workloads so if you are really paranoid about security you will probably want to disable it. You can view the effectiveness of TPS in vCenter by looking at the “shared” and “sharedcommon” memory counters to see how much you benefit from it. You can disable TPS in your current environment by changing advanced settings on each host as described in the KB article, updating the setting is pretty simple but having it take effect is tedious work:

1. Log in to ESX\ESXi or vCenter Server using the vSphere Client.
2. If connected to vCenter Server, select the relevant ESX\ESXi host.
3. In the Configuration tab, click Advanced Settings under the software section.
4. In the Advanced Settings window, click Mem.
5. Look for Mem.ShareScanGHz and set the value to 0.
6. Click OK.
7. Perform one of the following to make the TPS changes effective immediately:
   1. Migrate all the virtual machines to other host in cluster and back to original host.
   2. Shutdown and power-on the virtual machines.

While the impact and exposure may be minimal with this, the fact that someone has finally cracked those solid walls that have stood between VMs is a big deal. I’ve written about it previously in my post on Escaping the Cave, VMware officially describes this as “inter-process side channel leakage” and mentions this in the KB article:

[important]Side channel attacks that exploit information leakage from resources shared between processes running on a common processor is an area of research that has been explored for several years. Although largely theoretical, techniques are continuously improving as researchers build on each other’s work. Although this is not a problem unique to VMware technology, VMware does work with the research community to ensure that the issues are fully understood and to implement mitigation into our products when appropriate[/important]

Note they mention these types of vulnerabilities have been “explored” for years, meaning lots of people have been looking for a way to penetrate those walls as this is the holy grail type of hack of a virtual environment. Imagine if someone compromised a less critical VM through one of the many thousands of OS and application vulnerabilities that exist. At least the damage and exposure is contained within that one VM, but if some one could somehow use that compromised VM as a launchpad for attacking other VMs on a host that’s a real big deal. The fact that most of these types of vulnerabilities have been theoretical up until now really exposes VMware and their security foundation.

Overall vSphere is a very secure platform and has stood the test of time without any major issues which is a testament to how seriously they take security in vSphere. VMware will no doubt learn from this one and work to improve the security in vSphere to make it even better. However the circumstances of this one resulted from them trying to be more efficient with hypervisor resources by mixing VM data together. They will definitely need to be careful going forward as they continually look for new ways to optimize vSphere so they do it in a secure manner and do not risk exposure between VMs.

If you want to find out more about memory management in vSphere check out some of these links:

• Understanding Virtualization Memory Management Concepts (Great 2010 VMworld presentation from Kit Colbert)
• VMworld 2012: Session VSP1729 – Understanding Virtualized Memory Performance Management (Kit Colbert again)
• Understanding Memory Resource Management in VMware vSphere 5.0 (VMware tech paper)
• The Role of Memory in VMware ESX Server 3 (VMware tech paper)
• VMware vSphere 5 Memory Management and Diagram (VMware KB video)

Have you ever had that awful sick to your stomach, oh shit feeling when you just realized you lost a lot of important data whether it be photos, documents or other important stuff that can’t easily be replaced? It sucks doesn’t it, usually it takes just one instance like that to inspire us to start taking backups seriously. Unfortunately though it won’t bring back what you lost. Backups are one of those things that many people don’t think about especially when they store data on a location that is hosted on the internet.

There are a great many people that are blogging about virtualization these days and most of them are using WordPress as their platform of choice to do it. WordPress is an ideal platform for blogging but all that hard work you put into blogging could be wiped out if you don’t properly backup your WordPress site.

But doesn’t my hosting provider backup my site?

You should never trust that your hosting provider is backing up your website, many of them do not backup your content and if they do they probably do not guarantee them. Some hosting providers will offer a backup option as a paid add-on service. In addition they usually are not backing up your WordPress MySQL database which contains much of your valuable content. Take a look at this notice from my hosting provider, I think you’ll find the policy is similar with whatever provider you use, if you don’t know check with them.

How often should you backup your site?

Depending on how often you blog you should backup your WordPress instance at least once a month. If you are blogging several times a week you should probably do it daily or weekly. You should also do a backup before you upgrade WordPress to a newer version or update your plug-ins. Just like you do in the data center you should also plan on preserving older backups for as long as possible as often you may have something corrupted or malicious content that has been that way for a while that you need to go back a while to find a clean copy.

What should you backup?

With WordPress there are two main sets of data that you need to back up to ensure all your content is backed up and you can easily recover if needed, your WordPress files and your WordPress database. When you install WordPress on your website there are hundreds of files that get copied to specific directories that contain the complete WordPress web application. A new install of WordPress is only about 16MB in size with around 1,100 files but as you add content that will grow. Your WordPress database is typically hosted on a MySQL database that is installed and managed by your hosting provider. The WordPress database has many tables that store configuration and content for your WordPress website.

How do I back it up?

So now we know what needs to be backed up, how do we actually do it? There are several ways that you can backup WordPress:

1. Manually by copying all your files to a PC using FTP and then doing a SQL export of your WordPress MySQL database and copying that to your local PC as well.
2. Automatically by using some type of PHP script that can run scheduled on your hosting provider server using a scheduling tool like cron.
3. Automatically using a WordPress plug-in designed to backup WordPress.
4. Some hosting providers will do it if you pay for a add-on backup service.

Doing a manual backup

This method is OK for ad-hoc backups but it can be tedious to do on a frequent basis and it can easily slip your mind. I used this method for years, I did it nowhere near as often as I should of and I got lucky a few times were I almost lost a lot of data. To backup WordPress manually you will need to copy the all appropriate files and directories from the hosting provider web server to your local PC or even better to a cloud storage platform like Dropbox. Below are the files and directories that are come with a new install of WordPress.

The easiest way to move the files is to use an FTP application to copy them from your old server to your local PC and then to your new server. It’s also a good idea to periodically do this to backup WordPress. If you need a FTP client, check out FileZilla which is a free open source application. You may need to setup a FTP username/password on your hosting site before you can connect to it. Create a new site in FileZilla and give it a name (i.e. mywebsite-old), use the IP address or DNS name of your website and then enter in your login credentials. Once you connect to your web server you’ll see the directory listing of the contents, what you see will usually vary by hosting providers, some providers partition you off so you don’t see much of the web server files.

You may not need all the files you see to backup WordPress but its best to copy everything to a sub-directory on your PC so you do have a full backup just in case. In the figure above you can see the 3 WordPress directories that you need to copy for sure along with all the files that start with “wp” in the root directory. I’ve manually copied things to my site in the past (i.e. images) which I copy also. Other files that are part of the hosting platform you typically don’t need to bring over but it doesn’t hurt to copy them any way. Once you’ve copied everything to your PC it’s time to move on to the next step, backing up your MySQL database.

Your WordPress database is typically hosted on a MySQL database that is installed and managed by your hosting provider. The WordPress database has many tables that store configuration and content for your WordPress website. You can find a complete description of the database tables here. Log into your hosting provider control panel for your website and you should see a link for database management via phpMyAdmin which is a free software tool that is written in PHP that is used to administer MySQL over the Web. Once you launch phpMyAdmin you should be prompted for a username and password to connect to your database. You probably won’t know it or remember it but you can easily look it up by opening the wpconfig.php file that you copied to your PC as part of the backup in a text editor like Notepad and looking for the MySQL section which will contain your MySQL username/password.

Note some hosting providers may require you to whitelist your IP address to do remote MySQL administration, if they do there should be a section in your hosting control panel to put in your IP address. Once you are logged into phpMyAdmin you want to Export your database, click the Export link. You may be prompted for a quick export where you don’t need to enter a lot of options which will work just fine, if you do get a selection screen you typically can just use the defaults and then just hit Go and it will ask you for a location for the file on your PC and then begin the Export. It shouldn’t take more than a few minutes. Here’s how the Export screen looks with my hosting provider:

Once you have completed this it will create a .SQL file on your local PC that you should save with the other WordPress files that you copied. You now have everything you need to restore WordPress if needed by copying all the files you backed up back to the server and Importing the .SQL file back into WordPress, see my other post on moving to a new hosting provider for more info on how to do that.

Also note some hosting providers like GoDaddy provide a link in their control panel to kick off a database backup so you don’t have to go into PHPMyAdmin. They dump the resulting .SQL file in a db backup directory on your website, just make sure you copy the file from there to your local PC.

Doing an automatic backup with a PHP script and cron

I’m not going to go into much detail on this method, it can be a bit technical to setup. There are some WordPress plug-ins available that will make this easier to setup. Your hosting provider control panel should have a section to setup and manage cron jobs like below:

You then need to configure the scheduling and action for the cron job for the cron job to perform like below:

Again look for some WordPress plug-ins that support cron or a PHP script that is written to backup WordPress MySQL databases. If you are feeling adventurous you could also write your own PHP script. Here’s one I found by searching the internet. Some scripts may only backup the database so make sure you know what the script is doing and where it is storing your backups.

Doing an automatic backup with WordPress plug-in

This is probably the easiest and most convenient way to backup your WordPress site. There are many plug-ins available that will automate the backup of both your WordPress files and MySQL database so you don’t have to do anything but install the plug-in and configure it. You can search through the WordPress plug-in directory and you’ll see many of them. The one I ended up using which had 4.8 out of 5 stars and is free but has some paid add-on’s is UpDraftPlus Backup & Restoration. You can go to their add-on page which contains a big list of add-on’s and pricing for each that expand the flexibility, functionality and backup targets supported. Another popular WordPress backup plug-in is BackWPup.

You can backup your site just fine with the free version but it only puts the backup files on your hosting web server. If you want to use other backup destinations like Dropbox, Amazon E3, Rackspace, Google Drive and more it will cost you about $10 for each. If you do backup to your website only just remember to copy those files off periodically to somewhere safe.

Install UpDraftPlus like you would any other WordPress plug-in, once activated go into Settings, UpDraftPlus Backups in WordPress to setup your backup jobs and you will be at the main screen:

Here you can see your backup status and quick actions for backing up and restoring. It’s best to click on the Settings tab first to configure backup schedules and retention. Since most hosting providers now provide unlimited space don’t be afraid to retain a lot of backups.

You can also specify what files to backup, database encryption if you are really paranoid, reporting, remote storage options and other advanced settings. Note by default with the free version it will not backup your core WordPress files (i.e. wp-admin) but unless you customize yours you won’t have to worry about those as you can easily download those again if needed. All the files specific to your WordPress site are in the Themes and Plugins directories.

Once the backup runs you can look at the log files to see everything that occurred during the backup, it’s not something you’ll need to do regularly but I looked as I was curious. If you are using the free version which puts the backups into your WordPress directory you’ll see a new sub-directory under wp-content called updraft that contains your backup files all zipped up, make sure you backup these backup files somewhere else!

What about Backup as a Service?

If you prefer not to deal with your backups at all you can outsource them to a company that provides backup services for WordPress. Note both of these companies below backup both your WordPress files and database.

One such company that does this is blogVault. Their Basic plan starts at $9/month for backing up a single site and retain 30 days of backups. If you have more than one site you they have a Plus plan for $19/month that will backup 3 sites. It works by installing their WordPress plug-in on your site and then their server automatically contacts the plugin everyday to backup new changes to your site.

Another company that provides WordPress backup services is Backup Buddy. Their Blogger plan rate is $80/year for backing up 2 sites with 1GB of backup space available. Presumably with that much space available you could store more than 60 days of backups with them. They also have a Freelancer plan available for $100/year for up to 10 sites with 1GB of backup space. Again it works by installing their WordPress plug-in on your site and then configuring it, they have a video that demonstrates this process.

Summary

And that’s all there is to it, pick the service/method/plug-in that works best for you. For me I’ve setup UpDraftPlus and will also periodically do manual backups as well. Regardless of how you do it the important thing is that you are backing up your WordPress site which contains all your hard work that you do not want to ever take the chance of losing.