Virtualizing Domino – Pt.1 – The Journey Begins

I started doing Lotus Domino server administration back in 1994 on version 3.0, about 5 years ago the company I work for decided to outsource it to a hosting provider. While it was outsourced I was still pretty involved with the management and administration of the environment. We recently decided to bring it back in-house due to cost-savings and because we have a DR site that we can leverage for business continuity. I was the project lead for transitioning the environment to the provider and I am again the project lead for bringing it back. That means I have to architect our environment, build it, plan and perform the migration and make sure everything works smoothly. Since we virtualized most of our environment several years ago it’s only natural to also virtualize the new Domino environment. This series of blog posts will cover the many steps of the project and will focus on the virtualization aspect of it. The project will likely take about 3-4 months to complete and right now I’m at the very beginning stages. So I’ll begin with a recap of where the project is at right now.

Planning & architecting

This is the most importance phase and one you can’t afford to screw up, I chose to virtualize Domino so it was necessary to make sure I had a virtual environment that could handle the taxing resource demands that Domino servers require. Our environment is about 1,500 total users and somewhere between small and medium, we have many access methods including LAN, WAN, internet and both Notes Client, web-based, Blackberry and other smartphone users. I planned for 2 clustered servers at our corporate HQ, 1 server in the DMZ for Notes Client passthru, Lotus Traveler and SMTP, 1 Blackberry Enterprise Server and 1 server at our DR warm site. Web access to email from the public internet was a requirement so the challenge was trying to find a secure solution for that. I focused on 2 solutions for that, using a hardware reverse proxy device in the DMZ and using a server running reverse proxy software. I looked at hardware solutions from F5 Networks who make very nice hardware that is very customizable and integrates well with Domino, unfortunately we didn’t have the budget for it though. So the solution I chose was using IBM Websphere Secure Proxy Server running on a Windows server in the DMZ.

Next it was time to find hardware to use, I needed one new virtual host at corporate HQ, one at the DR site and more storage. We have a Hitachi AMS-500 SAN (2GB fiber) that we could add storage to as an option, but this was expensive and didn’t get us much storage for our budget. Instead I looked at lower cost iSCSI solutions, in particular the MSA line from HP, they are very affordable, easy to use and perform very well. My biggest concern though was making sure it had enough performance for the very intensive disk I/O demands that Domino had. So I did some research looking at IOPs numbers for the MSA and comparing them to Iometer tests that I ran inside a VM running on the AMS-500. I found that the MSA had equal or better performance compared to the AMS-500 so I chose that instead. We were able to obtain an iSCSI MSA with dual controllers, an extra shelf loaded with 450GB drives for a total of about 10TB raw storage and were able to stay in our budget. For servers we went with HP DL-385 G6 servers with AMD 6-core processors.

Now with Domino there are a couple of important hardware considerations that you should be aware of. First is Domino loves memory, it will use everything you give it as it does a lot of caching. So having an adequate amount of memory is a must. The next is with CPU’s, Domino  takes full advantage of the new AMD RVI and Intel EPT  technology in newer processors so having this is a must. In fact this can make a huge difference in performance compared to processors without it. Finally when it comes to virtualization using vSphere was a must, there have been numerous performance improvements in vSphere that are very beneficial to Domino. IBM has published some tech notes in the past about high CPU usage on virtualized Domino servers compared to physical ones with identical workloads. They point out that you should make sure you have suitable hardware and the proper architecture but also seem to place the blame on the overhead of the virtualization layer. They go as far as to not recommend virtualizing larger Domino workloads. Well the combination of newer CPU technologies and vSphere now make that possible and allow almost any Domino workload to be successfully virtualized. So to summarize:

• Only use CPUs with AMD RVI & Intel EPT
• Have plenty of memory, don’t plan on using overcommitment
• Make sure your storage can keep up with Domino
• Make sure you are using vSphere and not VI3

Next up is building the hardware including the MSA and vSphere servers and configuring the iSCSI storage and networking. Some additional bonuses with using vSphere is the many improvements to iSCSI including support for Jumbo Frames, faster initiators and support for multi-pathing. So stay tuned for part 2…