Welcome to vSphere-land!

I started doing Lotus Domino server administration back in 1994 on version 3.0, about 5 years ago the company I work for decided to outsource it to a hosting provider. While it was outsourced I was still pretty involved with the management and administration of the environment. We recently decided to bring it back in-house due to cost-savings and because we have a DR site that we can leverage for business continuity. I was the project lead for transitioning the environment to the provider and I am again the project lead for bringing it back. That means I have to architect our environment, build it, plan and perform the migration and make sure everything works smoothly. Since we virtualized most of our environment several years ago it’s only natural to also virtualize the new Domino environment. This series of blog posts will cover the many steps of the project and will focus on the virtualization aspect of it. The project will likely take about 3-4 months to complete and right now I’m at the very beginning stages. So I’ll begin with a recap of where the project is at right now.

Planning & architecting

This is the most importance phase and one you can’t afford to screw up, I chose to virtualize Domino so it was necessary to make sure I had a virtual environment that could handle the taxing resource demands that Domino servers require. Our environment is about 1,500 total users and somewhere between small and medium, we have many access methods including LAN, WAN, internet and both Notes Client, web-based, Blackberry and other smartphone users. I planned for 2 clustered servers at our corporate HQ, 1 server in the DMZ for Notes Client passthru, Lotus Traveler and SMTP, 1 Blackberry Enterprise Server and 1 server at our DR warm site. Web access to email from the public internet was a requirement so the challenge was trying to find a secure solution for that. I focused on 2 solutions for that, using a hardware reverse proxy device in the DMZ and using a server running reverse proxy software. I looked at hardware solutions from F5 Networks who make very nice hardware that is very customizable and integrates well with Domino, unfortunately we didn’t have the budget for it though. So the solution I chose was using IBM Websphere Secure Proxy Server running on a Windows server in the DMZ.

Next it was time to find hardware to use, I needed one new virtual host at corporate HQ, one at the DR site and more storage. We have a Hitachi AMS-500 SAN (2GB fiber) that we could add storage to as an option, but this was expensive and didn’t get us much storage for our budget. Instead I looked at lower cost iSCSI solutions, in particular the MSA line from HP, they are very affordable, easy to use and perform very well. My biggest concern though was making sure it had enough performance for the very intensive disk I/O demands that Domino had. So I did some research looking at IOPs numbers for the MSA and comparing them to Iometer tests that I ran inside a VM running on the AMS-500. I found that the MSA had equal or better performance compared to the AMS-500 so I chose that instead. We were able to obtain an iSCSI MSA with dual controllers, an extra shelf loaded with 450GB drives for a total of about 10TB raw storage and were able to stay in our budget. For servers we went with HP DL-385 G6 servers with AMD 6-core processors.

Now with Domino there are a couple of important hardware considerations that you should be aware of. First is Domino loves memory, it will use everything you give it as it does a lot of caching. So having an adequate amount of memory is a must. The next is with CPU’s, Domino  takes full advantage of the new AMD RVI and Intel EPT  technology in newer processors so having this is a must. In fact this can make a huge difference in performance compared to processors without it. Finally when it comes to virtualization using vSphere was a must, there have been numerous performance improvements in vSphere that are very beneficial to Domino. IBM has published some tech notes in the past about high CPU usage on virtualized Domino servers compared to physical ones with identical workloads. They point out that you should make sure you have suitable hardware and the proper architecture but also seem to place the blame on the overhead of the virtualization layer. They go as far as to not recommend virtualizing larger Domino workloads. Well the combination of newer CPU technologies and vSphere now make that possible and allow almost any Domino workload to be successfully virtualized. So to summarize:

• Only use CPUs with AMD RVI & Intel EPT
• Have plenty of memory, don’t plan on using overcommitment
• Make sure your storage can keep up with Domino
• Make sure you are using vSphere and not VI3

Next up is building the hardware including the MSA and vSphere servers and configuring the iSCSI storage and networking. Some additional bonuses with using vSphere is the many improvements to iSCSI including support for Jumbo Frames, faster initiators and support for multi-pathing. So stay tuned for part 2…

Having experienced several complete data center power outages I’ve learned some important lessons over the years when it comes to virtualizing your infrastructure. Today I experienced another power outage and my previous experiences ensured that I was ready for this one and I thought I would share some tips:

• DNS is the most critical component in your environment, for almost anything to work properly you need a DNS server up first. If you have all your DNS servers virtualized and they are on shared storage its going to be very difficult to bring them up because everything else that needs to come up first usually relies on DNS. Therefore you should make sure you have at least one DNS server on the local storage of one of your hosts, that way you can get it up early on and not have to wait for your shared storage to come up. If you want to go a step further keep one as a physical server also, so you can quickly and easily get DNS up right away.
• Active Directory, DHCP & any other authentication servers are also very important, having a workstation up and running is very handy so you can centrally connect to hosts and power things back on. If you’re using DHCP for your workstations and servers you again will want one up as soon as possible so you can get them on the network and get going. Not having an Active Directory server available can make Windows servers take a very long time to boot. So again, keep an AD & DHCP server on local storage or on a physical server so you can quickly get them up as soon as possible. AD, DNS & DHCP are critical to a Windows environment and without them available you’ll find that the rest of your environment is mostly useless.
• Know your ESX command line, if your vCenter Server and other workstations are not available you’ll need to start VM’s using the command line. Even if your DNS server is on a local VM you won’t be able to start it without the vSphere Client. Therefore you’ll have to log into the ESX console and manually start it, if you don’t know the command to do this that could be a problem. Keep a cheat sheet by your hosts with the basic commands that you’ll need like vmware-cmd to get things up and running using the console.
• Know your host IP addresses, if DNS is not up yet you won’t be able to connect to your hosts using putty or the vSphere client using their host names, you probably won’t know their IP address and without a DNS server can’t look them up. Therefore keep a list of your host IP addresses so you can use that to connect to them.
• Know how to re-scan storage, your hosts may come up before your shared storage, once your shared storage is up you’ll need to rescan from your hosts so they will see it and you can restart the VM’s on it. You can do this using the vSphere client by clicking on Configuration, Storage Adapters, selecting your HBA and clicking the Rescan button and then select search for new devices. You can also do this using the command line esxcfg-rescan utility.
• Make sure you know where your datacenter keys are if you use a electronic card scanner to open your doors. Most systems are placed in the datacenter and if the power goes out your doors are not going to work. There is nothing worse than running around trying to find keys in a crisis to get into the datacenter. And make sure you don’t keep the keys in datacenter or you’ll have to break down the door to get in. (thanks Tony DiMaggio for reminding me about this one)

Being prepared is critical in crisis situation to ensure you can react quickly to get things back up and running. Sometimes it takes a crisis to point out any shortcomings that you may have in your environment but thinking ahead can save you from big headaches later on.

I’ve been maintaining the vLaunchpad using Dreamweaver which has been difficult at times so I thought I would explore other options. I’ve resisted using Wordpress as I had for my main site because the page layout is kind of unique and not easily re-produced in Wordpress. Well after much theme hacking I think I have it ported over to Wordpress pretty well. I’ve done extensive CSS modifications to make sure I would fit as much information as possible in one area and not have to resort to lots of scrolling or multiple pages. It’s not done yet but it’s getting close, I still need to add all the blog links and do some other work to it. I’ve also moved it to the clouds, which is GoDaddy’s new hosted grid computing offering. So let me know what you think, I’m open to all your suggestions, tell me what you like, don’t like and want to see added to it. One thing you’ll notice is I added all of Hany Michael’s great vDiagrams to the sidebar for easy access to them.

So go check out the new version of the vLaunchpad and let me know what you think, just remember it’s not done yet and I have lots of changes to make to it. Thanks to the Netherlands gang (Duncan, Eric & Gabe) for their early feedback that helped me get the design right.

Due to cuts and other circumstances I will no longer be blogging at SearchVMware.com’s Virtualization Pro blog. While exploring other blogging opportunities I decided to come right back where I started, this website, vSphere-land.com. So besides my gigantic link collection you’ll be seeing lots of new content here including tips, news, reviews, opinions, top 10 lists and much more. For starters I’ll be doing a series documenting a new project from start to finish that I’m the lead on which is a Lotus Domino virtualization project.  So stay tuned for lots of great content coming from this site.

As I previously mentioned two randomly chosen voters in the top blog survey would win a copy of TrainSignal’s vSphere DVD training course. I exported the voting results to Excel and then used a random number generator to pick two row numbers and the winners are:

• mcgivern_jon at emc dot com from Westborough, MA
• brian_alexander at mentor dot com from Longmont, CO

David Davis from TrainSignal will be contacting you, thank you for your participation in the voting!

Here’s a table with the vote totals for all the VMware/virtualization blogs included in the survey, there were a few others that only had 1 or two votes that I didn’t include. Total votes includes any vote a blog received whether it was 1 through 10. Total points is the sum of each vote times it’s appropriate weight for the position (#1 vote = 10 points, #2 vote = 9 points, all the way down to a #10 vote = 1 point).

Well I’m not saying, watch the presentation to find out, I’ll be doing another post later on with the full results as well as some detail on the process I used to sort through the votes and score them to determine the winners. Congratulations to all the winners, there were many newcomers to the top 25 blog roll. I’ll be sending you graphics that you can display on your website to reflect your achievements. The winners of the TrainSignal vSphere DVD training course will be announced in my next post. Click here to see the results presentation in a new window

Voting for the top VMware/virtualization blogs will close this Friday night , 1/15 at 10:00pm MST so if you haven’t voted yet be sure and get your vote in. Over 850 votes have been cast already and the Top 25 blogs will definitely look different than it is right now. The results will be announced on Monday after I spend the weekend going through the data, I’ll be publishing a Powerpoint presentation converted to Flash with the voting details and the winners. So stay tuned…

2009 was a very busy year for me so I thought I would recap what I did and wrote during 2009. For starters I wrote a book that was published by Pearson Publishing entitled VMware VI3 Implementation & Administration. Having enjoyed that experience I also started writing a 2nd book on vSphere that will be published in 2010. It seems like I’ve been writing non-stop, besides the books I also write blog posts and tips for at least 6 different Tech Target websites. I am a member of the Server Virtualization Advisory Board for SearchServerVirtualization.com and also write a regular column for SearchVMware.com entitled Virtualization Viewpoints.

I was commissioned by Veeam to write a white paper for them to coincide with their new Veeam 4.0 product launch. I was named as one of the 300 vExperts by VMware in Fed. 2009. I attended VMworld this year as a member of the press and was again a judge for the Best of VMworld awards in the Security category. Besides all that I still find time to maintain this website, sorting through links and collecting and posting them is very time-consuming. Below is a list of some of work of mine that was published in 2009:

• Everything you need to know about vSphere and data storage (SearchStorage.com magazine)
• VMware 2009: Never a dull moment (SearchVMware.com)
• The Mechanics of VMware GO (SearchVMware.com)
• A review of VMware disk-to-disk backup apps: Veeam, Vizioncore, PHD Virtual and VDR (SearchDataBackup.com)
• Virtual-to-physical conversions: Why and how (SearchVMware.com)
• VMware vShield Zones: What it is and how it works (vShield Zones Pt. 1) (SearchVMware.com)
• Installing and configuring vShield Zones (vShield Zones Pt. 2) (SearchVMware.com)
• Quick tips for managing vShield Zones (vShield Zones Pt. 3) (SearchVMware.com)
• Determining the right size for your VMware VMFS data stores (SearchVMware.com)
• Virtualizing may cost more, but the benefits are worth it (SearchVMware.com)
• Choosing a virtual disk type for VMware ESX: Raw, thick and thin disks (SearchVMware.com)
• VMware vSphere: Got 64-bit hardware? (SearchVMware.com)
• Blades vs. rack servers: Basic purchasing principles still apply (Choosing hardware for virtual servers series Pt. 1) (SearchServerVirtualization.com)
• Selecting CPU, processors and memory for virtualized environments (Choosing hardware for virtual servers series Pt. 2) (SearchServerVirtualization.com)
• PowerShell scripting with VMware ESX tutorial: Installing and using the PowerShell tool (PowerShell series Pt. 1) (SearchVMware.com)
• PowerShell scripting for VMware ESX: Using Quest’s PowerGUI PowerPack script editor (PowerShell series Pt. 2) (SearchVMware.com)
• What is Changed Block Tracking in vSphere? (Virtualization Pro blog)
• Does virtualization really reduce administrator headcount and workloads? (Virtualization Pro blog)
• Several ways to explain virtualization to a non-techie person (Virtualization Pro blog)
• What are IP Pools and vApps? (Virtualization Pro blog)
• What is virtualization? (Virtualization Pro blog)
• Removing old hardware after a P2V conversion (Virtualization Pro blog)
• Are you taking advantage of VMware knowledge on the Web? (Virtualization Pro blog)
• Virtualizing Lotus Domino servers: Support concerns (Virtualization Pro blog)
• The process behind judging the Best of VMworld awards (Virtualization Pro blog)
• Using Storage VMotion to keep critical servers up when shared storage is down (Virtualization Pro blog)
• Scheduling virtual CPUs (Virtualization Pro blog)
• More details on VMware’s Fault Tolerance feature (Virtualization Pro blog)
• The key to successfully virtualized Domino servers (Virtualization Pro blog)
• VMware’s hardware compatibility list (Virtualization Pro blog)
• VMware Fault Tolerance: What it is and how it works (Virtualization Pro blog)
• Pointers for using thin-provisioned disks (Virtualization Pro blog)
• Troubleshooting mysterious VM resets (Virtualization Pro blog)
• What it means to be a vExpert (Virtualization Pro blog)
• Virtualizing Lotus Domino servers (Virtualization Pro blog)
• Lord of the virtual rings: Why hardware matters to your VMs (Virtualization Pro blog)
• A brief history of VMware (Virtualization Pro blog)
• Defragmenting virtual machine disk files (Virtualization Pro blog)

Here they are, the current top 20 VMware & virtualization related bloggers:

Can anyone knock Duncan Epping from the top spot? Chad Sakac made a run up to the #3 position in the last vote, can he go any higher? Interesting enough, Scott Lowe in the #2 position now works for Chad Sakac, can Scott stay ahead of his new boss? There are also many new bloggers out there producing some great content who could make the list like Hany Michael, Scott Sauer/Aaron Sweemer, Maish Saidel-Keesing, Scott Drummonds, Frank Denneman and Steve Chambers. Your votes will decide the winners and help decide the Top 25 bloggers. I’ve decided to expand the Top 20 to the Top 25 because there are so many great blogs out there.

So here’s how it works, you pick your top 10 favorite blogs, the survey will be open for two weeks. Once all the votes have been cast I’ll add them up and using weights (#1 vote = 10 points, #2 vote equals 9 points, etc.) and calculate the Top 25 blogs. Because of the cost of the online survey tool I’ve acquired a sponsor to help cover the costs. TrainSignal is the sponsor of this survey and as an added bonus they are giving away two copies of their awesome vSphere DVD training course to two voters who will be randomly selected. to vote you simply number your 10 favorite blogs from 1 to 10 with 1 being your most favorite blog. Your numbers must add up to 55 (1 through 10 added together) and we have controls in place so you can only vote once. The top 10 winners will receive a graphic image with their place number that they can display on their website.

When casting your vote we hope you are fair and use the following guidelines:

• frequency (how often they post)
• quality (how well the posts are)
• length (how long the posts are)
• longevity (how long they have been blogging for)

Please note for added protection this time against duplicate votes I’m now using IP address protection which limits one vote per IP address. While this method is more effective then using cookies it may prevent multiple users in corporate environments that use NAT from casting votes. As a workaround to that situation you can have people vote from a mobile device or home computer instead.

So bloggers, start your engines and let the voting begin, you can go to http://vote.vsphere-land.com or click the below link to cast your vote…

Click here to vote

Time for you to pick the top VMware & virtualization related blog websites. The last vote was in June 2009 and there have been many great new bloggers since then that are worthy candidates for the top bloggers. Since there are so many top blogs I’ve decided to expand the top 20 blogs to the top 25 blogs and you get to pick the winners. Voting will open on Monday, Jan. 4th and I’m using a better survey tool this time. Because of the cost I’ve gotten a sponsor for the voting, TrainSignal has graciously signed on to sponsor the voting and as an added bonus they they are providing two copies of their awesome vSphere DVD training course that will be awarded to two random voters. The voting will be to pick your top 10 favorite blogs, I will then count and weigh the votes to determine the complete top 25 list. The top 10 blogs will receive a nice graphic that they can display on their website. Stay tuned for details on the voting!

My secret project for the last month was putting together a gift for someone who would give you the shirt off his back if you asked for it. That someone is John Troyer from VMware who helps out bloggers, vExperts and the VMware community on a daily basis without asking anything in return. John has helped me out personally many a time doing things like writing the foreword to my book, getting a question answered by VMware engineers or getting me access to a beta program. I had planned on getting John a little gift for a while now as a token of my appreciation for all the help he has given me and with the holidays approaching I thought it was time to do it.

After thinking about it for a while I thought I would extend this to other people in the VMware community who know and deal with John and have also been affected by his generosity. I had to keep it a secret so John wouldn’t find out so I put together an email list of about 50 bloggers & vExperts who might want to contribute. The plan was to collect a small donation from anyone who wished to contribute which I would use to get John a gift with. I also had planned on getting him an appreciation plaque as well but after doing some research found an online group card that everyone could sign and add their own personal message to.

The response was overwhelmingly positive as everyone thought John definitely deserved this. As a result we were able to send John several gift cards and printed copies of the online group card that everyone had signed. In addition Chad Sakac from EMC kicked in with an Iomega storage unit for him as well. So thanks John for everything you do for us, I’m sure you already know we appreciate it but sometimes its nice to get a reminder of it anyway.

Update: Here’s the video link to John opening his surprise gift.

Update: Here’s a second video link recorded a few hours later.

I thought I’d give an update on some of the new things happening on this website.

1. The vLaunchpad has been updated and over-hauled, the blog roll has been alphabetized for easier viewing. I’ve added a new Storage blogs category since virtualization and storage seem to be so closely tied to each other. I added lots of new reference links, removed some old blogs that haven’t been updated in a long time and added lots of new ones that were missing.  I also have added a Twitter section with my group lists and created a Tweet grid of the Top 20 bloggers and Top 100 VMware people that can be accessed at http://tweets.vsphere-land.com.
2. I’ve created two Tweet lists using the new Twitter lists features so you can also quickly go to the Top 20 blogger or Top 100 VMware people tweets.
3. Lots of vSphere link updates, including many new links on Thin Provisioning. Be sure and check out the new white paper VMware just released on the performance impact of using thin disks.

It’s been 6 months since I did an updated  Top 20 blog list, last time I let everyone vote on it instead of me picking it. Time to do another one, some of the blogs currently in the top 20 haven’t been updated in a while and there are lots of excellent new ones out there. I’m going to expand it to the Top 25 and setup a survey using a new dedicated survey website. Voting will open Dec. 1st and run through Dec. 15th.

I’d like to highlight some of the excellent new blogs that I added to the vLaunchpad:

• Andi Mann - Ubergeek
• Lori MacVittie
• Pivot Point (VMware’s performance guru - Scott Drummonds)
• Virtual Cloud (Andre Leibovici)
• Virtualised Reality (Barry Coombs)
• Virtual Kenneth’s Blog (Kenneth van Ditmarsch)
• vSphere Blog (VMware)

Veeam approached me recently and asked if I was interested in writing a white paper for them to coincide with their launch of latest version of Veeam Backup & Recovery (4.0). The timeframe for writing it was short but I accepted it because I was interested in learning more about the vStorage APIs and also Veeam Backup & Recovery. While doing research for it I did learn a lot about the new vStorage APIs that I had not known before. Typically unless you’re a vendor or developer you don’t deal much with APIs, but I’m the curious type though and like to dig deep and find out how things work. There are many important new features in the vStorage APIs and other storage-related APIs in vSphere that are real game-changers for vendors if they choose to take advantage of them. Even if you’re not a developer you should know a bit about them so you have a better understanding of how things work in vSphere.

I learned two additional things while writing the white paper for Veeam, the first is their new 4.0 version of Veeam Backup & Recovery is the first of the many disk to disk backup applications to take full advantage of the new APIs in vSphere, the second is that there are some pretty smart folks at Veeam that are very passionate about their products. I’d like to thank Doug Hazelman & Anton Gostev for answering my many questions about how things work behind the scenes with their product. So go checkout the white paper and more importantly Veeam’s new 4.0 version which I’ll guarantee you’ll be impressed by. Also look for an upcoming tip on searchdatabackup.com that I did that compares how disk to disk backup vendors are using the vStorage APIs and where they are at with their product releases.