Having experienced several complete data center power outages I’ve learned some important lessons over the years when it comes to virtualizing your infrastructure. Today I experienced another power outage and my previous experiences ensured that I was ready for this one and I thought I would share some tips:
DNS is the most critical component in your environment, for almost anything to work properly you need a DNS server up first. If you have all your DNS servers virtualized and they are on shared storage its going to be very difficult to bring them up because everything else that needs to come up first usually relies on DNS. Therefore you should make sure you have at least one DNS server on the local storage of one of your hosts, that way you can get it up early on and not have to wait for your shared storage to come up. If you want to go a step further keep one as a physical server also, so you can quickly and easily get DNS up right away.
Active Directory, DHCP & any other authentication servers are also very important, having a workstation up and running is very handy so you can centrally connect to hosts and power things back on. If you’re using DHCP for your workstations and servers you again will want one up as soon as possible so you can get them on the network and get going. Not having an Active Directory server available can make Windows servers take a very long time to boot. So again, keep an AD & DHCP server on local storage or on a physical server so you can quickly get them up as soon as possible. AD, DNS & DHCP are critical to a Windows environment and without them available you’ll find that the rest of your environment is mostly useless.
Know your ESX command line, if your vCenter Server and other workstations are not available you’ll need to start VM’s using the command line. Even if your DNS server is on a local VM you won’t be able to start it without the vSphere Client. Therefore you’ll have to log into the ESX console and manually start it, if you don’t know the command to do this that could be a problem. Keep a cheat sheet by your hosts with the basic commands that you’ll need like vmware-cmd to get things up and running using the console.
Know your host IP addresses, if DNS is not up yet you won’t be able to connect to your hosts using putty or the vSphere client using their host names, you probably won’t know their IP address and without a DNS server can’t look them up. Therefore keep a list of your host IP addresses so you can use that to connect to them.
Know how to re-scan storage, your hosts may come up before your shared storage, once your shared storage is up you’ll need to rescan from your hosts so they will see it and you can restart the VM’s on it. You can do this using the vSphere client by clicking on Configuration, Storage Adapters, selecting your HBA and clicking the Rescan button and then select search for new devices. You can also do this using the command line esxcfg-rescan utility.
Make sure you know where your datacenter keys are if you use a electronic card scanner to open your doors. Most systems are placed in the datacenter and if the power goes out your doors are not going to work. There is nothing worse than running around trying to find keys in a crisis to get into the datacenter. And make sure you don’t keep the keys in datacenter or you’ll have to break down the door to get in. (thanks Tony DiMaggio for reminding me about this one)
Being prepared is critical in crisis situation to ensure you can react quickly to get things back up and running. Sometimes it takes a crisis to point out any shortcomings that you may have in your environment but thinking ahead can save you from big headaches later on.
I’ve been maintaining the vLaunchpad using Dreamweaver which has been difficult at times so I thought I would explore other options. I’ve resisted using Wordpress as I had for my main site because the page layout is kind of unique and not easily re-produced in Wordpress. Well after much theme hacking I think I have it ported over to Wordpress pretty well. I’ve done extensive CSS modifications to make sure I would fit as much information as possible in one area and not have to resort to lots of scrolling or multiple pages. It’s not done yet but it’s getting close, I still need to add all the blog links and do some other work to it. I’ve also moved it to the clouds, which is GoDaddy’s new hosted grid computing offering. So let me know what you think, I’m open to all your suggestions, tell me what you like, don’t like and want to see added to it. One thing you’ll notice is I added all of Hany Michael’s great vDiagrams to the sidebar for easy access to them.
So go check out the new version of the vLaunchpad and let me know what you think, just remember it’s not done yet and I have lots of changes to make to it. Thanks to the Netherlands gang (Duncan, Eric & Gabe) for their early feedback that helped me get the design right.
Due to cuts and other circumstances I will no longer be blogging at SearchVMware.com’s Virtualization Pro blog. While exploring other blogging opportunities I decided to come right back where I started, this website, vSphere-land.com. So besides my gigantic link collection you’ll be seeing lots of new content here including tips, news, reviews, opinions, top 10 lists and much more. For starters I’ll be doing a series documenting a new project from start to finish that I’m the lead on which is a Lotus Domino virtualization project. So stay tuned for lots of great content coming from this site.
As I previously mentioned two randomly chosen voters in the top blog survey would win a copy of TrainSignal’s vSphere DVD training course. I exported the voting results to Excel and then used a random number generator to pick two row numbers and the winners are:
mcgivern_jon at emc dot com from Westborough, MA
brian_alexander at mentor dot com from Longmont, CO
David Davis from TrainSignal will be contacting you, thank you for your participation in the voting!
Here’s a table with the vote totals for all the VMware/virtualization blogs included in the survey, there were a few others that only had 1 or two votes that I didn’t include. Total votes includes any vote a blog received whether it was 1 through 10. Total points is the sum of each vote times it’s appropriate weight for the position (#1 vote = 10 points, #2 vote = 9 points, all the way down to a #10 vote = 1 point).
Well I’m not saying, watch the presentation to find out, I’ll be doing another post later on with the full results as well as some detail on the process I used to sort through the votes and score them to determine the winners. Congratulations to all the winners, there were many newcomers to the top 25 blog roll. I’ll be sending you graphics that you can display on your website to reflect your achievements. The winners of the TrainSignal vSphere DVD training course will be announced in my next post. Click here to see the results presentation in a new window
Voting for the top VMware/virtualization blogs will close this Friday night , 1/15 at 10:00pm MST so if you haven’t voted yet be sure and get your vote in. Over 850 votes have been cast already and the Top 25 blogs will definitely look different than it is right now. The results will be announced on Monday after I spend the weekend going through the data, I’ll be publishing a Powerpoint presentation converted to Flash with the voting details and the winners. So stay tuned…
2009 was a very busy year for me so I thought I would recap what I did and wrote during 2009. For starters I wrote a book that was published by Pearson Publishing entitled VMware VI3 Implementation & Administration. Having enjoyed that experience I also started writing a 2nd book on vSphere that will be published in 2010. It seems like I’ve been writing non-stop, besides the books I also write blog posts and tips for at least 6 different Tech Target websites. I am a member of the Server Virtualization Advisory Board for SearchServerVirtualization.com and also write a regular column for SearchVMware.com entitled Virtualization Viewpoints.
I was commissioned by Veeam to write a white paper for them to coincide with their new Veeam 4.0 product launch. I was named as one of the 300 vExperts by VMware in Fed. 2009. I attended VMworld this year as a member of the press and was again a judge for the Best of VMworld awards in the Security category. Besides all that I still find time to maintain this website, sorting through links and collecting and posting them is very time-consuming. Below is a list of some of work of mine that was published in 2009:
So here’s how it works, you pick your top 10 favorite blogs, the survey will be open for two weeks. Once all the votes have been cast I’ll add them up and using weights (#1 vote = 10 points, #2 vote equals 9 points, etc.) and calculate the Top 25 blogs. Because of the cost of the online survey tool I’ve acquired a sponsor to help cover the costs. TrainSignal is the sponsor of this survey and as an added bonus they are giving away two copies of their awesome vSphere DVD training course to two voters who will be randomly selected. to vote you simply number your 10 favorite blogs from 1 to 10 with 1 being your most favorite blog. Your numbers must add up to 55 (1 through 10 added together) and we have controls in place so you can only vote once. The top 10 winners will receive a graphic image with their place number that they can display on their website.
When casting your vote we hope you are fair and use the following guidelines:
frequency (how often they post)
quality (how well the posts are)
length (how long the posts are)
longevity (how long they have been blogging for)
Please note for added protection this time against duplicate votes I’m now using IP address protection which limits one vote per IP address. While this method is more effective then using cookies it may prevent multiple users in corporate environments that use NAT from casting votes. As a workaround to that situation you can have people vote from a mobile device or home computer instead.
So bloggers, start your engines and let the voting begin, you can go to http://vote.vsphere-land.com or click the below link to cast your vote…