Andy’s Blog

Yeah, ok, probably a strange combination, but I run a Cisco router at home – for security, flexibility and ok, for the general geekiness of it.

Over Christmas I did a fair bit of rewiring for my home network (how difficult is it to find white CAT5e cable these days? Blimey! Thanks Universal Networks ). But a problem appeared – my XBOX would not automatically get it’s IP address when it powered up. First I thought it was the cable, but after remaking it several times and testing it I ruled that out. Then I threw a bog standard switch in the link and it all worked fine.

WTF.

A discussion with a friend of mine this morning made the penny drop. The Cisco has spanning tree enabled by default. DOH.

A quick

no spanning-tree vlan 1

And all was well – my XBOX now behaves properly Now I can listen to music while I’m having to work from home today.

Edit:
Ok, after a message from the friend in question, I’ve decided that I have to finally make an edit to this post to actually name (and shame) my on-tap Cisco Geek.  And I introduce you to Mr Hill, also online at Ninja Badger. And yes he is a *REAL* Cisco Geek.

I’ve been going through the infrequent hassle of moving my hosting provider.

I previously had a dedicated server (a very nice, very powerful box but I doubt anyone is interested in that ) with a company called UK Solutions down in Redditch, but recently decided that as I no longer needed the flexibility of my own virtualised system (and of course, want less hassle to devote to my other budding projects!) that I would migrate everything and terminate the co-lo agreement.

And so the fun began.

I started off a Reseller account with 34sp.com (basically an account that lets me throw all the sites etc up and let someone else deal with the hardware and management – much nicer for me these days!). As they are using Plesk things were a little fiddly to start with (and I still don’t get the icons loading, but hey, it works), but once I got the hang of things it all went smooth as anything. I even get daily reporting straight to my inbox And customer control panels. Dear god, that’s scary … I can get them to do their own email setup and leave me in peace. Next thing you know, I’ll be charging for hosting. Oh look, flying pig.

I fired off my contract cancellation into UK Solutions with a good couple of months left on my current term, and was rather dismayed to see that they added another 5 weeks onto the term to bring it up to a full quarter period. Seems that there is a 3 month exit clause on the contract. Argh. You would have thought that termination of contract, when you have a decent period left anyway, would let them have a little bit of flexibility on this clause especially when I’d been a customer of theirs for three years without any hassles.

Two points for signing off this minor rant

1. Check your contracts … always read the small print

2. You don’t ALWAYS get what you pay for. 34sp.com are a LOT cheaper than UK Solutions, and so far I have absolutely no complaints – even when logging fault reports, things get resolved swiftly. Keep up the good work guys.

Over the past few weeks, us geeks at Money Dashboard have been hard at work building our production environments, and as I am sure you can imagine we hit a few issues.

In order to help anyone else out who might be thinking about, or even is deploying a fairly complex environment around Hyper-V I thought I would share our findings.

First off, as I’m sure you can all appreciate, I am unable to go into any real specific details on our implementation, so some of this information maybe be a little strange, or difficult to follow – bear with me and hopefully if you ever find yourself in a similar situation it might just help you out!

iSCSI and Virtual Machines

If, like us, you are virtualising your SQL Server’s, then do not forget that you will need to bring some iSCSI (or whatever storage system) mounts through to the VM’s.  This on the surface does not pose a problem, but we DID hit problems when pulling iSCSI through. As you can imagine, we are running Jumbo Frames (MTU 9000) on our iSCSI network in order to optimise throughput, but the default VM adapters only support standard packet sizes (i.e. MTU 1500). In order to get around this, you need to use the Synthetic Network Adapter in the Hyper-V VM, and be sure to set the properties to enable Jumbo Frames. You must also have the physical nic on the server set for Jumbo Frames too. Always worth checking with the following command:

netsh interface ipv4 show subinterface

You may, like us, then notice some packet loss on the iSCSI Adapter. In our case this turned out to be something strange going on with the way the Synthetic adapters were behaving with our Broadcom nics (BCM5709 in case anyone is interested!). Disabling all the offload components (TCP, iSCSI and Checksum) fixed the problem, but we still do not know exactly why this was occurring …

QoS …. do NOT forget it

Make sure you split your Live Migration and Heartbeat traffic onto separate networks, and oh most certainly remember to apply those Quality of Service rules on the switchgear.

We forgot, and as soon as Live Migration kicked off, the complete Hyper-V cluster went mental … it thought that all the other nodes had failed, so EVERY node went to start EVERY virtual machine. As you can no doubt imagine, absolute chaos ensued, and the virtual machine disks were corrupted (one catch with using the new Cluster Shared Volumes it seems).

Dell EqualLogic, and BACS

We are lucky enough to be using pretty much all Dell kit, including the Dell EqualLogic PS6000XV as our SAN. One snag we did hit is that you really do not want to route your iSCSI traffic over a virtual adapter created through the Broadcom BACS suite … on the surface it will appear to work, but when you start looking at it carefully you will notice that there appears to be a significant bandwidth limitation creeping in somewhere. Not sure if it was the BACS drivers, or the Dell MPIO driver, but it disappeared when we reverted to use proper physical NICS. Equally, do not forget to install the Dell MPIO drivers into any virtual machines that are using the iSCSI

Adding Hyper-V Clusters in Microsoft System Center Virtual Machine Manager

When you are finally ready to add your machines into SCVMM, add the cluster name – not the individual machine names. It seems that if you add the individual machines, SCVMM does NOT treat them as an HA cluster. I haven’t found any logical way of merging multiple machines into a cluster in SCVMM, or any real reason why it doesn’t prevent you from adding the individual nodes anyway (it can see they are in a HA cluster configuration after all).

Summary of the kit we used:

Dell EqualLogic PS6000XV SAN
Dell PowerConnect 6248 gigabit switch stack
Dell R200 1U Rack Mount Servers
Dell M1000e Blade Centre

An awful lot of cabling.

Microsoft Windows Server 2008 R2 Enterprise (Both full and Core)
Microsoft Windows Server 2008 R2 Standard (Both full and Core)
Microsoft System Center Virtual Manager Manager 2008 R2
Microsoft System Center Operations Manager 2007 R2

And a ton of custom scripts.

Thanks go to Dave Veitch from Company Net for assisting in the configuration!

Hit another Gotcha with CRM 4.0 today when enabling the Outlook integration – when you install it under 64-bit, the odds on it will insist that you do not have Outlook installed, or it is not your default mail client.

To get round this, you need to tweak the registry a little:

Change the default value in both these keys to “Microsoft Outlook”

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Clients\Mail
HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Mail

And restart setup.

Problem solved

I wish installers patched round things like this, especially when they interface with standard mechanisms.

We have decided to have a look at Microsoft Dynamics CRM at work, and carrying out the installation seems fairly painless – until right at the end of the wizard I got an error.

“Action Microsoft.Crm.Setup.Server.MsiInstallServerAction failed. This update package could not be opened. Verify that the update package exists and that you can access it, or contact the application vendor to verify that this is a valid Windows Install update package”

It seems that the installer uses the Internet Explorer cache, which defaults to 50Mb. And guess what – the update is more than this, so it happily deletes it after downloading.

Fail! Simple increase the cache size and start again.

I’ve been having a bizare problem with Windows Installer on a fresh install of Windows 7 – I’ve been unable to get MSI installs to work with the constant error that another install is already running.

After a quick bit of diagnostic hunting in the event log I came across a recurring install “Microsoft Visual Studio Tools for Applications 2.0 – ENU”. Hmm. It was failing as it was unable to find the directory “”. So I created it. And amazingly the install completed successfully and I can finally install app again.

Very odd.

Quite often you find yourself looking for that tool you know exists, but can’t remember the name of. This morning I was in such a position.

Setting up a Windows Server 2003 x64 Virtual Machine under HyperV, and upon booting it you have no network. And only on Service Pack 1, so you can not install the Virtual Machine Integration tools (and give you the drivers). Catch 22

So, off I go, and download this handy utility: http://isorecorder.alexfeinman.com/Vista.htm. Install, create ISO, mount it under HyperV and bingo we can install SP2 on the VM without difficulty .

On the subject of creating ISOs, another handy tool for you kit is: http://www.minidvdsoft.com/isocreator/

Just thought I would throw this up quickly – I just hit a problem when installing the HyperV Integration tools on a new Windows Server 2003 Virtual Machine.

HyperV threw:

Error (13233)
Unable to install Virtual Guest Services on virtual machine Win NAS. The Hyper-V Integration Services setup exited with error code 60001.

Recommended Action
For more information, refer to the Hyper-V documentation for virtual machine integration services at http://go.microsoft.com/fwlink/?LinkId=118036 and then try the operation again.

The solution is simple – install Service Pack 2 on the Windows Server 2003 VM and try again. Would be nice if the HyperV toolkit checked the SP level and gave you a decent error however (and the above link is pretty ambiguous too).

Some time ago I blogged about this utility / management app that I had seen demonstration by some Microsoft guys.

Now I’ve actually had a chance to get round to installing it myself – and good god, it was not painless.

The architecture I’m guessing is fairly typical with most small shops – single (big) server running as File Share, Active Directory Domain Controller, Exchange Server, and generally our main point of administration.

We have another server in the office that is a Server Core install of Server 2008 designated to run our Hyper-V (well, some of them) sessions only. The other VM’s will be split across some other servers, but that’s for later

Which is why I thought we would go for SCVMM.

First off, installation of the main component ( server ) was started off on the main office server. No problems, we have SQL already installed, next next next.

Bang.

Setup Crashes.

A quick Google later gives me this link : http://cosier.wordpress.com/2008/03/16/system-centre-virtual-machine-manager-2007-scvmm-crashes-during-installation/
A few quick comments in the hosts file and off I go again.

Gets further.

Bang.

“Virtual Machine Manager Server installation has failed”

ARGH.

Lets see. The details are:

“An error has occurred while trying to configure Virtual Machine Manager. Uninstall Virtual Machine Manager from Add or Remove Programs and then run Setup again.
ID: 257. Details: No mapping between account names and security IDs was done”

At this point, I loose my temper, and restart the setup. And let it install the desired extra copy of SQL Express 2005. As you can imagine, this completes. How annoying. It seems that the setup program was trying to set database path permissions on the database it was created hosted on SQL Server 2008 – something it doesn’t need to do.