Change Evolution is 'The Way'

I'm working on a paper, document, anything, (probably just this post now since my schedule is so busy) on something that's been in the back of my mind for a while now, and every time I speak to a new opportunity or a customer I always wished that I had something substantial to leave behind to show that yes, it is possible to achieve the desired future state without pain.

What I'm talking about is how to get from A to Z without pain, fear, risk, or increased cost and time.

'A to Z' is an expression that we all use, but in Lehmann's terms it is getting to the desired future state from the current state.

What is the future state? For example a server migration project of 1000 Wintel servers into VMware infrastructure in 6 months.


So if A is the origin and Z is the destination, then the journey of getting from A to Z is the experience. It is the experience that is all too important. In a project's lifecycle, the primary purpose of a project is to bring benefit to something (an organisation for example). But the experience can vary dramatically. Z can be achieved but at what cost? Z can be achieved but it could take a long time. Z can also be achieved but after how many mistakes, issues and actions that were required to achieve Z?

Is there a way to define the experience? To reduce the amount of risk and unplanned change, to limit the exposure to mistakes and unknowns. To cap the amount of time and cost to achieving Z. 'The Way' then is called a methodology. A methodology is a collection of processes and frameworks which are used to control the execution of change within a project.

So while I’m in a pessimistic mood, let’s go over why there are difficulties from having a comfortable journey:
• Lack of planning
• Lack of clear objectives
• Lack of support and acceptance (See Steve Chamber's Barriers to Virtualisation)
• Lack of risk management
• Lack of a business case or project justification
• Lack of change control


Why is change so feared?

Let's assume that your project justification and initiation are all good and that your project plans, objectives, business case and RAID are all up to scratch and now you are ready to embark on a project that changes your IT infrastructure. Have you considered how you will manage change? Are there push backs from the business or application owners who don't really need or want anything to happen to their precious server due to changing the way a workload is run?

How can you alleviate their fears and introduce controlled change?

So let's take the classic CIO/IT Director from a few years ago at a time when x86 consolidation using virtualisation was still in its infancy (there are those that still think transitioning to a virtual infrastructure is a risk too far). These CIOs had fears around change - change of management, change of skills, change of processes and changes with operations. These fears were prevalent then and are still prevalent now. In my view the main enablers for change are the frameworks that can be used to get from A to Z.

Without change, an IT organisation will never be able to evolve into an IT organisation that has more reliable infrastructure, more efficient processes and more streamlined operations. Those companies that do embrace change and evolve are considered to be the most high performing IT Organisations.

The consensus is basically this: change causes fear, therefore projects such as P2V take forever to do, and without the correct methodology your P2V project could fail before it has actually begun. But by introducing controlled change and then putting the processes and governance in place; the strategy controls, manages change and provides a framework for effective management and delivery of the project.

The barrier to evolution is due to a fear of change, we alleviate this fear by controlling change. Change then becomes the enabler for evolution: please welcome Change Evolution.

So what is Change Evolution?

Change Evolution is a framework that uses ITIL/Visible Ops methodologies to control migration to virtualisation projects. It expedites ROI due to enablement of change management as part of BAU/Operations.

Change Evolution is a framework for delivering projects with

• less Risk
• less Time
• less Cost

How is this accomplished?

1. With baselined standard operating environments (SOE) which are standardised and adhere to strict change control.
   
2. With Standard Operating Procedures (SOP) which are auditable, repeatable and measurable and are strictly controlled. Because these procedures are defined and controlled as part of the framework, it is possible for any member of the project to use these procedures to assist with the grunt work of the project. These procedures enable the 'turning the handle' method of migrations where the migrations are streamlined into the control processes.
   
3. By working closely with the change control board (CCB). It is strategic to keep the CCB on your side, we are not re-inventing the wheel with change boards, we embrace them, but the amount of requests is submitted in a ‘turning the handle’ method in which P2V migrations are requested weeks in advance and each one follows the same migration methodology, processes and SOPs. Therefore these migrations can actually be integrated into operations quicker and with no risk.
   

By using a defined methodology that integrates with the change control processes it is possible for you to 
deliver 
record‐breaking 
project 
successes 
without
 risk 
and within 
strict
 time scales
 and
 budgets and above all with no pain.

Configuring vSwitch0 for Service Console and VMkernel portgroups

Following on from a previous post on how to configure vSwitch0 with active/passive network interfaces using vmware-vim-cmd for the two port groups, I have included some diagrams explaining how it all works.

The reason that Link Aggregation will not give any performance benefit if both the Service Console and the VMkernel share two active uplinks on a virtual switch configured with Route based on IP hash load balancing policy can be summarised as follows.

802.3ad/LACP aggregates physical links, but the mechanisms used to determine whether a given flow of information follows one link or another are critical.

You’ll note several key things in this document that are useful in understanding 802.3ad/LACP.

• All frames associated with a given “conversation” are transmitted on the same link to prevent mis-ordering of frames. So what is a “conversation”? A “conversation” is the TCP connection.
• The link selection for a conversation is usually done by doing a hash on the MAC addresses (Route based on source MAC hash) or IP address (Route based on IP hash).
• Link Aggregation achieves high utilisation across multiple links when carrying multiple conversations, and is less efficient with a small number of conversations (and has no improved bandwith with just one).
  

Link Aggregation applies between two network devices only. Link aggregation can load balance efficiently – but is not particularly efficient or predictable when there are a low number of TCP connections, hence it is not useful when there are only two TCP connections, the Service Console IP address and the VMkernel IP address. Using link aggregation with VMware ESX and Cisco networking requires the use of Route based on IP hash and static LACP/802.3ad.

So how does the VMkernel distribute traffic over the Link Aggregation Group (LAG)?
The VMkernel distributes the load across the Link Aggregation Group by selecting an uplink to the physical network based on the source and destination IP addresses together. Each source / destination IP conversation gets treated as a unique route, and is distributed across the LAG accordingly. Using an IP-based load balancing method allows for a single-NIC virtual machine to possibly utilise more than 1 physical NIC. Returning traffic may come in on a different NIC, so Link Aggregation must be supported on the physical switch.

The diagram below shows what you would want to achieve with vSwitch0 when using a single virtual switch with two network interfaces with two port groups configured for the Service Console networking and the VMkernel networking.


The Service Console uses vmnic0 as the active uplink and vmnic2 as the passive uplink. The VMkernel uses vmnic2 as the active uplink and vmnic0 as the passive uplink. If one of the network interfaces fail or the corresponding physical switch fails or a cable fails, then the portgroup will utilise the standby network interface.




The configuration settings seen within the VI Client are shown in the figures below.

The vSwitch0 configuration
The vSwitch uses Route based on IP hash with both adapters set as active.






The Service Console portgroup configuration

The Service Console port group uses vmnic0 as the active adapter and vmnic2 as the standby adapter and the load balancing policy is set to Route based on virtual port ID. This overrides the configuration inherited by the vSwitch.



The VMkernel portgroup configuration
The VMkernel port group uses vmnic2 as the active adapter and vmnic0 as the standby adapter and the load balancing policy is set to Route based on virtual port ID. This overrides the configuration inherited by the vSwitch.



Network Utilisation
The graph below shows the current utilisation of the network interfaces assigned to vSwitch0, under normal operations only vmnic0 is used because the Service Console traffic uses vmnic0.












Let's see what happens when I initiate a VMotion to the same server.

The VMotion traffic uses the active network interface vmnic2 as expected.

It is obviously apparent that using override active/passive uplinks for the Service Console and VMkernel port groups has significant advantages. By doing this we restrict the VMotion traffic from flooding the Service Console uplink adapter that can be experienced when using an active/active configuration.

Until Round Robin is here.. how to Load Balance over Active/Active Paths using scripts

Recently a customer had a few issues with having all VMs using the same path to the LUNs, this was down to putting too many workloads onto servers that were used as a proof of concept. Inadvertently, SAN problems arose so I was asked to checkover the storage.

First a little background on the infrastructure.... a number of rack servers plus a number of blade servers, hooked into two fabrics with IBM SVC as the backend. Each ESX server has two FC HBA, and each fabric switch had two connections to the SVC, therefore each ESX server has four possible paths to the LUNs. The paths were all active as shown on this pic:

As you can see the path policy is currently set to mru, most recently used path policy is best used in an active/passive configuration.



mru:

• LUNs presented on single Storage Processor at any one time
• Failover on NOT_READY, ILLEGAL_REQUEST or NO_CONNECT
• No preferred path policy
• No failback to preferred path if it returns online after failover

Since, esxcfg-mpath -l shows that we are in fact using active/active, it is best to change the policy to fixed path policy:

• LUNs presented on multiple Storage Processors at same time
• Failover over on NO_CONNECT
• Preferred path policy
• Failback to preferred path if it returns online after failover

So how do we now go about changing the policies on all our servers? Well we could use VI-Client and change each datastore to use a different path - doing this for 10 datastores per server with 20+ servers? howabout no! The alternative then would be to script it.

The script from Yellow Bricks is of particular use, as for each LUN it finds it uses a different path for each LUN. The script just sets each LUN up to use a preferred path, but obviously for default installations of ESX, you cannot use preferred path when you are using mru policy. So we must change all LUNs to use fixed path policy first.

By re-using the script form Yellow Bricks, I've come up with this:

#!/bin/bash
# vmhbafixedpath.sh Script to rescan vmhbas on ESX 3.5 host
# Written by hugo@vmwire.com
# 21/05/2008 18:20
for PATHS in 2 4 6 8
do
STPATHS=${PATHS}
COUNTER="1"

for LUN in $(esxcfg-mpath -l | grep "has ${STPATHS} paths" | awk '{print $2}')

do
esxcfg-mpath --lun=${LUN} -p fixed
COUNT=`expr ${COUNTER} + 1`
COUNTER=${COUNT}
if [[ ${COUNTER} -gt ${STPATHS} ]]
then
COUNTER="1"
fi
done
done


Then use the script from Yellow Bricks, to set up the preferred paths. Now the changes do not take into effect until the HBAs are rescanned, and the Storage is refreshed. The following script rescans the HBAs

#!/bin/bash
# rescanhbas.sh Script to rescan vmhbas on ESX 3.5 host
# Written by hugo@vmwire.com & nkouts
# 21/05/2008 18:50
# Assumes there is no vmhba0 and max vmhba9

for HBAS in 2 4
do

STHBAS=${HBAS}
COUNTER="1"

for HBA in $(esxcfg-info -w | grep vmhba | awk '{print $3}' | grep -e 'vmhba\+[1-9]' -o)
do
esxcfg-rescan ${HBA}
COUNT=`expr ${COUNTER} + 1`
COUNTER=${COUNT}
if [[ ${COUNTER} -gt ${STHBAS} ]]
then
COUNTER="1"
fi
done
done

And there is no known console based method to refresh the storage subsystem (anyone?) apart from using VI-Client, rebooting the ESX host or restarting the vmware management service:






service mgmt-vmware restart







UPDATE: Use /usr/bin/vmware-vim-cmd to refresh the storage
/usr/bin/vmware-vim-cmd hostsvc/storage/refresh

So now we have the servers using different paths for each datastore.



















It only took a couple of seconds to change the policies on each server using these scripts, obviously using these as part of a build script would be ideal for deployments where you know the SAN configuration.

Planning a VMware ESX deployment on IBM BladeCenter H - Part 2

In the previous post I covered the network design for a HS21 with 4 network interfaces. This post will continue with a diagrammatic representation of the interface table.

As described previously, this configuration provides full network fault tolerance on all levels: adapter, port, CAT5, switch bay and core switch.


Put your finger over any individual constituent part, i.e., pNic, interface, bay switch or core switch, to simulate a failure and there will always be an alternative path.

I'm waiting for the customer to decide on whether to include the CFFv daughtercard in this phase of the project, and will update this post with the new design if required.

Next up, environmentals...

Those of you familiar with the HP c-Class blades will probably know that there is a superb tool called the HP BladeSystem PowerSizer 2.9, I've been trying to find an equivalent from IBM, but as yet have not found anything that comes as close. (Any pointers will be appreciated)

Instead I've had to resort to using data obtained from The Edison Group study titled Blade Server Power Study - IBM BladeCenter and HP BladeSystem, Nov 7 2007, document titled "BLL03002USEN.pdf".

The results show, in summary a BladeCenter H chassis with 14 blades on full load will need 14,352.51 BTU/Hr with a peak power consumption of 4,208.80 Watts. Most modern datacenters with good power feeds will be able to accommodate that kind of load. Cooling requirements will be left to the customer to calculate.

Additionally, this single chassis will require 9 rack units and 4 power feeds due to the additional 2900W power supply modules.

Part 2.... Continued..

Thank you Aaron for your help with the power sizer.

Here is the output from the tool (not as nice as HP's offerring by the way)














In the next part... network design for the x3650.

Planning a VMware ESX deployment on IBM BladeCenter H - Part 1

Well here I am, starting a new project for a new customer at a new datacenter again. This time, its a large retail organisation looking to do the usual, consolidate, virtualise, go green etc etc. They have selected IBM System X and BladeCenter H as the platforms of choice for the new VMware ESX 3 environment. So here we go with the planning....

The BladeCenter H has eight switch bays and two Advanced Management Module (AMM) bays. The two AMM act in much the same way as the Onboard Administrator on HP C Class. There are two for redundancy. Two of the eight switch bays are used for FC Switches, for this project we are using Brocade 4Gb SAN switches.

The other bays are occupied by Cisco GbE Switch Modules.

HS21s are used for the initial phase of the project. These blades can accommodate upto 6 NICs and 2 HBAs, with 2 onboard and the other 4 provided by daughtercards. The customer has elected to use 4 NICs as opposed to the 6 that I normally recommend for ESX implementations. The two extra NICs are provided by the CFFh daughtercard, this daughtercard houses 2 network adapters AND 2 Fibre Channel HBAs.

The table below (from IBM) show the interface to bay mapping.

Since only 4 interfaces are available, teaming and VLANs will have to be used to provide resilience and to separate the SC and VMKernel networks.

I will be teaming Interface 0 (eth0) with Interface 3 (eth3) as opposed to the IBM table (dedicating an adapter to a service), as this will team one onboard port with one daughtercard port. Likewise eth1 will then be teamed with eth2.

* The location of the two Fibre Channel Adapters should be Daughter Card CFF-h, not v as shown in the IBM table.

The following diagram shows the correct mapping.














The table below details the network interconnects.

Interface is the network adapter inside a blade, Location is where the interface is, Chassis Bay is where the interface terminates at the rear of the BladeCenter chassis, pSwitch is the external core switch that the Chassis Bay uplinks to, vSwitch is the ESX virtual switch that the Interface provides an uplink for, vLAN is the ID that is assigned to each Port Group and Service is the type of port group assigned to a vSwitch.