Sunday, December 15, 2013

Stay clear of the Intel - Storage Controller Update optional update

Shannon a colleague of mine  recently came across an issue when building a Hyper-V host and pulling down the latest updates. He installed the Intel - Storage Controller - Intel (R) Ethernet Setup Driver for FCoE optional update.

This subsequently removed all NIC Configuration installed on the Hyper-V host and converted the NIC's to FCoE Storage Controllers.

Stay clear of this optional update if you aren't needing to configure FCoE configuration on your Hyper-V host.



Sunday, November 10, 2013

Hotfix: SCCM 2012 R2 and ADK 8.1 - PXE Boot (WDS) issues

A hotfix has been released recently to resolve WDS (PXE enabled Distribution Point) issues relating to SCCM 2012 R2 and ADK 8.1:
http://support.microsoft.com/kb/2905002

After ADK 8.1 is installed on the SCCM 2012 R2 Site Server (with a PXE enabled DP) there appears to be no issues.  However after a subsequent reboot the WDS service fails to start (Automatic or Manual) and prevents clients from PXE booting.

Furthermore the following application event logs are recorded:
Faulting application name: svchost.exe_WDSServer, version: 6.3.9600.16384, time stamp: 0x5215dfe3
Faulting module name: MSVCR100.dll, version: 10.0.40219.1, time stamp: 0x4d5f034a
Exception code: 0xc0000005
Fault offset: 0x000000000005f61a
Faulting process id: 0xae4
Faulting application start time: 0x01cec5d767184634
Faulting application path: C:\Windows\system32\svchost.exe
Faulting module path: C:\Program Files\Microsoft Configuration Manager\bin\x64\MSVCR100.dll

Tuesday, November 5, 2013

SCCM 2012 PORTALWEB is not responding to HTTP requests

After installing the Application Catalog in Config Manager 2012 on Windows Server 2008 R2 it is common to see the following error in SMS_PORTALWEB_CONTROL_MANAGER:

Portal Web Site Control Manager detected PORTALWEB is not responding to HTTP requests.  The http status code and text is 500, Internal Server Error.
Possible cause: Internet Information Services (IIS) isn't configured to listen on the ports over which PORTALWEB is configured to communicate.
Solution: Verify that the designated Web Site is configured to use the same ports which PORTALWEB is configured to use.

Possible cause: The designated Web Site is disabled in IIS.
Solution: Verify that the designated Web Site is enabled, and functioning properly.

For more information, refer to Microsoft Knowledge Base.



This issue occurs because the Applicationhost.config file for Windows Process Activation Service (WAS) has the following section defined, and this section is incompatible with the .NET Framework 4.0.

<add name="ServiceModel" type="System.ServiceModel.Activation.HttpModule, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="managedHandler" />
This section should be defined as follows:
<add name="ServiceModel" type="System.ServiceModel.Activation.HttpModule, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="managedHandler,runtimeVersionv2.0" />

Note: You can find the Applicationhost.config file in the following location:
%windir%\system32\inetsrv\config
To resolve this issue ASP.NET needs to be registered with .NET Framework 4.0.:

1. Navigate to the following path:
%windir%\Microsoft.NET\Framework\v4.0.30319 (x86 Windows)
OR
%windir%\Microsoft.NET\Framework64\v4.0.30319 (x64 Windows)

2. Run "aspnet_regiis.exe" /iru


Viola! Keep an eye on status manager over the next hour and you will see that this error message will disappear.

Sunday, November 3, 2013

Hyper-V Replica Essentials - Book Review

I have recently read a new eBook on Hyper-V Replica named 'Hyper-V Replica Essentials', written by Vangel Krstevski. This Book provides an easy to read fundamental break down of Hyper-V 2012 Replica and its feature set. The book also provides a simple step-by-step on how to install the base components for a Hyper-V Replica environment:
  • Installing Hyper-V
  • Installing Failover Clustering with Server 2012 shared storage functionality
  • Configuring Hyper-V Replica
  • Configuring alternate authentication types (Kerberos or SSL Authentication)
  • Managing and Administering Hyper-V Replica.
For those of you who would like to get started with Hyper-V 2012 Replica I thoroughly recommend reading this short book.

You can get your hands on a copy of the eBook (PDF, Kindle etc.) by going to Packt Publishing website: http://bit.ly/1aDezz9


Thursday, October 17, 2013

Include Cumulative Update or Patch in SCCM Task Sequence

I think this is fairly well publicised however I thought I would post my way of integrating a Cumulative Update or Patch into a SCCM Task Sequence. I will use the latest Configuration Manager 2012 SP1 Cumulative Update 3 (http://support.microsoft.com/kb/2882125) as an example, however this process can be reused for future Cumulative Update or Hotfix releases.

1. Make a copy of the SCCM Client Source (folder) and copy it to your package source location. The SCCM client can be found under: C:\Program Files\Microsoft Configuration Manager\Client.


2. Depending on the hardware architecture (x86 or x64) copy the associated hotfix from:
C:\Program Files\Microsoft Configuration Manager\hotfix\KB2882125\Client\x64
or
C:\Program Files\Microsoft Configuration Manager\hotfix\KB2882125\Client\i386

Create a x64 and x86 folder under a new 'Patch' folder.

Copy the configmgr2012ac-sp1-kb2882125-<architecture>.msp file to the associated folder in the copied client source.
 
 
3. Create a new package pointing to the copied Configuration Manager Client Install source and name it 'Configuration Manager Client with CU3' or similar. There is no need to create any associated program as the Task Sequence step will take care of the client install.
 
 

4. Open you Task Sequence and select the 'Setup Windows and Configuration Manager' step. Modify the installation properties to include the following switch:
 
PATCH="%_SMSTSMDataPath%\OSD\<packageID\Patch\x64\configmgr2012ac-sp1-kb2882125-x64.msp"
 
or
 
PATCH="%_SMSTSMDataPath%\OSD\<packageID>\Patch\x86\configmgr2012ac-sp1-kb2882125-i386.msp"
  







Monday, October 7, 2013

SCVMM 2012 SP1 Hyper-V host lowercase to uppercase

So for the slightly OCD folks out there who have installed SCVMM, added the hosts to be managed and noticed that some names are in lower case and some are in uppercase. Functionally not a big deal but doesn't look consistent.

I came across this great little tid bit by Larry Rayl: http://blogs.catapultsystems.com/lrayl/archive/2013/05/14/tips-amp-tricks-for-sc-2012-sp1-vmm-fix-for-the-upper-and-lower-case-host-listings.aspx

Simply follow the below steps:

1. Backup the SCVMM Database (SCVMM console or in SQL Management Studio). I need to stress this is a must for any DB level changes required for SCVMM!

2. Log on to SQL Management Studio and apply the following script:

SELECT [ComputerName], UPPER(LEFT([ComputerName], CHARINDEX('.', [ComputerName], 1) -1)) +
            RIGHT([ComputerName], LEN([ComputerName]) - CHARINDEX('.', [ComputerName], 1) + 1)
FROM [VirtualManagerDB].[dbo].[tbl_ADHC_Host]


UPDATE [VirtualManagerDB].[dbo].[tbl_ADHC_Host]
SET [ComputerName] = UPPER(LEFT([ComputerName], CHARINDEX('.', [ComputerName], 1) -1)) +
                               RIGHT([ComputerName], LEN([ComputerName]) - CHARINDEX('.', [ComputerName], 1) + 1)



3. After running the script, restart “System Center Virtual Machine Manager Agent Service” on the VMM Server.

4. After this all hosts will appear in uppercase.

So all those OCD folks out there...can rest at ease!
Thanks Larry!

Sunday, September 29, 2013

Hyper-V 2012 Live Storage Migration Stops at 99% Dell Compellent Storage ODX

Recently with a Hyper-V 2012 implementation I came across an issues where Live Storage Migrations would stop at 99% and would inevitably fail.

The customer was using Dell Compellent Shared Storage with iSCSI connectivity.
The Dell Compellent Storage supported ODX functionality and thus ODX functionality in Windows Server 2012 was enabled.

This issue was resolved with the following Microsoft Hotfix:
Offloaded Data Transfers fail on a computer that is running Windows 8 or Windows Server 2012
http://support.microsoft.com/kb/2796995

Hyper-V 2012 CSV IO issues when deleting or moving Virtual Disks

Recently I was involved in a Hyper-V 2012 implementation. The environment consisted of Dell M620 Blades and Dell Compellent Shared Storage over 10Gbps iSCSI connectivity.

After the implementation, we began migrating VM's from the previous VMware environment to the new Hyper-V 2012 environment. As part of this process we were performing V2V conversions in Virtual Machine Manager 2012 SP1. As you may be aware during a V2V conversion process the VM is converted to VHD, to leverage the new VHDX capacity and performance features we then converted each VHD to VHDX format. This process then involved removing the left over VHD....and this is where things began to appear to be a problem.

During the process of deleting the VHD from a Clustered Shared Volume (CSV) we began to see all the VM's on the hosts to pause or restart. After further investigation we came across the following event entries:

A.After we deleted the vhd files, there is “IO_TIMEOUT” issue reported on the host machine.

00002e64.00001a4c::2013/07/23-13:46:37.844 INFO  [DCM] CsvFs event CsvFsVolumeStateChangeFromIO for volume NSW_NSWHYPERV-01_06_T23_DR:bcd163b1-00bd-4aa0-b524-9e3758f106ec, status STATUS_IO_TIMEOUT(c00000b5)
00002e64.00001a4c::2013/07/23-13:46:37.844 INFO  [DCM] UnmapPolicy::enter_CountingToBad(bcd163b1-00bd-4aa0-b524-9e3758f106ec): goodTimer P0...75, badTimer R0...150, badCounter 1 state CountingToBad

B. To handle the CSV c00000b5 event, the volume state was changed into ‘Draining State’, then ‘Pause’;

00002e64.00001a4c::2013/07/23-13:46:37.849 INFO  [DCM] CsvFs Listener: state [volume bcd163b1-00bd-4aa0-b524-9e3758f106ec, sequence <4:9><23>, state CsvFsVolumeStateChangeFromDrain->CsvFsVolumeStateDraining, status 0x0]
00002e64.00000ac0::2013/07/23-13:46:37.849 INFO  [DCM] FilterAgent: MarkBad: volume NSW_NSWHYPERV-01_06_T23_DR, state ActiveDirectIO, csvfs state CsvFsVolumeStateDraining, status CsvFsVolumeStateDraining
00002e64.00000ac0::2013/07/23-13:46:37.849 INFO  [DCM] FilterAgent: MarkBad() Volume \\?\GLOBALROOT\Device\Harddisk3\ClusterPartition2\ has gone into draining state
00002e64.00000ac0::2013/07/23-13:46:37.851 INFO  [DCM] MappingManager::PauseVolume 'NSW_NSWHYPERV-01_06_T23_DR'
00002e64.00000ac0::2013/07/23-13:46:37.851 INFO  [DCM] FilterAgent: pausing volume bcd163b1-00bd-4aa0-b524-9e3758f106ec, target \\?\GLOBALROOT\Device\Harddisk3\ClusterPartition2\
00002e64.00000ac0::2013/07/23-13:46:37.851 INFO  [DCM] FilterAgent: ChangeCsvFsState: uniqueId bcd163b1-00bd-4aa0-b524-9e3758f106ec, state CsvFsVolumeStatePaused, sequence <4:10><24>
00002e64.00001a4c::2013/07/23-13:46:37.881 INFO  [DCM] CsvFs Listener: state [volume bcd163b1-00bd-4aa0-b524-9e3758f106ec, sequence <4:10><24>, state CsvFsVolumeStateChangeFromPause->CsvFsVolumeStatePaused, status 0x0]
00002e64.00000ac0::2013/07/23-13:46:37.881 INFO  [DCM] short name is C:\CLUSTE~1\NSW_NS~4
00002e64.00000ac0::2013/07/23-13:46:37.881 INFO  [DCM] PauseNFilter bcd163b1-00bd-4aa0-b524-9e3758f106ec, NTFS: 00000000-0000-0000-0000-000000000000
00002e64.00000e24::2013/07/23-13:46:37.881 INFO  [DCM] Set NfltPauseComplete event for  bcd163b1-00bd-4aa0-b524-9e3758f106ec
00002e64.00000ac0::2013/07/23-13:46:37.881 INFO  [DCM] PauseNFilter completed for bcd163b1-00bd-4aa0-b524-9e3758f106ec
00002e64.00000ac0::2013/07/23-13:46:37.881 INFO  [DCM] FilterAgent: PauseSnapshots completed for NSW_NSWHYPERV-01_06_T23_DR:bcd163b1-00bd-4aa0-b524-9e3758f106ec
00002e64.00000ac0::2013/07/23-13:46:37.881 INFO  [DCM] volume paused 'NSW_NSWHYPERV-01_06_T23_DR'
00002e64.000031a4::2013/07/23-13:46:37.881 INFO  [DCM] PauseDisk completed for resource 'NSW_NSWHYPERV-01_06_T23_DR'

C. After several minutes, the CSV comes online.
00002e64.00000ac0::2013/07/23-13:49:15.602 INFO  [DCM] ActivateNetworkPath bcd163b1-00bd-4aa0-b524-9e3758f106ec completed
00002e64.00000ac0::2013/07/23-13:49:15.602 INFO  [DCM] ActivateVolume: NSW_NSWHYPERV-01_06_T23_DR:bcd163b1-00bd-4aa0-b524-9e3758f106ec
00002e64.00000ac0::2013/07/23-13:49:15.602 INFO  [DCM] Bitlocker not installed or loaded
00002e64.00000ac0::2013/07/23-13:49:15.602 INFO  [DCM] FilterAgent: ChangeCsvFsState: uniqueId bcd163b1-00bd-4aa0-b524-9e3758f106ec, state CsvFsVolumeStateActive, sequence <><31>
00002e64.00001a4c::2013/07/23-13:49:15.602 INFO  [DCM] CsvFs Listener: state [volume bcd163b1-00bd-4aa0-b524-9e3758f106ec, sequence <><31>, state CsvFsVolumeStateChangeFromActivate->CsvFsVolumeStateActive, status 0x0]

During a period of up to 15 minutes, VM's would be hung or restarted. The recovery of the VM's would vary and in some extreme cases we actually had data corruption.

Needless to say we got Microsoft Premier Support involved.

We were quickly advised to apply: http://support.microsoft.com/kb/2813630/en-us
Disable ODX in our environment:
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 1

However we were reluctant to do this as we in fact had a nice new storage array that supports this super cool functionality. We applied the hotfix and didn't disable ODX (as mentioned above).

Unfortunately this didn't resolve our issues...DAMN!

We then came across the following hotfix's and also applied these to all hosts:
KB2796995
KB2838043 


Unfortunately this didn't resolve our issues either...DAMN!

We then went and tried to update the Broadcom Network Drivers on all hosts
No luck either!

Finally we applied the following hotfix (KB2870270):
http://support.microsoft.com/kb/2870270

Horaaa!!!! This was the lucky hotfix. This took over a month to resolve and was a very time consuming and frustrating process. I hope this helps anyone else out there who is currently going through similar pain...







Wednesday, September 25, 2013

Pause SCCM Task Sequence with PowerShell

Lately I have found myself adding allot of pauses into Task Sequences for some buggy applications. It seems to be allot more prevalent in SCCM 2012 SP1 and Windows 7 Task Sequences.

Anyhow for whatever reason you need to create a pause in a Task Sequence (Sophos is an application that comes to mind) here is the steps:

1. Create a 'Run Command Line' step in the task sequence.

2. Add the following PowerShell command to the command line field in the new step:
powershell.exe -executionpolicy bypass -command "Start-Sleep -s 120"

Note: -s is used to specify the number of seconds to pause for. -m switch can be used however this is milliseconds NOT minutes :)

Happy Task Sequencing!

Thursday, September 12, 2013

Disable Features in SCCM Task Sequence

Back doing an SOE for a customer and thought I would share how i disable and enable features in a SCCM Task Sequence. This is a good alternative if you do not want to modify the unnatend.xml file.

For a while now DISM.exe has been used to modify features and files in a WIM image. I have put together a simple command that will disable a feature. You can run this within your task sequence as a seperate command or run multple iterations within a batch script.

Here is an example of how to disable all the games in a Task Sequence:

start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:InboxGames /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:"More Games" /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:Solitaire /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:SpiderSolitaire /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:Hearts /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:FreeCell /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:Minesweeper /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:PurblePlace /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:Chess /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:Shanghai /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:"Internet Games" /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:"Internet Checkers" /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:"Internet Backgammon" /Quiet /NoRestart
start "" /w /d "%SystemRoot%\System32" "dism.exe" /online /disable-feature /featurename:"Internet Spades" /Quiet /NoRestart


I have put these commands into a batch file named customisations.bat. I then add the customisations to a package in SCCM.

From there i create a command line step in the Task Sequence to the batch file from the package.




Monday, September 9, 2013

Testing Windows Server 2012 R2 Hyper-V Storage QoS with IOMeter

Recently I did some testing of Storage QoS in Server 2012 R2 with IOMeter. I was quite impressed that IO can quickly be reduced on an IO hungry VM.
For those who haven't heard of this great new functionality.

Storage Quality of Service (QoS)
Provides the ability to specify maximum and minimum I/O loads in terms of operations per second (IOPS) for each virtual disk on the VM.
Prevents VM’s from consuming all of the available I/O bandwidth to the underlying physical resource.
Supports Dynamic, Fixed and Differencing Virtual Disks.



Here is a demo of this in action. At first I maxed out the IOPs with IOMeter, I then went on to configure Storage QoS and straight away the IOP's are reduced.


SCCM 2012 SP1 Reports not running from the Config Manager Console

Recently I had an issue running SCCM 2012 SP1 reports from within the Configuration Manager Console. They would run from SQL Reporting Services fine.
The SQL Reporting Services were running from a remote server.

I checked the smsadminui.log a came across the following error:
System.BadImageFormatException


The issue was a result of Report Viewer not working as expected.
To resolve the issue I removed the Report Viewer and reinstalled.
You can re-run the Report Viewer from the following path:
C:\Program Files\Microsoft Configuration Manager\AdminConsole\bin\ReportViewer.exe

Sunday, September 8, 2013

Hyper-V Replica over a dedicated network


Recently I was involved in a Hyper-V Replica engagement with one of our large customers. One of the project requirements was to use a dedicated and isolated Replication Network.

The customer required Hyper-V Replica to replicate VM's between two Hyper-V Failover Clusters and as a result the Hyper-V Replica Broker role was installed http://blogs.technet.com/b/yungchou/archive/2013/05/05/hyper-v-replica-broker-explained.aspx

 
As you may have already discovered is that by default the Hyper-V Replica IP will be configured on the Parent Network. This is problematic when requiring large amounts of data to be replicated over a shared management network.


Before beginning it is important that the following is ready:
  • Your dedicated network or team.
  • You have an internal Certificate Authority (if you are using an isolated network). Alternatively you can make your Replica Network routable and you can use Kerberos Authentication for Hyper-V Replica. If this is a test environment a self-signed certificate will suffice also J
  • Create a Server and Client Authentication Certificate Template that allows the use of Subject Alternative Names. This is very important when replicating between Failover Clusters. This blog provides great detail on how to configure a Hyper-V Replica Certificate Template: http://blogs.technet.com/b/virtualization/archive/2012/07/10/requesting-hyper-v-replica-certificates-from-an-enterprise-ca.aspx
 

To configure Hyper-V Replica over a dedicated and isolated replication network there are a few things I have discovered along the way. Here are the steps I took:


1. Ensure you have adequate teamed network to provide Hyper-V Replica Traffic, in this instance I wanted to share the Hyper-V Replica traffic with the CSV Network. In an ideal world you would have a dedicated team or converged fabric for this.


2. Before running the Hyper-V Replica Broker Role install. The Network must be configured to ‘Allow clients to connect through this network’ in Failover Cluster Manager, Networks. Right click the network you want to use for Hyper-V Replica traffic and click ‘Properties’
 


3. Install the Hyper-V Replica Broker role with your required name and corresponding IP Address, note that by default the Parent Network is selected. You need to change this to the dedicated network team. Replica Broker Install steps are here: http://technet.microsoft.com/en-us/library/jj134153.aspx#BKMK_1_4

4. Request a certificate from you CA, the requirements for the Certificate is to ensure that the following names exist as Subject alternative names (alternatively for large environments you could use a wildcard). As mentioned follow this great blog post on how to create the Certificate Template and Request from the Hyper-V Hosts.

                - All Hyper-V Replica Broker Role Fully Qualified Domain Names.

                - All Hyper-V Host Fully Qualified Domain Names.

5. Hyper-V Replica can use one of two primary methods to trust inbound/outbound replication, Kerberos authentication or Certificate-based authentication. By default, Hyper-V Replica is configured using Kerberos authentication, in order for Kerberos authentication to function the Hyper-V Replica Network must be able to communicate with Active Directory. In scenarios where Hyper-V is not Domain-joined or the Hyper-V Replica network is isolated, certificate-based authentication is the only authentication method that can be leveraged.
 
6. If you are using separate networks for Hyper-V Replica Traffic, you will need to configure a persistent route to ensure Hyper-V Replica traffic will route correctly. Use route add –p command on all Hyper-V hosts.


7. Configure your Hosts file to include the FQDN and NetBIOS names of each Hyper-V host that needs to be replicated “to and from”. It was discovered the Hyper-V Replica Broker will still use DNS to resolve the names of each Hyper-V host and as a result will still resolve the Parent Network IP Address. The only option is to update the C:\windows\system32\drivers\etc\hosts file to include the replica network IP address on EVERY SINGLE HYPER-V Host.





8. You can now configure Hyper-V Replica on your Virtual Machines. Check Task Manager Network performance to ensure that the Initial Replication is copying across the dedicated Hyper-V Replica Network.

That’s it! This will need to configured identically on the corresponding Replica Site.

 

Thursday, August 29, 2013

SCCM 2012 SP1 Prerequisites - PowerShell Script


For those of you who have done a SCCM 2012 SP1 install would know it can be quite frustrating making sure all the prerequisites are in place before kicking off an install.

I particularly struggle with the Application Catalogue pre-requisites.

 
I came across this great script that will install the required prerequisites for any role and site server combo:


 
Here are the different combinations, as you can see almost every prerequisite scenario is covered off:

.\Install-SCCMPrereqs.ps1 -SiteType CAS

.\Install-SCCMPrereqs.ps1 -SiteType Primary

.\Install-SCCMPrereqs.ps1 -SiteType Secondary

.\Install-SCCMPrereqs.ps1 -SystemRole ApplicationCatalog

.\Install-SCCMPrereqs.ps1 -ExtendSchema

.\Install-SCCMPrereqs.ps1 -InstallWADK

.\Install-SCCMPrereqs.ps1 -InstallWSUS

Thursday, August 1, 2013

Brisbane Hyper-V User Group - New Facebook Group

Aaron Whittaker - Brisbane Hyper-V User Group Host has opened a new Facebook Group for Brisbane Hyper-V professionals to collaborate and share their experiences. Feel free to join.                                                                                                                                     
https://www.facebook.com/groups/bnehyperv

Sunday, July 7, 2013

Data Protection Manager 2012 SP1 and Azure - Long Term Data Retention Options

I recently had an engagement with a customer to provide a solution to facilitate long term retention (10+ years) of backups to the "cloud". The customer wanted to use Microsoft System Center 2012 Data Protection Manager (DPM) with Azure Storage to retain long term backups.

The requirements were as followed:
  • The solution must leverage Microsoft DPM 2012 SP1.
  • Short term data retention (1 week) should be kept on premise on a local DPM Storage Pool.
  • Long term data retention (10 + years) should be kept in Windows Azure Storage.
  • No tapes are to be used for Long term data retention.
What a great idea! Keep short term data on premise for quick DR recovery (short RTO) and use windows Azure to recover ad-hoc files, mailboxes etc (Longer RTO).

Many of you may already know that DPM (as of 2012 SP1) has an inbuilt Windows Azure connector to retain backups in the cloud. See TechNet: http://technet.microsoft.com/en-us/library/jj728752.aspx
This at first seems like a great option (A single solution and single interface for long term data retention in Azure).

During discovery and testing the following roadblocks/issues for meeting the long term retention requirements were encounted with the inbuilt DPM to Azure solution:
  • The maximum retention range for Windows Azure backups of DPM is 120 days. The retention range depends on the synchronization settings. If you set to two backups per day then the retention range will be 60 days.
  • You can synchronize to Windows Azure Backup at a maximum of twice per day.
  • A maximum of 448 days is available for Disk Based Recovery (http://technet.microsoft.com/en-us/library/jj628009.aspx)
At this stage Microsoft's stance on long term retention of backup data (over 448 days Disk Based and 120 days Azure based) is for data to go to tape. With the recent acquisition of StorSimple (http://www.storsimple.com/) Appliances by Microsoft this may potentially change in the future.

In the mean time there is still a solution or workaround to retain long term data in Azure and that is to "trick" DPM into thinking it is writing to a tape by using a Virtual Tape Library (VTL) Product. Another cloud appliance (such as StorSimple or the Riverbed WhiteWater) is then needed to copy the backup data to the cloud for long term retention.

At a high level the solution would look like this:
  • Short term data retention for protection groups (under 448 days) to on premise DPM Disk (Storage Pool).
  • Install a Virtual Tape Library (VTL) on the DPM Server such as FireStreamer (https://www.cristalink.com/fs/).
  • Configure the Virtual Tape Library (VTL) to write it's files to a cloud appliance such as Riverbed WhiteWater (http://www.riverbed.com/products-solutions/products/cloud-storage-whitewater/) or StorSimple (http://www.storsimple.com/).
  • Configure the cloud appliance to connect to Windows Azure and setup the disk and azure retention policies i.e. when data is over a week old copy the data up to Azure storage.
  • Configure your Protection Groups to retain long term backups using tape and point it to the Virtual Tape Library.

The solution is not a fully integrated solution (I have faith Microsoft will introduce something in the future). However once configured the long term retention of data is treated as a traditional Tape method, the good thing is there is no requirement for physically managing the tapes off site (this is all handled by the cloud appliance.

I hope this is useful for anyone thinking of using DPM 2012 SP1 for long term backup (over 120 days) retention in Azure.

Monday, June 3, 2013

New Features: Windows Server 2012 R2

With the unveiling of Windows Server 2012 R2 features at Microsoft TechEd North America there has been allot of talk about the new features that are coming our way.

I came across a New Features article for Windows Server 2012 R2 on Windows IT Pro:
http://windowsitpro.com/windows-server-2012/new-features-windows-server-2012-r2

Some of the new features (specifically around the Hyper-V role) that I am looking forward  to seeing are:
  • Compression of Memory for Live Migrations. This will dramatically reduce Live Migration times.
  • Dynamic VHDX resizing. Finally we will be able to expand Virtual Disks connected to the virtual SCSI controller. This will be a feature that many Hyper-V customers have been hanging out for!
  • Hyper-V Replica Frequency. This will add new functionality from the standard 5 minute replication intervals by providing 30 seconds, 5 minutes and 15 minute intervals to choose from. This will provide more functionality for different link speeds and DR requirements.
  • DeDuplication of VHDX. By removing common blocks this not only going to save space but is set to dramatically increase performance.
  • Shared VHDX. In cases where Guest level clustering is required, this provides an alternative solution that will not require the Guest to connect directly to the Sstorage Array (using iSCSI or virtual Fibre Channel). Instead multiple guests can connect to a VHDX at the same time to provide the "Shared Storage" functionality required.
  • USB Pass through. Another great functionality that makes Hyper-V more competitive to the other Virtualisation Products on the market.
Windows 2012 R2 Preview Release is set for end of June 2013.
Final versions of Windows 2012 R2 is heard to have a release date of end of calendar year 2013.

Looking forward to it!

Friday, May 17, 2013

V2V or P2V failed state and cannot remove VM

Recently I was working with a customer on a VMware to Hyper-V migration. We were leveraging the great inbuilt V2V tool that is part of Virtual Machine Manager 2012. At some point during the virtual-to-virtual conversion the process failed.

After the conversion failed the VM was listed in Virtual Machine as 'Virtual-to-virtual conversion failed'. I decided to clean up the vm and restart the process.
- I tried to delete the VM with no luck.
- I then tried to repair the VM and this also failed as well.

Repairing in the past generally resolves a "stuck" VM. As a result I had to follow the below procedure to clear the error message and allow the VM to be deleted.

Please note this procedure is not a Microsoft official supported procedure and should be carried out with caution (make sure you take a backup of the VMM Datbase):

Backup the VMM 2012 Database:
1. Launch the Virtual Machine Manager Console and go to the 'Settings' view. In the settings view click 'Backup'.





















Edit the relevant table in the VMM Database:
1. Launch SQL Management Studio. Expand the 'VirtualManagerDB' database.




 

2. Expand 'Tables' and scroll down and find 'dbo.tbl_WLC_VObject'.














3. Edit the table and search for the "stuck" VM.

4. We then want to find the 'ObjectState' column for that VM and change the value to '1' and press Enter.
 





















5. In Virtual Machine Manage refresh the Cluster.

6. After the Cluster refreshes the VM can now be deleted.

Check Hyper-V Integration Services Versions

Ensuring that all your hosts have Integration Services Installed and are all the same version is important to keep your Hyper-V environment in a optimal/stable state.

I use the following PowerShell command to list all VM's and associated Integration Services version on a Hyper-V host:
Get-VM | ft Name, IntegrationServicesVersion

I use the following PowerShell command to list all VM's and associated Integration Services version on a entire cluster:
Get-ClusterGroup | Get-VM | ft Name, IntegrationServicesVersion

Hyper-V 2012 Best Practices

I came across a great article on Hyper-V 2012 Best Practices, written by a Microsoft Premier Field Engineer:

Wednesday, May 8, 2013

VMM 2012 SP1 VM Template - Region Settings revert back to en-US

I thought I would share a common problem that is experienced when configuring Regional Settings on a VM template.

Here is the scenario:
You build your VM with your required OS and make the required customisations for your environment. In most cases (unless you are in the US) this would require changing the Regional Settings. Once you are happy you run the sysprep oobe - generalize and shutdown the VM ready to be moved to the library.

You generate the VMM template from the VM in your library. You run your first test a realise that all the Regional setting have been changed back to English US.

During the above process the regional settings is reverted back to 'en-US'. This functionality is acknowledged in this MS KB article: http://support.microsoft.com/kb/2709539

To configure the proper regional settings for your template you can run a number of PowerShell cmdlets that will update the unattend settings (specifically oobeSystem/Microsoft-Windows-International-Core/UserLocale).

For Australia I generally need to configure the User Locale to 'en-AU' (English-Australia) and the keyboard to 'en-AU' also.

To do this I run the following PowerShell cmdlets (no need to change SystemLocale or UILanguage for Australia region settings):

$template = Get-SCVMtemplate | where {$_.Name -eq "Template - Windows Server 2008 R2 SP1 Enterprise"}
$settings = $template.UnattendSettings;
$settings.add("oobeSystem/Microsoft-Windows-International-Core/UserLocale","en-AU");
$settings.add("oobeSystem/Microsoft-Windows-International-Core/SystemLocale","en-US");
$settings.add("oobeSystem/Microsoft-Windows-International-Core/UILanguage","en-US");
$settings.add("oobeSystem/Microsoft-Windows-International-Core/InputLocale","0c09:00000409");
Set-SCVMTemplate -VMTemplate $template -UnattendSettings $settings



The above PowerShell commands can be modified for your region.

VMM VM Template - AutoLogonCount and AutoLogonCredential for RunOnce

Recently I created a VM template in VMM 2012 SP1 for a customer. A number customization scripts were created and added to RunOnce. The customer wanted the VM template to automatically logon with the local Admin account so all the RunOnce scripts would run automatically.

This is a pretty standard request and when configuring any unattend Windows install this is generally achieved by modifying the unnattend.xml file. However in VMM the Set-SCVMTemplate PowerShell cmdlet can be leveraged to configure this setting.

The 'Get-SCVMTemplate' cmdlet will show the current AutoLogon configuration. The AutoLogonCount and AutoLogonCredentialproperties must both be configured to achieve the Auto Logon functionality:

Get-SCVMTemplate -Name "VM Template Name"

When using AutoLogon in your VM Template it is also important to have a VMM 'Run As' account configured that has the local Admin credentials stored. Once the local Admin 'Run As' account is created, add it to the VM Template OS Properties for the local admin account.

The following PowerShell cmdlets can be used to configure AutoLogonCount and AutoLogonCredential properties:

#Get Template by name
($template = Get-SCVMtemplate | where {$_.Name -eq "VM Template Name"}) | fl Name, OperatingSystem, VirtualizationPlatform


#Get local admin account
($raa = Get-SCRunAsAccount -Name "VM Local Admin Run As") | fl Name, Username, Domain


#Set auto logon
(Set-SCVMTemplate -VMTemplate $template -AutoLogonCredential $raa -AutoLogonCount 1) | fl Name, AutoLogonCredential, AutoLogonCount



By using Run As account for the local admin account for the OS Properties and the AutoLogonCredential this allows the local admin account to be changed without needing to rerun the above PowerShell scripts.

Happy Templating!

Wednesday, May 1, 2013

ODX Trap! DPM 2012 & Hyper-V 2012 a CSV volume goes offline when you try to create a backup

I recently assisted a customer that was experiencing outages of their Hyper-V VM's that corresponded with a DPM 2012 CSV snapshot. After further investigation the following Failover Cluster event logs were found (around the same time of the VM outages):

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: Date and time
Event ID: 5120
Task Category: Cluster Shared Volume
Level: Error
Keywords:
User: SYSTEM
Computer: Computer name
Description: Cluster Shared Volume 'Volume1' ('name’) is no longer available on this node because of 'STATUS_IO_TIMEOUT(c00000b5)'. All I/O will temporarily be queued until a path to the volume is reestablished.

Log Name: System
Source: Microsoft-Windows-FailoverClustering
Date: Date and time
Event ID: 5142
Task Category: Cluster Shared Volume
Level: Error
Keywords:
User: SYSTEM
Computer: Computer name
Description: Cluster Shared Volume 'Volume3' ('Cluster Disk 4') is no longer accessible from this cluster node because of error 'ERROR_TIMEOUT(1460)'. Please troubleshoot this node's connectivity to the storage device and network connectivity.

It was evident that DPM 2012 snapshots were causing the IO errors. I came across this hotfix: http://support.microsoft.com/kb/2813630

The hotfix was applied across all the Hyper-V hosts.

A great feature of Windows 2012 is that it is Offloaded Data Transfer (ODX) capable straight out of the box! However when your Storage Array is not ODX capable this is where the CSV I/O errors will start occurring on snapshot operations.

As a result I had to run the following PowerShell command to turn off ODX on all the Hyper-V hosts (see: http://technet.microsoft.com/en-us/library/jj200627):

Check to see if ODX is Enabled/Disabled
Get-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode"

Disable ODX
Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "FilterSupportedFeaturesMode" -Value 1

This is my first Private Cloud and Hyper-V blog (it is also my very first blog). I'm a Consultant for a Brisbane Integrator and I specialise in System Center and Hyper-V technologies. I have decided to share my journey and findings with everything that entails Private Cloud, System Center and Hyper-V.

I hope to start off slowly with a few blogs here and there over the next few months and let's see what happens from there!