Ken Schaefer

Debugging an IIS / SharePoint performance issue using WinDBG

Ken — Wed, 01 Jul 2009 11:19:00 GMT

After a recent SharePoint 2007 migration exercise at a large customer, we started experiencing performance issues reported by end users when the system was under load. In the Windows Event Logs we saw the following event:

Event Type: Warning
Event Source: W3SVC-WP
Event ID: 2262
Description:
ISAPI 'C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

IIS 6.0 would shortly recycle the web application pool, but during that period users directed to that WFE server by the load balancer weren't able to have their requests served. After looking through obvious causes, we ended up obtaining a dump file of the worker process. We used the process of orphaning a worker process but a dump file can just as easily be obtained using common tools such as IISState or IIS DebugDiag.

Once the dump file was obtained I performed the following analysis.

Start WinDBG and point it the dump file. WinDBG is part of the Debugging Tools for Windows

Since this is a SharePoint issue, I suspected that we may need to investigate both managed and native code. To assist with debugging managed code we need to do a bit of extra preparatory work:

Load the SOS (Son of Strike) extension: .load SOS
Make available a copy of mscordacwks.dll to WinDBG. Note that this has to match the version on the server where the dump file was taken from. It's easiest to just copy this from the server (from the %systemroot%\Microsoft.Net\Framework\<frameworkVersion> folder. It will have to be renamed to mscordacwks_xxx_xxx_2.0.50727.yyyy.dll - where XXX is either I386 or AMD64 depending on the platform, and yyyy is the version number (this can be found by right-clicking on the .dll file and checking the version number). WinDBG will tell you what the actual filename you need is if you get it wrong. You can just copy-n-paste that value to assist in renaming the file
To make available this file copy it into a folder of your choosing and add it to the WinDBG symbols path. I copied it into c:\temp, and to add that to the path type: .sympath+ c:\temp
Add the Microsoft public symbol server to the path (if it isn't already). To do this type: .symfix
Now acquire symbols for Microsoft modules in the dump by typing in: .reload

Now we can begin analysis of the dump file. The first command gives us an indication of how long threads have been running in the process:

0:031> !runaway
User Mode Time
Thread       Time
13:1218      0 days 0:00:56.437
37:138c      0 days 0:00:32.718
31:b80       0 days 0:00:32.406
43:1208      0 days 0:00:30.125
33:17ec      0 days 0:00:23.578
38:fa8       0 days 0:00:14.937
36:147c      0 days 0:00:13.609
40:1680      0 days 0:00:11.468
15:153c      0 days 0:00:08.359
44:15a4      0 days 0:00:07.968
16:12e4      0 days 0:00:06.968
27:1554      0 days 0:00:05.750
25:16b0      0 days 0:00:05.656
24:15f8      0 days 0:00:05.453
35:17a8      0 days 0:00:05.406
26:d98       0 days 0:00:05.359
41:cb0       0 days 0:00:05.296
11:d50       0 days 0:00:05.031

We can see here that we have quite a few very long running threads (there were 71 threads in total in this dump file - I have truncated the list).

The next step is to have a look at the longest running threads and see what they are doing. To change to a thread type: ~<threadnumber> s. In this case we change to thread 13 (the longest running thread) and then dump the unmanaged stack:

0:013> ~13 s
ntdll!NtWaitForSingleObject+0xa:
00000000`77ef0a3a c3 ret
0:013> k
Child-SP RetAddr Call Site
00000000`0309aa68 000007ff`771d5280 ntdll!NtWaitForSingleObject+0xa
00000000`0309aa70 000007ff`7731173e mswsock!WSPRecv+0x66b
00000000`0309abb0 000007ff`770f3518 ws2_32!WSARecv+0x166
00000000`0309ac60 000007ff`72499c0e wsock32!recv+0x38
00000000`0309acc0 000007ff`5fe51ae7 dbnetlib!ConnectionRead+0x4fe
00000000`0309ada0 000007ff`5fe52f2d sqloledb!CDataSource::ConnectionTransact+0xf7
00000000`0309ae20 000007ff`5feaac34 sqloledb!CDBConnection::SendPacket+0x25d
00000000`0309aee0 000007ff`5fe44b69 sqloledb!CStmt::SQLExecRPC+0x4d4
00000000`0309aff0 000007ff`5fe45df3 sqloledb!CCommand::ExecuteHelper+0x2a9
00000000`0309b090 000007ff`5fe46cc7 sqloledb!CCommand::Execute+0xc73
00000000`0309b180 000007ff`5aa0e4cc sqloledb!CImpICommandText::Execute+0x187
00000000`0309b200 000007ff`5aa0f65b oledb32!CCommandText::DoExecute+0x4fc
00000000`0309b440 00000000`0a913a72 oledb32!CCommandText::Execute+0x8ab
00000000`0309b6f0 00000000`0a84241f STSWEL!Voledb::ExecQuery+0x37e
00000000`0309b850 00000000`0a8846bd STSWEL!VdocumentStore::httpGetDocument+0xbdf
00000000`0309c850 00000000`0a886720 STSWEL!VhttpManager::loadFileCore+0x5c5
00000000`0309d0f0 00000000`3569fa9b STSWEL!VhttpManager::loadFileAndMetaInfo+0xc4
00000000`0309d1b0 00000000`356aaa8c OWSSVR!GetExtensionVersion+0x890af
00000000`0309d570 00000642`7f600887 OWSSVR!GetExtensionVersion+0x940a0
00000000`0309d8f0 00000642`806070ff mscorwks!DoCLRToCOMCall+0x177

Note: to dump the managed stack type !clrstack (this requires SOS extension to be loaded). I did dump the managed stack but it isn't relevant in this case. The oldest item on the native stack is a call from managed code into native code (mscorwks!DoCLRToCOMCall) and examining the managed stack doesn't tell us anything.

From the native stack we can see a few things (read a stack from the bottom up):

We have some SharePoint related components (OWSSVR and STSWEL) that are called.
Eventually these call into OleDb
OleDb then appears to call into the SQL Server OleDb Provider (sqloledb)
The SQL Server OleDb Provider then calls into WinSock to send a command across the network to a remote SQL Server
The stack ends with the Windows Socket in a receive state awaiting a response (ws2_32!WSARecv)

I examined most of the threads running in the process, and a vast majority of ASP.NET worker threads where in a similar state. Our next step is to try to figure out what SQL command or stored procedure we are calling.

To do this, I took a bit of a guess (I suppose I could have look at the definitions of the OleDb APIs etc). I suspected that a function such as oledb32!CCommandText::Execute would probably have, as one of it's parameters, the actual command to be executed (mirroring the OleDb connection and command objects' .Execute method)

Using the kb command we can get parameter information. The address of the third parameter was 00000000`0309bc40 (note this dump is from an x64 system).

Our next step is to examine some memory around that parameter address. It turns out that approximately 240 bytes further we can find the stored procedure being called:

0:013> dc 00000000`0309bc40+0x240
00000000`0309be80 0ef44048 00000000 110a0008 00000000 H@..............
00000000`0309be90 0f04afe8 00000000 00000000 00000000 ................
00000000`0309bea0 0309bea8 00000000 003f007b 0063003d ........{.?.=.c.
00000000`0309beb0 006c0061 0020006c 00720070 0063006f a.l.l. .p.r.o.c.
00000000`0309bec0 0046005f 00740065 00680063 006f0044 _.F.e.t.c.h.D.o.
00000000`0309bed0 00460063 0072006f 00740048 00700074 c.F.o.r.H.t.t.p.
00000000`0309bee0 00650047 00280074 002c003f 002c003f G.e.t.(.?.,.?.,.
00000000`0309bef0 002c003f 002c003f 002c003f 002c003f ?.,.?.,.?.,.?.,.

Looking through the other threads, the majority are calling this stored procedure, with a few calling a different one. it maybe that this one sproc is blocking itself, or the interaction of these two are blocking each other. Or possible some other issue in the cluster. For now, our job is done, and we hand over to the DBA team to do some investigation into what is happening in SQL Server.

Note: Due to blog spam, comments are disabled. Please use the contact form for questions.

Windows Home Server and IIS 7.0 restore problems

Ken — Wed, 01 Jul 2009 11:09:00 GMT

Windows Home Server automatically ignored folders called "temp" when configuring backups. Normally this isn't a problem. However IIS 7.0 does create a folder called temp (by default at c:\inetpub\temp. In this location are stored application pool configuration files that are generated on-the-fly by IIS.

It appears that the WHS bare-metal restore doesn't restore this folder, and IIS 7.0 then can experience issues (in addition to any other folders named "temp") that might not exist. The solution may be as simple as creating a folder called "temp" for IIS to store app pool config files in.

Note: comments are disabled due to blog spam. Please use the contact form.

Dell Latitude E6400, STOP 0x101 and Windows Server 2008 R2 Hyper-V

Ken — Mon, 06 Apr 2009 11:55:00 GMT

My work recently gave me a new Latitude E6400 (the E6500 was just a bit too heavy) which is a great machine for running virtual machines out of the office. It has an internal 7200 RPM drive, a modular drive (120GB 1.8" 5400 RPM) and I can connect an external 2.5" 7200 RPM drive via the eSata port. It also supports up to 8GB of RAM, and has a C2D 2.8GHz CPU. Considering where we were only 4-5 years ago with laptops, it's an amazing advance.

I installed Windows Server 2008 R2 Build 7000 (Beta 1) and the Hyper-V v2 role and almost immediately started getting STOP 0x00000101 BSODs (CLOCK_WATCHDOG_TIMEOUT). Ben Armstrong reports this is a known issue in Beta 1. Mike Kolitz suggested, based on dump files, that this might have been fixed in build 7006. Luckily my work's TAP program has access to various interim builds. I pulled down Build 7068, and no more BSODs. Unfortunately the Intel WLAN drivers seem to crash my WAP. One step forward...one step back :-)

Home Data Centre Upgrade #3

Ken — Thu, 26 Mar 2009 09:32:00 GMT

After adding extra disks to the home "data centre" and then getting a new backup device (Dell RD1000), today I took a punt on upgrading the RAM. When I originally bought my Dell PE SC1430 it was rated at a maximum of 8GB RAM. The latest models offer a maximum of 16GB of RAM, and after finding nothing on the interwebs that might indicate that my particular model couldn't cope with more than 8GB, I tentatively splashed out on an extra 2x4GB FBDIMMS. It seems all PE SC1430s (with the latest BIOS) support more than 8GB of RAM (well, at least 12GB, and I suppose 16GB if you populate 4x4GB).

Currently my production VMs consume around 7GB of RAM (Exchange and Ops Manager 2007 around 2GB each, plus Windows Home Server takes another 1GB), so testing new OSes (like Windows Server 2008 R2) is difficult. With the extra unallocated RAM, it'll be easier to test beta server OSes :-)

MVP Summit 2009

Ken — Fri, 20 Mar 2009 12:05:00 GMT

Earlier this month I was lucky enough to attend the 2009 Microsoft MVP summit in Seattle. The bulk of the summit consisted of two days of sessions with our product teams (I popped across to some Directory Services sessions as well), and a one day executive keynote session.

Some heavy hitters turned up for the executive keynote - Steve Ballmer was good value as always

Soma managed to rope in four of Microsoft's technical fellows - some of the heaviest technical hitters in the company for a Q&A around Microsoft's future developer direction. It was a pity that so much of the Q&A time for this session was wasted with questions and general complaining that wasn't relevant to any of the people on stage.

For two days I was out at Redmond - building 42 - where the IIS team is based:

Conference Room 2200 was where our sessions were held:

Tomorrow I'll be writing up a follow up post on topics covered.

On an unrelated note I've also become a bit addicted to FlightMemory - a site where you can record flights taken. Inputting the flights that I still have records for, I've flown 274 flights totalling around 588,000 kms in the last 6 or so years. The site generates nice maps as well:

Internet Explorer 8 - now available

Ken — Fri, 20 Mar 2009 11:53:00 GMT

It's all over the web, and now here too. Internet Explorer 8 available for download from Microsoft's website. Unfortunately no update for the rather buggy IE8 that shipped with Windows 7 betas.

IIS and Kerberos Part 9 - Cross Forest Delegation scenario with UPN suffix routing

Ken — Thu, 26 Feb 2009 11:44:00 GMT

As an extension of the previous article on Cross Forest (or Cross Domain) Kerberos Authentication this article examines how to configure cross forest authentication and delegation when users are accessing an arbitrary website URL.

In this scenario we have the same two Forests as in Part 8. Forest A (domainA.local) contains our resource servers (web server and SQL server). Forest B (domainB.local) contains our users and client PC.

Users are going to access a web site at www.myCompany.com, a domain that has no direct relationship between either the resource domain or user domain. Companies might need to implement this type of setup when they wish to have a single URL that users on either the internal network or externally can access. Alternatively I have seen scenarios where companies what to have a portal address (e.g. intranet.company.com) that then reverse proxies a number of internal web applications, and Kerberos authentication and transparent delegation to the proxied web applications makes for a simplified user experience.

A diagram of the process involved:

Wireshark/Ethereal packet captures of the actual traffic are available for download (rename to .pcap). I’ll explain the packets to look for a bit further down in the blog post.

The configuration steps required for this setup are:

Determine some mechanism so that the users can resolve www.myCompany.com (DNS is a given, but if you are using a split-brain DNS then your internal DNS will need to have an appropriate zone as well as your public DNS)
Create an additional UPN (user principal name) suffix in your resource Forest (domainA.local). To do this:
- Open the Active Directory Domains and Trusts Administrative Tool
- Right-click on the top level "Domains and Trusts" node
- On the UPN suffixes tab add www.myCompany.com and click Add. Note: you can add myCompany.com and this will add all hosts under myCompany.com. Adding www.myCompany.com will also work (but will also permit hosts under www.myCompany.com such as www.www.myCompany.com)
Configure Name Suffix Routing across the Forest Trust. To do this:

Open the Active Directory Domains and Trusts Administrative Tool in DomainB.local (the user domain)
Right-click on your domain (domainB.local) and choose Properties
On the Trusts tab select DomainA.local under either options (Domains trusted by this domain or Domains that trust this domain) – it doesn’t matter which one. Click the Properties button
On the Name Suffix Routing tab select *.www.myCompany.com and click Enable
Click OK to exit all the dialogues

Steps 4 & 5 are generic Kerberos configuration steps that aren’t specific to cross-Forest scenarios: Add the requisite SPN (Service Principal Name). To learn about SPNs review Part 2 in this series. In this case we need to add an SPN for http/www.myCompany.com in domainA.local. If the web application pool is running under Network Service, Local Service or LocalSystem the SPN should be added under the computer account of the web server. If the web application pool is running under a custom user account, the SPN should be added under that user account in domainA.local. NOTE: if you are running IIS 7.0 and using kernel mode authentication (the default) then you should add the SPN under the machine account. See Part 6 on new features in IIS 7.0

After adding the SPN, you should see the following in Active Directory:
Add the website www.myCompany.com to the Intranet Security Zone of the user’s computer. Recall from Part 3 that IE will not attempt Kerberos authentication unless the website is in the Intranet Security Zone. This can be done manually, via the IEAK, or using Group Policy.

After this is all configured and replicated around the environment then the following should be observable in the packet capture. Note that this exchange is similar to that seen in the previous packet capture (some stuff is actually missing from this packet capture as the machines already have name resolution and some referals already established. It is worth reviewing Part 8 packet capture with more detailed descriptions if you are seeing this for the first time). The only real difference is that we can see the routing required for http/www.myCompany.com service ticket:

Packet 6 – HTTP request by client
Packet 9 – Initial 401 response from web server
Packet 18 – DomainA.local DC returns service ticket for http/www.myCompany.com to client
Packet 21 – new HTTP request by client including Kerberos ticket
Packets 47-50 – tickets granted to access backend SQL Server
Packet 59 – HTTP 200 response to client with data from backend SQL Server

For reference the machines in question are:

Machine	Domain	IP address	Role
svr03-r2-dc-1	DomainA	192.168.132.10	DC
svr03-r2-dc-2	DomainB	192.168.132.11	DC
svr03-r2-web-1	DomainA	192.168.132.20	Web Server
svr03-r2-sql-1	DomainA	192.168.132.21	SQL Server
cltxp-pro-1	DomainB	192.168.132.50	Client

New IIS 7.0 configuration reference available

Ken — Wed, 21 Jan 2009 06:38:00 GMT

Bill Staples has let the cat out of the bag. IIS 7.0 has a new, comprehensive configuration file reference available at iis.net

Tech.Ed 2008 over

Ken — Sat, 20 Sep 2008 13:05:00 GMT

It's been a long time between blog posts. Between a couple of Tech.Eds, being sick with the flu, and a large enterprise System Center Operations Manager 2007 deployment, it's been pretty busy the past month. In addition, handling bathroom and kitchen renovations is consuming pretty much all of the spare time on weekends.

This year I was privileged to deliver a few presentations at Tech.Ed South East Asia 2008 - one on IIS 7.0 for IT Pros, and a second on Web Farm Scenarios and IIS 7.0. Both of the session decks are available for download from the Tech.Ed SEA website (download the Server track ZIP file).

I also delivered the IIS 7.0 Security and Performance Tuning session at Tech.Ed Australia. My apologies if the session was quite up to scratch - I was suffering from the flu - but it's still the top rated Server track session, so thanks to all that filled in feedback. Click the session title link to download the session deck (1.2MB). Thanks to Wade Hilmo and Pete Harris from the IIS product group for helping me pull that session together.

Some random photos from Tech.Ed 2008:

Tech.Ed SEA 2008 presenter badge

The closing locknote at Tech.Ed Australia 2008. I didn't snap the slide that showed I was beating Steve Riley in the Presenter scores :-)

IIS and Kerberos Part 8 - a simple cross Forest/Domain delegation scenario

Ken — Sun, 29 Jun 2008 10:45:00 GMT

In this part we extend, slightly, upon the previous scenario, by adding delegation. Now we need to allow IIS, in our resource Forest (or domain) to delegate the end user’s credentials, to a backend service (SQL Server in this case):

The machines this case are:

Machine	Domain	IP address	Role
svr03-r2-dc-1	DomainA	192.168.132.10	DC
svr03-r2-dc-2	DomainB	192.168.132.11	DC
svr03-r2-web-1	DomainA	192.168.132.20	Web Server
svr03-r2-sql-1	DomainA	192.168.132.21	SQL Server
cltxp-pro-1	DomainB	192.168.132.50	Client

A packet capture is available for download (taken from the IIS server).

Opening the capture in Wireshark you should see the following (the bullet point numbers correspond to the numbers in the image below):

XP client makes a request to IIS server (Packet 14) and IIS server responds with 401 Access Denied (Packet 17)
XP client contacts DomainB Domain Controller for Kerberos ticket (Packet 19 – note the request for http/svr03-r2-web-1)
DomainB DC returns a referral to DomainA DC (packet 20)
XP client looks up the necessary service records for DomainA (packets 21-24) before requesting a service ticket from the DomainA DC (packet 33)
The DomainA DC returns a service ticket to the XP client (packet 34)
XP client makes a new request to IIS, supplying it’s Kerberos authentication data (packet 37)
IIS contacts its local DomainA DC seeking a referral to DomainB (packets 52-55)
DomainA DC refers IIS to DomainB DC
IIS requests a Kerberos ticket, on behalf of the end user, from DomainB DC (packet 61)
DomainB DC returns the necessary ticket (packet 62)
IIS now connects to SQL Server (packet 65), and gets the results of the query. The resulting webpage is returned to the client (packet 87)

The requirements to configure this scenario aren’t significantly beyond that to configure a basic cross-Forest/cross-Domain scenario featured in the previous part:

A two-way trust is required. This can use Selective Authentication. However Forest-Wide authentication may be administratively simpler to configure
An appropriate SPN needs to be registered for the backend SQL Server (similar to a single domain delegation scenario)

In the next part I will discuss publishing an arbitrary FQDN for the IIS host (e.g. a public facing internet site) and UPN suffix routing.

NOTE (Feb 2009): I finally got around to publishing promised part - see Part 9

Note: A listing of parts is available in the FAQ

IIS and Kerberos Part 7 - A simple cross Forest scenario

Ken — Tue, 13 May 2008 12:13:00 GMT

Note: I have created a list of all the IIS and Kerberos parts

I'm finally getting around to writing this section on IIS and Kerberos. This initial post will cover the basics of a cross-Forest Kerberos authentication scenario. In the next few posts we'll cover more complex situations including delegation and ISA Server publishing.

The basics of cross-domain Kerberos authentication (in the same Forest) are the same as a cross-Forest scenario, so I've covered the cross-Forest scenario in these posts, and steps that are unnecessary for a cross-domain scenario can be omitted.

Our setup involves a resource Forest (domainA.local) and a user Forest (domainB.local). A network packet capture is included (it can be opened using Wireshark/Ethereal - rename the extension back to .cap), and to help decipher the capture the machines involved are:

Machine	Domain	IP address	Role
svr03-r2-dc-1	DomainA	192.168.132.10	DC
svr03-r2-dc-2	DomainB	192.168.132.11	DC
svr03-r2-web-1	DomainA	192.168.132.12	Web Server
cltxp-pro-1	DomainB	192.168.132.50	Client

In the scenario the client in DomainB.local attempts to connect to svr03-r2-web-1 in DomainA.local. The sequence of packets are:

Client connects to web server and gets 401 (Packets 4 and 6)
Client connects to DC in local Domain asking to a ticket to http/svr03-r2-web-1.domainA.local (Packet 8)
The DC in DomainB.local provides a referral to DomainA.local (Packet 9)
The client connects to a DC in DomainA.local asking for a ticket (Packet 12)
The DC in DomainA.local provides a Kerberos ticket to the client (Packet 13)
The client again connects to the web server, presenting its Kerberos ticket (Packet 15)
The server responds with a 200 OK (Packet 21)

And the user successfully authenticates using Kerberos:

Things to be aware of in this simple scenario:

Typically a client will be connecting using the FQDN (fully qualified domain name) of the web server. Since Kerberos is only attempted if the website is in Internet Explorer's Intranet security zone, the website will need to be added to that security zone either using a GPO or manually
Clients must be able to contact domain controllers in the resource Forest in order to get appropriate Kerberos tickets. If there are some DCs in the resource domain that are unreachable (e.g. due to firewalls ec) then you need to ensure that clients in the user Forest only get referrals to reachable DCs
EDIT: Forest trusts can only be created when using a Windows 2003 functional level Forest. The Forest functional level can be raised using the Active Directory Domains and Trusts Admin MMC tool. Before you can raise the Forest functional level, you need to raise the Domain functional level of all Domains within the Forest to Windows Server 2003. If your Forest functional level is Windows 2000, only an external trust can be created, which does not permit Kerberos authentication.
EDIT: Only a one-way trust (the resource Forest trusts the User forest) is required for this scenario. In future scenarios (e.g. when we introduce delegation) a two-way trust will be required. However we can limit the access the Resource forest has to the User forest using Selective Authentication
EDIT: If you need guidance on creating a Forest Trust, then Microsoft's TechNet has a good guide

SCVMM 2008 Beta 1 install fails at the WAIK prerequisite step

Ken — Thu, 08 May 2008 10:50:00 GMT

I was just trying to install SCVMM 2008 Beta 1 today. When installing the SCVMM 2008 Server, it failed installing the WAIK prerequisite asking me to instead install this manually.

Attempting to run the WAIK msi directly from the prerequisites folder (\prerequisites\WAIK\1033) failed asking me to "Insert the WAIK setup CD". I think this problem might be caused because I'm install SCVMM 2008 from a DVD. I copied the files from that folder onto the hard disk of the machine, and then attempted to run the MSI again, and WAIK installed successfully.

MVP Summit 2008

Ken — Tue, 22 Apr 2008 04:20:00 GMT

Last week I was in Seattle attending the Microsoft MVP Summit for 2008. Certainly this year's summit was much better organised than some previous summits in terms of interaction with the IIS product group.

Whilst we've seen a bunch of interesting stuff coming out from the product group over the past few months (WebDAV, MSDeploy, Powershell Provider, Bitrate Throttling, Admin Pack - including the Config Editor). However over the next few months expect to see a number of significant additional releases. Whilst I'm probably not at liberty to disclose what these are, think about the major market that IIS 7.0 has gone after (e.g. hosting with Apache) and some of the major features and modules that the competing platform has that IIS 7.0 doesn't currently, and you'll probably be pretty close to the mark in terms of upcoming features.

In addition to getting the inside scope from the product group, the MVP Summit also offers opportunities to talk and network with other MVPs, as well as an executive briefing. This year Ray Ozzie and Steve Ballmer came by to talk to us. Whilst I've had the opportunity to listen to many of Micosoft's senior executives in other forums (Tech.Eds, Partner events etc), what is refreshing about the MVP Summit is that these executives will spend half an hour (or more) taking questions, without notice, from the floor. Whilst they are naturally guarded about the answers they can give (if press are present), we're still above to canvas a range of topics. And more than once a product has changed somewhat due to the questions or feedback given during these sessions.

I, for one, am looking forward to the next MVP Summit in 2009 (assuming I'm reawarded of course!). As a small bonus, whilst browsing Barnes and Noble in downtown Seattle, I came across a most excellent book that everyone should have a copy of :-)

Potential Critical Security issue in Windows Server 2003/2008 - IIS may be a vector for compromise

Ken — Fri, 18 Apr 2008 06:59:00 GMT

As some of you may be aware, Cesar Cerrudo of Argeniss presented a session at the just completed Hack in a Box conference where exploit code was demonstrated that allows certain code running with restricted privileges (e.g. Network Service) to gain high privileges (e.g. LocalSystem). The exploit appears to rely on the fact that certain other processes running as network service have SeImpersonatePrivilege, and the malicious code can use this to gain additional privileges on the system.

Microsoft has released an advisory on this potential vulnerability, and if you are running IIS 6 or IIS 7, you are urged to examine the potential implications and workarounds posted.

Edit: 19/04/2008 - the slides from Cesar's presentation have been posted on the Argeniss website

Converting from VMWare Server to Hyper-V RC0

Ken — Mon, 24 Mar 2008 06:06:00 GMT

This Easter weekend, having a bit of downtime, I decided to convert my virtual infrastructure at home from VMWare Server to Hyper-V. The major blocking issue was a lack of RAID controller drivers from 3Ware for their 9650SE-series cards, but thanks to Justin Ho it seemed like I was good to go. The timely release of Hyper-V RC0 meant that I could use an updated version of Hyper-V, and also install my Windows Server 2008 machine using my local en-au settings rather than en-us.

The servers that I had running where:

Server1 - Windows Home Server
Server2 - Exchange 2007 (Windows Server 2003 x64)
Server3 - Operations Manager 2007 + WSUS (Windows Server 2003 x86)
Server4 - ISA Server 2006 (Windows Server 2003)
Server5 - Domain Controller 1 (Windows Server 2003)
Server6 - Domain Controller 2 (Windows Server 2003 x64)

To speed up the conversion time, I removed DC2 from the domain (and recreated it as a brand new VM at the end of the process. It is now my first Windows Server 2008 DC). I also removed the Operations Manager 2007 machine (and recreated this on Windows Server 2008)

The steps I used to convert these VMs:

Made a backup of all my virtual machines before I started!
DCPromo DC2, and remove it from the domain
Uninstall Operations Manager clients from all managed servers, then remove Server3 from the domain
Uninstall the VMWare Tools from each remaining virtual machine
Shutdown all remaining machines and make a backup of the VMDK files (again)
Convert the VDMK files to VHD files. You can use System Center Virtual Machine Manager (SCVMM) to do this. Alternatively I used the free VDMKtoVHD tool from VMToolKit. Note that if your VMDK files are pre-allocated fixed sized disks, they will become dynamically expanding VHD disks after the conversion (empty space isn't converted)
Configure my 3Ware 9650SE RAID controller BIOS per Justin Ho's instructions (see earlier)
Format my arrays, and install a brand new copy of Windows Server 2008 x64
Install the Hyper-V RC0 update
Install the Hyper-V role, as well as desired features (Backup and PowerShell)
Create the necessary virtual networks in Hyper-V
Create new virtual machines using the newly converted VHD files and boot the machines

Some issues that I discovered:

My VMWare machines were using SCSI disks connected to a SCSI controller. Unfortunately booting Hyper-V machines requires IDE disks at the moment. Since the IDE mass storage controller wasn’t set to start in my VMs, they Blue Screened with STOP 0x7B (Inaccessible_Boot_Device). I fixed this issue by inserting the OS setup CD and doing a repair on the OS.
EDIT: Steen has a great tip below for how to get around this issue. It requires you to add a dummy IDE disk to your VMWare VM prior to do the conversion (to get the IDE mass storage controller into a started state)
There appears to be an issue with guest OSes talking to a virtualised ISA Server when all the machines are using the new VMBus NICs and the NICs are connected to a Private or Internal Hyper-V network (the issue doesn't appear to manifest if the NICs are bridged to a physical NIC). Networking doesn't work to well, and when running ISA's monitoring tools, packets are missing. To fix this issue, I changed the NICs on my ISA Server that were connected to Private or Internal networks to using the Legacy (Intel 21140) NIC. Since ISA Server 2006 only runs on Windows Server x86, there are supplied Intel 21140 drivers on the Hyper-V Integration Services disc.

So this was the picture beforehand:

and this is the picture aftewards:

Performance appears to be much snappier under Hyper-V compared to VMWare Server, especially with respect to Disk I/O. Additionally, I can now backup my virtual machines when running (well, I hope I can) using my new RD1000 device.