Ken Schaefer

Seen on Scott Guthrie's blog, is an announcement for an upcoming release of IIS Express.

IIS Express offers the functionality of IIS 7.0, even back on Windows XP, running as a user mode application (i.e. admin rights not required to run this  ad hoc web server). Simply right-click on a folder, and you can start an instance of IIS running from that folder. It appears that most of the major functionality of IIS 7.0 (including the modular architecture) is included. Interesting times.

More info: http://isc.sans.edu/diary.html?storyid=8935

Verify your IIS website, and if you are maintaining the server, verify you're up-to-date with patches.

You can get this error for a variety of reasons (the associated error message gives a few possible reasons and things to check). In my case it appears that attempting to install SCVMM 2008 R2 on a server that has dashes (-) in the host name causes the installation to fail with this error. Renaming the host so that it no longer includes dashes in the hostname (SVR08-SCVMM-1 to SVR08SCVMM1) allows the installation to complete.

Other things I tested (to no avail):

• Disable all NICs on the machine except the one used to connect to the DC
• Rebuild the machine completely (three times)
• Use a different domain (in fact, I built an entirely new Windows Server 2008 DC)
• Disabled IPv6
• Used a known good SQL Server, rather than installing a local copy of SQL Server (in case it was something to do with the SQL Server SPNs)
• Used different domain accounts as well as LocalSystem

Symptoms of the problem are the following error message:

Virtual Machine Manager cannot process the request because an error occurred while authenticating SVR08-SCVMM-1.domainB.local. Possible causes are:
1) The specified user name or password are not valid.
2) The Service Principal Name (SPN) for the remote computer name and port does not exist.
3) The client and remote computers are in different domains and there is not a two-way full trust between the two domains. Log in by using an account on the same domain as the VMM server, or by using an account on a domain that has a two-way full trust with the domain of the VMM server, and then try the operation again. If this does not work, purge the Kerberos tickets on the VMM server by using kerbtray.exe, available at http://go.microsoft.com/fwlink/?LinkId=93709. Then, reset the SPN for SVR08-SCVMM-1.domainB.local by using setspn.exe available from http://go.microsoft.com/fwlink/?LinkId=93710. If this still does not fix the problem, make SVR08-SCVMM-1.domainB.local a member of a workgroup instead of a domain, restart the computer, rejoin the domain, and then try the operation again.
ID: 2917. Details: The network path was not found (0x80070035)

In the detailed installation log the error occurs right after a Copy a File step.

Disclaimer: I don't claim to be an expert in Data Protection Manager 2007! However after recent experiences, I believe that the following items should be considered when evaluating DPM 2007 SP1 for protecting your Sharepoint/MOSS 2007 farm. Note: DPM 2007 SP1 is required to protect backend SQL Server clusters. Don't even consider DPM 2007 without SP1.

The diagram outlines a typical minimum configuration for a highly available MOSS 2007 infrastructure, with DPM 2007 included. An organisation is free to add additional servers for further redundancy as required:

DPM 2007 SP1 backs up your MOSS 2007 farm in two ways:

• The databases are backed up directly from SQL Server using the SQL Server VSS writer
• A DPM agent installed on a MOSS WFE generates the heirachy of site collections, sites, items etc

When restoring an item, site, collection etc the following process is used (restoring an entire database or farm can be done directly to SQL Server):

• The relevant content database is copied to the standby/restoration MOSS server
• MOSS APIs are used to extract the relevant information and saved to a MOSS backup file
• This file is copied to the WFE that hosts the DPM agent, and is imported into your Production MOSS farm

When deploying DPM 2007 SP1 to protect your MOSS 2007 farm, the following needs to be considered:

1. The DPM agent can only be installed on a single WFE in the farm. Even if you build a highly available MOSS farm with multiple web front ends, middle tier servers and a backend SQL Server cluster, your backups may fail because the single WFE with the agent installed is unavailable. You can transfer the agent between WFEs, but this is a process you need to manage yourself (whether manual or automated by scripts)
2. The account that the DPM agent runs under needs to be both a SharePoint 2007 farm admin account, and a local Windows Administrator account on the WFE. Whilst the SharePoint minimum security configuration doesn't require the Farm Admin account to be a Windows Administrator account, you need to promote that account on the relevant WFE. If you wish to easily transition the WFE that the DPM agent is installed on between WFEs, you need to make the Farm Admin account a Windows Administrator account on all WFEs
3. To restore anything less granular than a single content database, DPM will copy the entire relevant content database to the restoration MOSS server. This can involve significant amounts of time (e.g. you have a 100GB content DB, and you wish to restore a single document, DPM will still copy the entire 100GB content DB). This content DB is then mounted on the temporary server (ensure you have enough disk space on this server to cater for your largest content DB) and the relevant item(s) extracted. Maintaining this restoration SharePoint server consumes additional hardware, Windows OS license and SharePoint licence.
4. When restoring any items to the Production SharePoint infrastructure, other than the original location, the Site Collection template of the source (i.e. what's being restored) and the destination (your temporary location) must match. The site template itself doesn't matter. This is not a requirement when using SharePoint's native backup/restore tools, but some requirement of the way DPM interacts with SharePoint. Unfortunately there is no way, from within the DPM console, to tell what the relevant site collection templates are
5. If DPM is performing a backup, you can’t do a restore without cancelling the in progress backup. However in my experience the actual backup can take a long time (in the order of many hours) if the WFE that you have your DPM agent on is busy (e.g. participating in crawling content). DPM backups will also fail if the SQL Server is heavily loaded (e.g. backups of SQL Server or other maintenance operations are in progress). This can restrict what SLAs you are able to offer to your end users.
6. Adding and removing content databases makes DPM unhappy. Adding a new content database merely requires a consistency check job to run. If you remove a content database this requires an entirely new base replica to be created. This can quickly blow out your storage requirements if you are in an environment (e.g. Dev/Test) when content databases are often added/removed
7. DPM backs up in a couple of ways – it backs up the databases directly from the SQL Server via VSS, and then gets a catalogue of restorable items from the WFE. If the latter fails, half the time you don’t seem to get a decent warning about it. Instead, when you try to restore you find out that you can only restore an entire database. When you attempt to drill down to content, you simply can't (double-clicking on a content database in the console doesn't result in an error - simply nothing happens)
8.  Installing DPM relies on a bunch of hotfixes and other stuff to be installed to get it working properly. There’s even a DPM hotfix you need to install if you install MOSS Feb 2009 CU, because somehow that MOSS CU stops DPM discovering your MOSS installation as a protectable item. VSS is another thing that seems to require continual patching.
9. The SQL Server VSS writer runs as LocalSystem on the SQL Server. This means that LocalSystem requires permissions to the Master and MSDB databases (DataReader seems sufficient) on every instance on every node on the backend SQL Server cluster. If you are have configured your SQL Server cluster to use separate process identities and removed LocalSystem from being able to login (e.g. to prevent Windows administrators from accessing SQL Server), then you'll need to add LocalSystem back in as a permitted login in SQL Server. The VSS writer needs to be able to find out where all the databases are located physically in the file system, and whether they are being mirrored or not, and to do that it needs to query Master and MSDB

Overall, in a smaller environment, DPM 2007 would be simple and easy to use. The console is quite intuitive (even if the setup requirements are a bit obtuse). In a larger enterprise enviroment, I would hesitate to recommend DPM 2007. There are simply too many limitations in the backup and restore process, that end up hobbling the product.

It seems I must have snuck back in by the skin of my teeth:

Dear Ken Schaefer,

Congratulations! We are pleased to present you with the 2009 Microsoft® MVP Award! This award is given to exceptional technical community leaders who actively share their high quality, real world expertise with others. We appreciate your outstanding contributions in Internet Information Services technical communities during the past year.

The Microsoft MVP Award provides us the unique opportunity to celebrate and honor your significant contributions and say "Thank you for your technical leadership."

Toby Richards
General Manager
Community & Online Support

Kingcope has published an exploit to the Bugtraq mailing list for IIS FTP service running on IIS5, IIS6 and IIS7 (when running FTP v6). Note that IIS 7 running FTP v7 and IIS 7.5 are not affected.

Microsoft has an official advisory, and some more details are available on the Secunia blog. My fellow MVPs are not reporting 100% results against every version of IIS FTP, but everyone is advised to follow work arounds on the Microsoft website, and keep an eye out for developments.

Windows 7 and Windows Server 2008 R2 have RTMed. Some new features include (of course) IIS 7.5, new Active Directory functionality (recycle bin), the Live Migration for Hyper-V virtual machines, and direct access/branch cache (for clients)

Yay - about time. We can now avoid the situation where we have to load multiple copies of Outlook to access multiple Exchange profiles.

I haven't worked out a way to rename the actual mailboxes (that actually works). For those stuck on an older version of Outlook, the ExtraOutlook tool is still available.

After a recent SharePoint 2007 migration exercise at a large customer, we started experiencing performance issues reported by end users when the system was under load. In the Windows Event Logs we saw the following event:

Event Type: Warning
Event Source: W3SVC-WP
Event ID: 2262
Description:
ISAPI 'C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

IIS 6.0 would shortly recycle the web application pool, but during that period users directed to that WFE server by the load balancer weren't able to have their requests served. After looking through obvious causes, we ended up obtaining a dump file of the worker process. We used the process of orphaning a worker process but a dump file can just as easily be obtained using common tools such as IISState or IIS DebugDiag.

Once the dump file was obtained I performed the following analysis.

Start WinDBG and point it the dump file. WinDBG is part of the Debugging Tools for Windows

Since this is a SharePoint issue, I suspected that we may need to investigate both managed and native code. To assist with debugging managed code we need to do a bit of extra preparatory work:

• Load the SOS (Son of Strike) extension: .load SOS
• Make available a copy of mscordacwks.dll to WinDBG. Note that this has to match the version on the server where the dump file was taken from. It's easiest to just copy this from the server (from the %systemroot%\Microsoft.Net\Framework\<frameworkVersion> folder. It will have to be renamed to mscordacwks_xxx_xxx_2.0.50727.yyyy.dll - where XXX is either I386 or AMD64 depending on the platform, and yyyy is the version number (this can be found by right-clicking on the .dll file and checking the version number). WinDBG will tell you what the actual filename you need is if you get it wrong. You can just copy-n-paste that value to assist in renaming the file
• To make available this file copy it into a folder of your choosing and add it to the WinDBG symbols path. I copied it into c:\temp, and to add that to the path type: .sympath+ c:\temp
• Add the Microsoft public symbol server to the path (if it isn't already). To do this type: .symfix
• Now acquire symbols for Microsoft modules in the dump by typing in: .reload

Now we can begin analysis of the dump file. The first command gives us an indication of how long threads have been running in the process:

0:031> !runaway
 User Mode Time
  Thread       Time
  13:1218      0 days 0:00:56.437
  37:138c      0 days 0:00:32.718
  31:b80       0 days 0:00:32.406
  43:1208      0 days 0:00:30.125
  33:17ec      0 days 0:00:23.578
  38:fa8       0 days 0:00:14.937
  36:147c      0 days 0:00:13.609
  40:1680      0 days 0:00:11.468
  15:153c      0 days 0:00:08.359
  44:15a4      0 days 0:00:07.968
  16:12e4      0 days 0:00:06.968
  27:1554      0 days 0:00:05.750
  25:16b0      0 days 0:00:05.656
  24:15f8      0 days 0:00:05.453
  35:17a8      0 days 0:00:05.406
  26:d98       0 days 0:00:05.359
  41:cb0       0 days 0:00:05.296
  11:d50       0 days 0:00:05.031

We can see here that we have quite a few very long running threads (there were 71 threads in total in this dump file - I have truncated the list).

The next step is to have a look at the longest running threads and see what they are doing. To change to a thread type: ~<threadnumber> s. In this case we change to thread 13 (the longest running thread) and then dump the unmanaged stack:

0:013> ~13 s
ntdll!NtWaitForSingleObject+0xa:
00000000`77ef0a3a c3              ret
0:013> k
Child-SP          RetAddr           Call Site
00000000`0309aa68 000007ff`771d5280 ntdll!NtWaitForSingleObject+0xa
00000000`0309aa70 000007ff`7731173e mswsock!WSPRecv+0x66b
00000000`0309abb0 000007ff`770f3518 ws2_32!WSARecv+0x166
00000000`0309ac60 000007ff`72499c0e wsock32!recv+0x38
00000000`0309acc0 000007ff`5fe51ae7 dbnetlib!ConnectionRead+0x4fe
00000000`0309ada0 000007ff`5fe52f2d sqloledb!CDataSource::ConnectionTransact+0xf7
00000000`0309ae20 000007ff`5feaac34 sqloledb!CDBConnection::SendPacket+0x25d
00000000`0309aee0 000007ff`5fe44b69 sqloledb!CStmt::SQLExecRPC+0x4d4
00000000`0309aff0 000007ff`5fe45df3 sqloledb!CCommand::ExecuteHelper+0x2a9
00000000`0309b090 000007ff`5fe46cc7 sqloledb!CCommand::Execute+0xc73
00000000`0309b180 000007ff`5aa0e4cc sqloledb!CImpICommandText::Execute+0x187
00000000`0309b200 000007ff`5aa0f65b oledb32!CCommandText::DoExecute+0x4fc
00000000`0309b440 00000000`0a913a72 oledb32!CCommandText::Execute+0x8ab
00000000`0309b6f0 00000000`0a84241f STSWEL!Voledb::ExecQuery+0x37e
00000000`0309b850 00000000`0a8846bd STSWEL!VdocumentStore::httpGetDocument+0xbdf
00000000`0309c850 00000000`0a886720 STSWEL!VhttpManager::loadFileCore+0x5c5
00000000`0309d0f0 00000000`3569fa9b STSWEL!VhttpManager::loadFileAndMetaInfo+0xc4
00000000`0309d1b0 00000000`356aaa8c OWSSVR!GetExtensionVersion+0x890af
00000000`0309d570 00000642`7f600887 OWSSVR!GetExtensionVersion+0x940a0
00000000`0309d8f0 00000642`806070ff mscorwks!DoCLRToCOMCall+0x177

Note: to dump the managed stack type !clrstack (this requires SOS extension to be loaded). I did dump the managed stack but it isn't relevant in this case. The oldest item on the native stack is a call from managed code into native code (mscorwks!DoCLRToCOMCall) and examining the managed stack doesn't tell us anything.

From the native stack we can see a few things (read a stack from the bottom up):

• We have some SharePoint related components (OWSSVR and STSWEL) that are called.
• Eventually these call into OleDb
• OleDb then appears to call into the SQL Server OleDb Provider (sqloledb)
• The SQL Server OleDb Provider then calls into WinSock to send a command across the network to a remote SQL Server
• The stack ends with the Windows Socket in a receive state awaiting a response (ws2_32!WSARecv)

I examined most of the threads running in the process, and a vast majority of ASP.NET worker threads where in a similar state. Our next step is to try to figure out what SQL command or stored procedure we are calling.

To do this, I took a bit of a guess (I suppose I could have look at the definitions of the OleDb APIs etc). I suspected that a function such as oledb32!CCommandText::Execute would probably have, as one of it's parameters, the actual command to be executed (mirroring the OleDb connection and command objects' .Execute method)

Using the kb command we can get parameter information. The address of the third parameter was 00000000`0309bc40 (note this dump is from an x64 system).

Our next step is to examine some memory around that parameter address. It turns out that approximately 240 bytes further we can find the stored procedure being called:

0:013> dc 00000000`0309bc40+0x240
00000000`0309be80  0ef44048 00000000 110a0008 00000000  H@..............
00000000`0309be90  0f04afe8 00000000 00000000 00000000  ................
00000000`0309bea0  0309bea8 00000000 003f007b 0063003d  ........{.?.=.c.
00000000`0309beb0  006c0061 0020006c 00720070 0063006f  a.l.l. .p.r.o.c.
00000000`0309bec0  0046005f 00740065 00680063 006f0044  _.F.e.t.c.h.D.o.
00000000`0309bed0  00460063 0072006f 00740048 00700074  c.F.o.r.H.t.t.p.
00000000`0309bee0  00650047 00280074 002c003f 002c003f  G.e.t.(.?.,.?.,.
00000000`0309bef0  002c003f 002c003f 002c003f 002c003f  ?.,.?.,.?.,.?.,.

Looking through the other threads, the majority are calling this stored procedure, with a few calling a different one. it maybe that this one sproc is blocking itself, or the interaction of these two are blocking each other. Or possible some other issue in the cluster. For now, our job is done, and we hand over to the DBA team to do some investigation into what is happening in SQL Server.

Note: Due to blog spam, comments are disabled. Please use the contact form for questions.

Windows Home Server automatically ignored folders called "temp" when configuring backups. Normally this isn't a problem. However IIS 7.0 does create a folder called temp (by default at c:\inetpub\temp). In this location are stored application pool configuration files that are generated on-the-fly by IIS when a web application pool is started.

It appears that the WHS bare-metal restore doesn't restore this folder (in addition to any other folders named "temp" on the system), and IIS 7.0 can then can experience issues. The solution may be as simple as creating a folder called "temp" for IIS to store app pool config files in.

Note: comments are disabled due to blog spam. Please use the contact form.

My work recently gave me a new Latitude E6400 (the E6500 was just a bit too heavy) which is a great machine for running virtual machines out of the office. It has an internal 7200 RPM drive, a modular drive (120GB 1.8" 5400 RPM) and I can connect an external 2.5" 7200 RPM drive via the eSata port. It also supports up to 8GB of RAM, and has a C2D 2.8GHz CPU. Considering where we were only 4-5 years ago with laptops, it's an amazing advance.

I installed Windows Server 2008 R2 Build 7000 (Beta 1) and the Hyper-V v2 role and almost immediately started getting STOP 0x00000101 BSODs (CLOCK_WATCHDOG_TIMEOUT). Ben Armstrong reports this is a known issue in Beta 1. Mike Kolitz suggested, based on dump files, that this might have been fixed in build 7006. Luckily my work's TAP program has access to various interim builds. I pulled down Build 7068, and no more BSODs. Unfortunately the Intel WLAN drivers seem to crash my WAP. One step forward...one step back :-)

After adding extra disks to the home "data centre" and then getting a new backup device (Dell RD1000), today I took a punt on upgrading the RAM. When I originally bought my Dell PE SC1430 it was rated at a maximum of 8GB RAM. The latest models offer a maximum of 16GB of RAM, and after finding nothing on the interwebs that might indicate that my particular model couldn't cope with more than 8GB, I tentatively splashed out on an extra 2x4GB FBDIMMS. It seems all PE SC1430s (with the latest BIOS) support more than 8GB of RAM (well, at least 12GB, and I suppose 16GB if you populate 4x4GB).

Currently my production VMs consume around 7GB of RAM (Exchange and Ops Manager 2007 around 2GB each, plus Windows Home Server takes another 1GB), so testing new OSes (like Windows Server 2008 R2) is difficult. With the extra unallocated RAM, it'll be easier to test beta server OSes :-)

Earlier this month I was lucky enough to attend the 2009 Microsoft MVP summit in Seattle. The bulk of the summit consisted of two days of sessions with our product teams (I popped across to some Directory Services sessions as well), and a one day executive keynote session.

Some heavy hitters turned up for the executive keynote - Steve Ballmer was good value as always

Soma managed to rope in four of Microsoft's technical fellows - some of the heaviest technical hitters in the company for a Q&A around Microsoft's future developer direction. It was a pity that so much of the Q&A time for this session was wasted with questions and general complaining that wasn't relevant to any of the people on stage.

For two days I was out at Redmond - building 42 - where the IIS team is based:

Conference Room 2200 was where our sessions were held:

Tomorrow I'll be writing up a follow up post on topics covered.

On an unrelated note I've also become a bit addicted to FlightMemory - a site where you can record flights taken. Inputting the flights that I still have records for, I've flown 274 flights totalling around 588,000 kms in the last 6 or so years. The site generates nice maps as well: