On my current project, the customer is implementing a PKI (Public Key Infrastructure) as part of the platform migration. Whilst we had verified our design on x86 (32 bit), we thought we should also verify our design on x64 (64 bit Windows) just in case there were any gotchas.

After installing Certificate Services on our root and subordinate CAs (Certificate Authorities), including the web enrollment interface, we needed to configure our certificate chaining heirarchy.

Imagine our surprise when bringing up the /certsrv virtual directory resulted in a "File Not Found" error. Obviously there were a few things we needed to check first. Working off the IIS request processing pipeline, we can see that we need to ensure that we don't have any ISAPI filters (like URLScan) blocking the requests, and that the ASP web service extension needs to be enabled. That all checked out OK.

What was more surprising was the lack of a HTTP substatus code. In the log files we were seeing 404 0 3:
HTTP Status = 404
HTTP Substatus = 0
Win32 Status = 3 (The system can not find the file specified)

Certificate Services locates its web enrollment pages in %systemroot%\system32\certsrv. Doing a little experimenting revealed that no other files anywhere in the system32 folder were accessible, however moving the Certificate Services files outside the system32 folder (and editing the #include files to remove the path checking code) allowed web enrollment to work.

That behaviour allowed us to quickly ascertain that we must have been running 32 bit worker processes (w3wp.exe). 32 bit applications on x64 Windows have access to system32 rerouted transparently to a special syswow64 folder. This would would explain why the system was telling us that it couldn't locate the Certificate Services files, because they aren't located in the syswow64 folder, but underneath the system32 folder.

However by default, IIS does not run 32 bit worker processes, and you need to set a particular metabase property: Enable32BitAppOnWin64 to allow IIS to create 32 bit worker processes.

So what, on these boxes, had enabled this property? It turned out that VMWare Server had been installed onto our test boxes. VMWare Server, even though it can host 64bit guests, is still a 32bit application. And because VMWare Server has a web interface, the VMWare Server installation must have configured IIS to allow its web interface to run.

By setting Enable32BitAppOnWin64 back to false, we were able to bring up our Certificate Services web enrollment interface. However at the same time, VMWare Server's web interface stopped working. But at least we could finish the validation of our design.