Troubleshooting Advanced Crashes in IIS 6.0 Processes with DebugDiag 1.1

Summary:

Ideally, we want to set DebugDiag up so that it will monitor all the IIS processes (inetinfo.exe, all w3wp.exe’s, all dllhost.exe’s) for a crash/access violation/second-chance exception and, when detected, will dump everything that is going on in the memory spaces at the time of the crash into .dmp files. We can then perform post-mortem analysis on the .dmp files to determine the cause of the crash.

KB References:

931370 The Debug Diagnostic 1.1 tool is now available

http://support.microsoft.com/default.aspx?scid=kb;EN-US;931370

919789 How to use the Debug Diagnostics tool to troubleshoot an IIS process that stops unexpectedly

http://support.microsoft.com/default.aspx?scid=kb;EN-US;919789

921464 How to use the Debug Diagnostics 1.0 tool to analyze a memory dump file

http://support.microsoft.com/default.aspx?scid=kb;EN-US;921464

Troubleshooting Steps:

1. INSTALL DEBUGDIAG

Please download and install Debug Diagnostic Tool v1.1 from http://www.microsoft.com/downloads/results.aspx?pocId=&freetext=debugdiag&DisplayLang=en

When you launch debugdiag.msi it will give you an option to change the default installation path from the C:\drive to another drive.

Since dump files can be large, it may be advisable to install the tool to another drive with the most free space.

2. LAUNCH DEBUGDIAG

Launch DebugDiag (Start Button > Programs > Debug Diagnostics Tool 1.1)

3. DISABLE APP POOL RECYCLING

Disable Health Monitoring with the following steps:

Open the IIS manager and drill down into the Application Pools

Right-click the Application Pool that has been crashing and select ‘Properties’

On the recycling tab, please uncheck all three of the “recycle worker processes” options.

Also uncheck both of the options for “memory recycling” while trying to get the dumps.

They can be added again after we are finished troubleshooting.

On the Performance tab please uncheck "Shutdown worker processes after being idle for"

On the Health tab please uncheck "Enable rapid-fail protection" and “Enable pinging”

Click Apply and OK

4. SET UP A CRASH RULE

When asked which rule type to choose, choose “Crash” and click NEXT

For select target type you can choose either “All IIS/Com+ related processes” or “A Specific IIS Web Application Pool”

From the Advanced Configuration Window, choose “BREAKPOINTS”

And click the Add Breakpoint button. . .

Select Kernel32!ExitProcess from the list of breakpoints

And

Change the action type to “Full User Dump”

Click OK

Select Kernel32!TerminateProcess from the list of breakpoints

And

Change the action type to “Full User Dump”

Click OK

Click the “Save & Close” button. . .

Select Next. . .

Activate the rule when you’re ready for the tool to start monitoring the IIS processes for a crash. . .

Note how the status is set to active and the userdump count is set to 0. The userdump count should increase when a crash is detected.

4. WAIT

Feel free to log off the server while waiting for the crash to occur. Since debugdiag runs as a service, you do not have to be logged into the machine.

5. COMPRESS

After you have confirmed that the userdump count has increased from 0 to 1, please expand the Tools menu in Debugdiag, select “Advanced Data Collection,” and select “Create Full Cabinet File”

Creating the full cabinet file will compress the event logs, the most recent IIS log, the metabase, the httperr log, all .net config files, and all dmp files together into one convenient .cab file.

6. UPLOAD

After the compression process is finished, please find and upload the .cab file.

The easiest way to find the cabinet file is to click the icon of the manila folder in debugdiag (as seen in the screenshot below). This will take you to the default dump location of: C:\Program Files\IIS Resources\DebugDiag\Logs\<Name of Rule>. If you chose a non-default location for the dumps to be sent to, you will probably need to zip those dmp files and upload them as well. (The Advanced Data Collection > Create Full Cabinet File only compresses the folders and files found in C:\Program Files\IIS Resources\DebugDiag\Logs\

Please upload the .cab file to the following secure workspace:

Upload the files to the secure workspace or ftp site provided to you by your IIS/ASP.net support engineer.