Troubleshooting with Perfmon and DebugDiag 1.1 for High-Memory W3WP.exe Hangs

in IIS 6.0 and ASP.net 2.0 web applications

DEBUG TRUE CHECK

Check all web.config files to see if Debug is set to True. In production web servers, I highly recommend that debug be set to false—especially if you’re seeing OOM exceptions. This is a common cause (but certainly not the only cause) of OOM exceptions. If Debug is set to True in any web.config file on the afflicted server, I’d recommend setting Debug to False in every web.config file followed by an iisreset at your next available server maintenance window.

Also ask web developers to consider whether dlls are compiled in release mode or debug mode.

To read more on this, please consider this page: ASPNETDebugTrue.

INSTALL DEBUGDIAG

Debugdiag may be downloaded from http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3

(IF possible) DISABLE APP POOL RECYCLING

Disable Health Monitoring with the following steps:

Open the IIS manager and drill down into the Application Pools

Right-click the Application Pool that has been crashing and select ‘Properties’

On the recycling tab, please uncheck all three of the “recycle worker processes” options.

Also uncheck both of the options for “memory recycling” while trying to get the dumps.

They can be added again after we are finished troubleshooting.

On the Performance tab please uncheck "Shutdown worker processes after being idle for"

On the Health tab please uncheck "Enable rapid-fail protection" and “Enable pinging.”

Please ensure that Web Garden is set to only 1.

Click Apply and OK

Create and Start a Performance Monitor capture

Start Button > Run > Open: Perfmon [Enter]

Expand "Performance Logs and Alerts"

Right Click on "Counter Logs"

Choose "New Log Settings..."

Enter a descriptive name (such as “HighMemoryIISHang”)

Note the log file location for later (or go to the "Log Files" tab and change the location)

Click the "Add" button

Click the "All Counters" and "All Instances" radio buttons

Select the following from the "Performance Object" dropdown, being sure to "Add" each one as you select it:

Add every Object that begins with “.NET” (such as, .NET CLR Data, .NET CLR Exceptions, .NET CLR Interop, etc.)

Add every Object that begins with ASP.NET (such as ASP.NET, ASP.NET Applications, etc.)

Memory

Process

Processor

Thread

Web Service

Click "Close"

Click "OK"

Leak Tracking

The ideal situation is to begin leak tracking in debugdiag on a w3wp.exe process a few minutes after it has been recycled. We want to begin leak tracking while the w3wp.exe process has a relatively small memory footprint and track it while it grows. But we don’t want to begin leak tracking while the w3wp.exe is brand new, however.

We also need to know which w3wp.exe to monitor.

You can use iisapp.vbs to know if the problematic webapp has spawned a new w3wp.exe and, if so, what its pid number is.   

Example:

C:\WINDOWS\system32>cscript.exe iisapp.vbs

Microsoft (R) Windows Script Host Version 5.6

Copyright (C) Microsoft Corporation 1996-2001. All rights reserved.

W3WP.exe PID: 4340   AppPoolId: ASPNET20Apps

W3WP.exe PID: 4568   AppPoolId: DefaultAppPool

W3WP.exe PID: 5380   AppPoolId: ProblematicWebApp

W3WP.exe PID: 3510   AppPoolId: ClassicASPapps

W3WP.exe PID: 1011   AppPoolId: ASPNET11Apps

W3WP.exe PID: 4340   AppPoolId: MSSharePointAppPool

Third, open Debugdiag, move to the Processes tab, and find the w3wp.exe which corresponds to your problematic web application.  Right-click this w3wp.exe and choose “Monitor for Leaks” from the menu.

Keep in mind that when using this method you may need to begin monitoring a new w3wp.exe if the server is rebooted, iis gets restarted, or the application pool gets recycled before any memory dumps on the process being monitored for leaks are made. 

Wait for the hang to occur.

Wait for memory to grow to the point where the process begins to become unresponsive.

Manually Triggering a Hang Dump

When the web application(s) begin to exhibit hung performance, here are the steps to create one set of hang dumps of the IIS processes

a) Launch DebugDiag

b) Click Cancel if given the choice of making a Crash Rule or Hang Rule

c) Expand the Tools menu

d) Select “Create IIS/Com+ Hang Dump” and wait for dump creation to occur (may take 30 seconds or more depending on size of the w3wp.exe)

e)  Select “Advanced Data Collection”

  f)  Select “Create Full Cabinet File”

Creating the full cabinet file will compress the event logs, the most recent IIS log, the metabase, the httperr log, all .net config files, and all dmp files together into one convenient .cab file.

IISRESET

After the dmp files are fully written to, feel free to restart IIS to recover temporarily from the hang

It is important to NOT restart any IIS processes before the logs and dmp files are all completely finished being written to as an IISRESET will potentially ruin the dumps.  Please wait for DebugDiag to finish its work before recycling IIS.  This could take anywhere from thirty seconds to perhaps more than three minutes (depending on the size of the iis processes).

Confirm in DebugDiag that the userdump count for the crash rule has increased from Zero to One or more.

STOP PERFMON

Stop the Performance Monitor log

In Performance Monitor:

1. Right click on your log that is now listed under "Counter Logs"

2. Choose "Stop Log"

3. Save it as a .blg file to the location of your choice

After the compression process is finished, the easiest way to find the cabinet file is to click the icon of the manila folder in debugdiag.

Upload the files to the workspace your IIS/ASP.net support engineer created for you.

1. The dumpfiles and event logs should already be compressed into one cabinet file which by default is located at:

C:\Program Files\IIS Resources\DebugDiag\Logs\<Name of Rule>

Please upload whatever .cab file exists at this location.

2. If you chose a non-default location for the dumps to be sent to, you will probably need to zip those dmp files and upload them as well. (The Advanced Data Collection > Create Full Cabinet File only compresses the folders and files found in C:\Program Files\IIS Resources\DebugDiag\Logs\