Troubleshooting High-Memory Dllhost.exe Hangs
in IIS 5.0 and Classic ASP web applications
with Perfmon and DebugDiag 1.1
Overview
Before the web application becomes sluggish
again, we want to:
1. install
DebugDiag,
2. add
some simple test pages to test with,
3. get a
Perfmon capture started,
4. restart
the com+ component corresponding to the dllhost.exe in focus,
5. browse
to an asp page to instantiate its corresponding com+ component,
6. inject
leak trackers with DebugDiag into the dllhost.exe were focusing on,
7. wait
for the next hang,
8. when
the web application becomes hung, we want to trigger two sets of memory dumps
using DebugDiag.
Memory dumps provide a “snapshot” of
what is loaded in memory at a given time. To create a hang dump during a
hang is to get an insider’s picture of everything the threads, heaps, critical
sections, etc. are doing. We want to wait until the web site becomes
sluggish/non-performant e before we trigger a dump of the memory spaces of all
the processes related to IIS (inetinfo.exe, dllhost.exe’s, aspnet_wp.exe).
DebugDiag will write the memory space of each of these processes to a .dmp file
and it is important to not run an IISRESET to recover the site’s functionality
until after the .dmp files have been completely written.
Install DebugDiag
Debugdiag may be downloaded from here:
Note: Accept all defaults during the installation
unless you have very limited freespace on your system partition or prefer not
to install to the system partition. Since
dump files will generally be created in the same path, please be sure to
install this on a drive that has at least one gigabyte of free space. The
default install path is C:\Program Files\IIS Resources\DebugDiag but may be
changed at the beginning of the installation. If free space is limited,
you may get an idea of the minimum size of the .dmp files by opening
taskmanager and adding up the amounts of virtual memory used by inetinfo.exe,
dllhost.exe’s, aspnet_wp.exe, and/or w3wp.exe’s. We will have a .dmp file for
each of these processes that you see in taskmanager.
Test pages
When troubleshooting what seems to be a
sluggish web application, it is important to focus on specific pages to test
before and during the sluggishness. It
is also a good idea to add some simple, dynamic test pages to test with –
perhaps create a test.htm page, a test asp page, and a test aspx page. Please keep them simple (no outbound
database calls unless you make that a fourth test page) and copy them into any
and every web application that you’re trying to troubleshoot. At the time of the hang—either just before
triggering the memory dumps or perhaps after the memory dumps but before an
iisreset—you’ll want to test all the test pages to see more precisely what is
and what is not hung.
Examples…
Save the following into a text file and
rename to test.aspx.
It doesn’t matter whether 1.0, 1.1, 2.0 versions or vb.net or c#--works on all
of them. It should show the date and time on the server.
<%=System.DateTime.Now%>
Save the following into a text file and
rename to test.asp.
It should show the date and time on the server.
<%=now()%>
If your asp
pages talk to backend databases, also consider having your developers create a
test-database.asp page which can be used to begin to gauge the response from
the backend (whether a SQL database, Oracle Database, LDAP directory, etc.) and
other possible data-access issues which might resemble a hang. If a simple asp
page is served without delay while an complex asp page which connects to a
backend database seems to show the hang, this is important to know as it may
begin to re-focus the troubleshooting towards a data-access issue or backend
database blocking or such rather than a bottleneck inside of IIS.
If any of
the test pages or any of the main pages of your web application give an error
message of some kind, please either screenshot the error message for us and/or
copy all the text from the browser into an email. Generally speaking it’s a good idea in
advance to uncheck “show friendly http error messages” from the IE client’s
Advanced properties (Tools menu > Internet Options > Advanced tab…).
Create and Start a Performance Monitor capture
Start Button > Run > Open: Perfmon [Enter]
Expand "Performance Logs and Alerts"
Right Click on "Counter Logs"
Choose "New Log Settings..."
Enter a descriptive name (such as “Hang”)
Note the log file location for later (or go to the "Log Files"
tab and change the location)
Click the "Add" button
Click the "All Counters" and "All Instances" radio
buttons
Select the following from the "Performance Object" dropdown,
being sure to "Add" each one as you select it:
Add every Object that begins with “.NET”
(such as, .NET CLR Data, .NET CLR Exceptions, .NET CLR Interop, etc.) with All counters
selected
Add every Object that begins with ASP.NET (such as ASP.NET, ASP.NET
Applications, etc.) with All counters selected
Add the object “Active Server Pages” with All counters selected
Add the object “Memory” with All counters selected
Add the object “Process” with All counters selected
Add the object “Processor” with All counters selected
Add the object “Thread” with All counters selected
Add the object “Web Service” with All counters selected
Click "Close"
Click "OK"
(Preliminary – before the hang occurs)
When you’re ready to begin leak tracking please Recycle the Com+ package
which corresponds to the problematic dllhost.exe
You can either do this with an IISRESET…
Or it can be done through the Com+ console (Start > Run > Open:
Dcomcnfg [Enter]) by starting and stopping the com+ package in focus
We want to do this to start with a fresh worker process before we begin
monitoring that worker process for leaks.
Instantiate
the Com+ Component
You may need to actually browse to an
asp page in the web application to spin the dllhost.exe up.
Inject
Leak Trackers
Debugdiag, move to the Processes tab,
and find the dllhost.exe which corresponds to your problematic web
application. Right-click this w3wp.exe and choose “Monitor for Leaks”
from the menu.
Keep in mind that when using this
method you may need to begin monitoring a new dllhost.exe if the server is
rebooted or IIS is restarted
Wait for the hang to occur.
Manually Triggering a Hang Dump
When the web application(s) begin to
exhibit hung performance, here are the
steps to create two sets of hang dumps of the IIS processes
a) Launch
DebugDiag
b) Click
Cancel if given the choice of making a Crash Rule or Hang Rule
c)
Expand the Tools menu
d)
Select “Create
IIS/Com+ Hang Dump” and wait for dump creation to occur (may take 30
seconds or more depending on size of the w3wp.exe)
e)
Wait for three or four minutes after the first set of hang dumps is
finished
Use this time to test the
test.htm, test.asp, test.aspx pages and other pages. Note the behavior of each page. Note any
error messages.
f)
Trigger a second set of hang dumps with the “Create IIS/Com+ Hang Dump” again
Note:
If for some reason you cannot respond in time to manually trigger hang
dumps, you’ll need to set up a hang rule.
IISRESET
After the dmp files are fully written
to, feel free to restart IIS or recycle your Com+ component to recover
temporarily from the hang.
It
is important to NOT restart any IIS processes before the logs and dmp files are
all completely finished being written to as an IISRESET will potentially ruin
the dumps. Please wait for DebugDiag to finish its work before recycling
IIS. This could take anywhere from thirty seconds to perhaps more than
three minutes (depending on the size of the iis processes).
STOP PERFMON
Stop the Performance Monitor log
In Performance Monitor:
1. Right click on your log that is now
listed under "Counter Logs"
2. Choose "Stop Log"
3. Save it as a .blg file to the
location of your choice
COLLECT
DATA
From the Tools menu, select “Advanced
Data Collection”
Select “Create Full Cabinet File”
Creating the full cabinet file will
compress the event logs, the most recent IIS log, and all dmp files together
into one convenient .cab file.
After
the compression process is finished, the easiest way to find the cabinet file
is to click the icon of the manila folder in debugdiag.
Upload the Data
Zip and upload the cab file (which contains
all .dmp files, iis log, and event logs) to the workspace or ftp site your
support engineer has created for you.
Also stop, save, zip and upload the
Perfmon .blg file
The dump files and
event logs should already be compressed into one cabinet file which by default
is located at:
C:\Program
Files\IIS Resources\DebugDiag\Logs\<Name of Rule>
Please
upload whatever .cab file exists at this location.
If you chose a
non-default location for the dumps to be sent to, you will probably need to zip
those dmp files and upload them as well.
(The Advanced Data Collection > Create Full Cabinet File only
compresses the folders and files found in C:\Program Files\IIS
Resources\DebugDiag\Logs\