An educated guess (Timur Fanshteyn): Debugging

Showing posts with label Debugging. Show all posts

Wednesday, February 11, 2009

Production Debugging a Memory Leak

I wrote before about not believing in regular system reboots. One of the services we wrote had a serious memory leak and process size grew over 1GB within 2 days requiring us to perform regular service restarts. This is not something that we were able to replicate in development or QA environment so I’ve decided to do some production debugging.

I love reading the blog of Tess Ferrandez on low level .NET Debugging. http://blogs.msdn.com/tess. The has a series walk trough sessions one of them is on Memory Leaks http://blogs.msdn.com/tess/archive/2008/03/25/net-debugging-demos-lab-7-memory-leak.aspx

I can’t really provide the original code for our service, but I was able to replicate the basic leak in a sample app, and below are steps to find out what it is.

Sample (on skydrive.live.com)

LeakyCache.zip-download

Sample Setup: Open LeakyCache.zip Compile it if you want, or just run the included executable. Click “Leak” to leak memory.

LeakyApp

Download Debugging Tools for Windows form Microsoft and install it on the server that is running the problem application.
Copy SOS.DLL from “C:\Windows\Microsoft.NET\Framework\v2.0.50727” to “c:\Program Files\Debugging Tools for Windows (x86)” to get access to debugging library for .NET 2.0
Execute ADScript to take a memory dump of the LeakyCache application
"c:\Program Files\Debugging Tools for Windows (x86)\adplus.vbs" -hang -pn LeakyCache.exe -o c:\temp\LeakDump

Start WinDbg
"c:\Program Files\Debugging Tools for Windows (x86)"\windbg
From the File Menu, select “Open Dump File” and open the created dump file from C:\temp\LeakDump\
Load SOS debugging using command
.load SOS

Now the fun begins :)

Run !dumpheap –stat

What you’ll see is that the most memory is used by data type is System.String (53MB) and Dictionary+Entry (22MB). Also notice that there are more then 1 million string entries. Most of them are very small (<55 bytes average).
To see the entries. Use command (Press CTRL+BREAK to stop the flow) to see the list of addresses.
!dumpheap -type System.String -max 100
!do 022b8978

Substitute the address of one of the items instead of the 022b8978
I underlines a Text String that you can see. In my experience of debugging my apps, based on the data, I can tell what is stored, and probably have some ideas about where that data is generated or should it be cleaned.
Run !gcroot [Reference] to see exactly what class is holding a reference to the object

A walkthrough like this will not necessary solve a problem in the application, but it can point out to a possible issue in the application that can be solved. To me, a memory leak is not a problem that should be ignored, but is a bug that can be fixed.

Note: Huge thanks to Tess for the wonderful blog http://blogs.msdn.com/tess

Technorati Tags: Debugging,Development,WinDbg

Monday, February 09, 2009

Keep release PDBs to help with debugging production problems

Not many developers know that PDB files are generated during release builds are just as helpful as they are in debug builds.

For some background information, PDB Files contain debugging symbols that are used by .NET Debuggers (including Visual Studio) to lookup source code information. If symbols are available, debugger will be able to show not just the function where exception happened, but also the line number in the source file where exception occurred.

Currently, our current build process copies results of every build into a separate output folder, away from the source code. A new step was just added to make a copy of PDBs into a subfolder as well. Here’s a snippet of XML that I’ve added to the .csproj target

    <CreateItem Include="$(TargetDir)\*.pdb">
      <Output TaskParameter="Include" ItemName="PDBFiles" />
    </CreateItem>
    <Copy SourceFiles="@(PDBFiles)" 
            DestinationFolder="$(OutputPath)\PDB" />

One way to use PDB files is to provide them with your application. If PDB file is available at the time of exception, Exception information will include line numbers and source code file name in the exception.

You can also debug release versions of the executable using Visual Studio. From the Tools menu, select “Attach to Process”, Select your executable. After debugging session starts, In Debug menu, Window->Modules, Right click on the module for your executable, and select “Load Symbols From”. Point to your PDB file, and you are done. It will be important to have source code available if you want to step through. That however is a completely different issue.

Technorati Tags: Debugging,Development,Tips

Friday, January 25, 2008

WCFTestClient - a testing utility from Visual Studio 2008

I stumbled upon an excellent utility for WCF Testing that comes with Visual Studio 2008 - WCFTestClient.

The tool is an simple way to test WCF clients HTTP and TCP bindings. Some things are not supported, however, for basic WCF Testing, this definitely beats the old ASMX test page.

Note: Also check out the WCFSvcHost utility from Visual Studio to host an arbitrary WCF Service.

Technorati Tags: Debugging, Development, Microsoft Office, Tips, Tools, WCF, Visual Studio