Production Debugging a Memory Leak

I wrote before about not believing in regular system reboots. One of the services we wrote had a serious memory leak and process size grew over 1GB within 2 days requiring us to perform regular service restarts. This is not something that we were able to replicate in development or QA environment so I’ve decided to do some production debugging.

I love reading the blog of Tess Ferrandez on low level .NET Debugging. http://blogs.msdn.com/tess. The has a series walk trough sessions one of them is on Memory Leaks http://blogs.msdn.com/tess/archive/2008/03/25/net-debugging-demos-lab-7-memory-leak.aspx

I can’t really provide the original code for our service, but I was able to replicate the basic leak in a sample app, and below are steps to find out what it is.

Sample (on skydrive.live.com)

LeakyCache.zip-download

---

Sample Setup: Open LeakyCache.zip  Compile it if you want, or just run the included executable. Click “Leak” to leak memory.

1. Download Debugging Tools for Windows form Microsoft and install it on the server that is running the problem application.
2. Copy SOS.DLL from “C:\Windows\Microsoft.NET\Framework\v2.0.50727” to “c:\Program Files\Debugging Tools for Windows (x86)” to get access to debugging library for .NET 2.0
3. Execute ADScript to take a memory dump of the LeakyCache application
   "c:\Program Files\Debugging Tools for Windows (x86)\adplus.vbs" -hang -pn LeakyCache.exe -o c:\temp\LeakDump

1. Start WinDbg
   "c:\Program Files\Debugging Tools for Windows (x86)"\windbg
2. From the File Menu, select “Open Dump File” and open the created dump file from C:\temp\LeakDump\
3. Load SOS debugging using command
   .load SOS

Now the fun begins :)

1. Run !dumpheap –stat
    
   What you’ll see is that the most memory is used by data type is System.String (53MB) and Dictionary+Entry (22MB). Also notice that there are more then 1 million string entries. Most of them are very small (<55 bytes average).
2. To see the entries. Use command  (Press CTRL+BREAK to stop the flow) to see the list of addresses.
   !dumpheap -type System.String -max 100
   !do 022b8978 
   
   Substitute the address of one of the items instead of the 022b8978
   I underlines a Text String that you can see. In my experience of debugging my apps, based on the data, I can tell what is stored, and probably have some ideas about where that data is generated or should it be cleaned.
3. Run !gcroot [Reference] to see exactly what class is holding a reference to the object
   
   

A walkthrough like this will not necessary solve a problem in the application, but it can point out to a possible issue in the application that can be solved. To me, a memory leak is not a problem that should be ignored, but is a bug that can be fixed.

Note: Huge thanks to Tess for the wonderful blog http://blogs.msdn.com/tess

Technorati Tags: Debugging,Development,WinDbg

Production Debugging a Memory Leak

Keep release PDBs to help with debugging production problems

Not many developers know that PDB files are generated during release builds are just as helpful as they are in debug builds.

For some background information, PDB Files contain debugging symbols that are used by .NET Debuggers (including Visual Studio) to lookup source code information. If symbols are available, debugger will be able to show not just the function where exception happened, but also the line number in the source file where exception occurred.

Currently, our current build process copies results of every build into a separate output folder, away from the source code. A new step was just added to make a copy of PDBs into a subfolder as well. Here’s a snippet of XML that I’ve added to the .csproj target

    <CreateItem Include="$(TargetDir)\*.pdb">
      <Output TaskParameter="Include" ItemName="PDBFiles" />
    </CreateItem>
    <Copy SourceFiles="@(PDBFiles)" 
            DestinationFolder="$(OutputPath)\PDB" />

One way to use PDB files is to provide them with your application. If PDB file is available at the time of exception, Exception information will include line numbers and source code file name in the exception.

You can also debug release versions of the executable using Visual Studio. From the Tools menu, select “Attach to Process”, Select your executable. After debugging session starts, In Debug menu, Window->Modules, Right click on the module for your executable, and select “Load Symbols From”. Point to your PDB file, and you are done. It will be important to have source code available if you want to step through. That however is a completely different issue.

Technorati Tags: Debugging,Development,Tips

Keep release PDBs to help with debugging production problems

The Social Processional Network (Twitter, Friendfeed, Facebook, Linkedin, etc..)

I guess the original idea behind a social network is to socialize – communicate to others. The idea of group communication is not new at all. Some people might remember Bulletin Boards from 15-20 years ago, Compuserve / AOL (Still alive somewhat) / NNTP News groups (still available but are very different). Social networks got popular with MySpace and FaceBook. They were originally discarded as “play time” activities and are even now often restricted from access through corporate networks. However their professional use is increasing rapidly.

Stack Overflow

Even though I’ve been using social networking in my professional day-to-day for a wile, I’ve recently started to get a lot more engaged. Some of the dynamics of the site are astounding. I wrote a few comments on my www.StackOverflow.com experience. I’ve been using the site as a public forum for development related questions. It is amazing how fast other developers respond to the questions asked. if a question in well phrased, it can get multiple answers within minutes (sometimes even seconds)

LinkedIn

Another tool is my network arsenal is my LinkedIn. The network grows exponentially as you start adding contacts. At this point, I have 185 direct connections. However, my total network is more then 2.2 million people. Considering that I only add people that I’ve been communicating with into my contacts list, that’s 2.2 million people that I can be pretty comfortable in asking for an introduction to. Until recently, I’ve used LinkedIn to do some background research on potential candidates. However, recently, I’ve started to get involved in the professional groups and ask questions. Unlike StackOverflow, LinkedIn has a very broad range of people that use it to connect to each other. This allows me to tap into the pool of resources and information that is wider then any one area of expertise. At the same time, since this is not a pure “social” network, all communication is professional.

Blog

StackOverflow and LinkedIn are great places to ask for information. Blog on the other hand is a great way to share it. There are blog networks that will provide you not only a place to host your blog, but as a community of readers that are interested in the contents. That will get you feedback on the topics you write.

Twitter

Micro Blogging with Twitter is relatively new (from October 2006). However it is hard to understate the extend of the idea, the importance (and simplicity) of the technology and the reach of the network. There is a lot of talk about twitter on the web

• How to Use Twitter as a Twool
• Poll results: 7 reasons why 69% of users think Twitter will go mainstream
• Wikipedia

Twitter can be used for anything from saying “I am bored” or “I am having lunch” to a more professional “Listening to so-and-so at the conference” or “Found this great article at http://www…” the short statement you make is broadcasted to all subscribers. You can include a subscribers' @name and the message (though still public) will be flagged on his screen so it is noticed. The whole interaction feels like room full of people with multiple conversation going on. You focus on a conversation with one or two people, but you can still overhear others. If you hear something interesting, you join in.

FriendFeed

FriendFeed is an interesting service that takes information push to the next level. It creates one channel for sharing all your information. It connects to over 30 networks including Twitter, Blogs, Facebook, LinkedIn, Flickr, Del.ici.os, and others. It allows to setup friends and will follow their feeds as well. You can even setup an “imaginary” friend to organize someone’s information if they are not participating in FriendFeed.

Importance of Information Push

One thing to notice, is that I am putting a lot of emphasis on pushing the information to others. That is the important aspect of having a social network. Google does excellent job in providing an ability to search and access public information. However, it is the ability to interact by asking questions, by publicizing your ideas is what makes social networks really useful in professional atmosphere.

Technorati Tags: Business,Tools,Internet,Social Network

The Social Processional Network (Twitter, Friendfeed, Facebook, Linkedin, etc..)