How To Do Hang Dump analysis
This article walks through the analysis of an application hang caused by a chain of RPC calls.The first part of the article discusses the manually generated application memory dump (user mode dump) and the second part focuses on the manually generated kernel mode dump (complete memory dump).
The user observed sporadic hangs in a particular application and was instructed by technical support engineers to capture an application memory dump using the SysInternals tool procdump with the command line "procdump -ma HungApplication.exe".
Since LPC is implemented in the Windows kernel, to perform any further analysis involving this LPC call requires a kernel mode dump of this system. The next section describes the steps for analyzing a complete memory dump of this system. Note that in more recent version of Windows (i.e. Vista+) the LPC mechanism has been superseded by the Advanced Local Inter-process Communication (ALPC) mechanism.
Since the application dump analysis was inconclusive and the support engineer was unable to attribute the hang to a particular component, the engineer requested the user to generate a complete memory dump of the entire system to be able to debug at the kernel level. This required the system to be configured to generate a complete memory dump via command line "wmic recoveros set DebugInfoType = 1". It also required the system to be configured to crash when the user pressed the keyboard combination Right-Ctrl + Scroll-Lock + Scroll-Lock as described in Microsoft Knowledge Base Article KB244139. Fortunately the system on which the HungApplication.exe was running was already configured for both these options from a past technical support incident. So the user was able to bug-check the system manually, reboot the system and obtain the complete kernel dump from c:\Windows\Memory.dmp.
The output of "!analyze -v" shows that this is a manually generated dump. To start the analysis obtain the call stack of the thread that was identified in the user mode dump analysis i.e. thread (TID=0xc8).
In the above analysis there are bunch of threads that are waiting in a chain for this operation to complete. This last thread in this chain is the RPC client thread in the application HungApplication.exe. From the information obtained so far, one can conclude that the network stack is preventing the application HungApplication.exe from making forward progress and hence it is responsible for the hang.
Not all bugs can be found prior to release, which means not all bugs that throw exceptions can be found before release. Fortunately, Microsoft has included in the Platform SDK a function to help developers collect information on exceptions that are discovered by users. The MiniDumpWriteDump function writes the necessary crash dump information to a file without saving the whole process space. This crash dump information file is called a minidump. This technical article provides info about how to write and use a minidump.
Do nothing. Windows automatically generates a minidump whenever a program throws an unhandled exception. Automatic generation of a minidump is available since Windows XP. If the user allows it, the minidump will be sent to Microsoft, and not to the developer, through Windows Error Reporting (WER). Developers can gain access to these minidumps through the Windows Desktop Application Program.
If you implement a custom routine for unhandled exceptions, you are strongly urged to use the ReportFault function in the exception handler to also send an automated minidump to WER. The ReportFault function handles all of the issues of connecting to and sending the minidump to WER. Not sending minidumps to WER violates the requirements of Games for Windows.
Use a product from the Microsoft Visual Studio Team System. On the Debug menu, click Save Dump As to save a copy of a dump. Use of a locally saved dump is only an option for in-house testing and debugging.
Add code to your project. Add the MiniDumpWriteDump function and the appropriate exception handling code to save and send a minidump directly to the developer. This article demonstrates how to implement this option. However, note that MiniDumpWriteDump does not currently work with managed code and is only available on Windows XP, Windows Vista, Windows 7.
This example demonstrates the basic usage of MiniDumpWriteDump and the minimum information necessary to call it. The name of the dump file is up to the developer; however, to avoid file name collisions, it is advisable to generate the file name from the application's name and version number, the process and thread IDs, and the date and time. This will also help to keep the minidumps grouped by application and version. It is up to the developer to decide how much information is used to differentiate minidump file names.
It should be noted that the path name in the preceding example was generated by calling the GetTempPath function to retrieve the path of the directory designated for temporary files. Use of this directory works even with least-privileged user accounts, and it also prevents the minidump from taking up hard drive space after it is no longer needed.
Dumpchk.exe is a command-line utility that can be used to verify that a dump file was created correctly. If Dumpchk.exe generates an error, then the dump file is corrupt and cannot be analyzed. For information on using Dumpchk.exe, see How to Use Dumpchk.exe to Check a Memory Dump File.
You can also use WinDbg, a debugger that is part of the Windows Debugging Tools, to debug a minidump. WinDbg allows you to debug without having to use Visual Studio. To download Windows Debugging Tools, see Windows Debugging Tools on Windows Hardware Developer Central.
Developers also need to be aware of how their copy-protection scheme might affect the minidump. Most copy-protection schemes have their own descramble tools, and it is up to the developer to learn how to use those tools with MiniDumpWriteDump.
Over the course of the last year I have been tasked with analyzing our production environments, specifically looking at performances issues, hangs and crash analysis using the Debug Diagnostic Tool, Performance Monitor and Debugging Tools for Windows (WinDbg).
Having the tools and commands is one thing, understanding context is a whole other question, if you need help with that I would strongly recommend visiting the blog of Tess Fernandez. A few years back she produced a legendary series of detailed hang analysis articles that remain wholly relevant today.
When unexpected crashes occur in your managed application you are often left with little evidence of the issue; capturing and analyzing memory dumps may be your last best option. Thankfully Visual Studio is a great tool for analyzing your apps memory dumps! In this post we show you how easy it is to get important insights from a crash dump, and the steps to resolve the issue using Visual Studio.
Memory dumps are a great diagnostic tool because they are a complete snapshot of what a process is doing at the time the dump is captured. There are several tools available for capturing memory dumps including Visual Studio, ProcDump, DebugDiag and WinDbg. The relative strength of each tool depends on your environment and the scenario you are investigating (e.g. high CPU, memory leaks, first/second chance exceptions, etc.).
In the following example, I use the versatile ProcDump command-line utility from Sysinternals to capture a full user-mode dump (-ma) when an unhandled exception (-e) occurs (1145 is the process id of my application).
The Dump Summary page highlights several pieces of important information from the dump file including the OS Version and CLR Version. I can also search through a list of the modules that were loaded into memory at the time I captured the memory dump.
Over the years Visual Studio has developed first class support for handling and debugging memory dumps. It allows you to consider the impact your code is having during the exact moment of a catastrophic failure. Having the ability to investigate a problem with the same tools used for developing code can help save time determining and providing a solution.
The ".dmp" file includes the stop error message, a list of the drivers loaded at the time of the problem, kernel, processor, and process details, as well as other information depending on the type of dump file you have.
Although Windows 10 automatically creates dump files, the only problem is that you won't find any built-in tools to open this type of file, and this is when the Microsoft Windows Debugging (WinDbg) tool can help. WinDbg is an advanced tool designed for debugging kernel-mode and user-mode code, reviewing processor registries, and analyzing crash dumps.
The information will be different depending on the problem. For example, this result points out that this was a manually initiated crash with an "e2" error code, which is correct since, for this guide, we use these instructions to force a dump file. The WinDbg even makes an excellent job describing the crash in a language anyone can understand ("The user manually initiated this crash dump").
Whenever a computer running Windows suddenly reboots without displaying any notice or blue or black screen of death, the first thing that is often thought about is a hardware failure. In reality, crashes are often caused by malfunctioning device drivers and kernel modules. In case of a kernel error, computers running Windows do not show a blue or black screen unless they are configured to do so. Instead these systems suddenly reboot without any notice. WhoCrashed shows the drivers that have been crashing your computer with a single click. In most cases it can pinpoint the offending drivers that have been causing misery on your computer system in the past. It does post-mortem crashdump analysis and presents all gathered information in a comprehensible way. In case your system has been crashing because of a hardware failure, WhoCrashed will provide you with guidance to trace the root cause of the problem. Normally, debugging skills and a set of debugging tools are required to do post-mortem crash dump analysis. Using WhoCrashed you do not need any debugging skills to be able to find out what drivers are causing trouble to your computer. 2b1af7f3a8