BSOD Analysis - Getting Started

The article on 0x124 crashdumps ARC referred too is great, but is kinda outdated and uses an older, insufficient and more difficult method for gaining info on the problem. Your best bet is to just use the !errrec (yes, that's 3 'r's, not 2, as in error record) 'command'* in Windbg and give it Arg2 of the bugcheck code (2nd of the 4 numbers in parentheses next to the 0x124 number), so it'll be !errrec <Arg2> (no <> of course). That'll give you plenty of info, with certain lines like the actual error message which is a lot easier to read (and reliable) than whipping out an Intel/AMD dev manual and trying to interpret MCi status codes.

Articles that explain further are here and here.

* - In Windbg terminology, a command is something built into the Windbg (KD/CDB) engine, and is usually prefixed with a period (.) like .reload or .process, whereas an extension is something that adds more robust functionality and can be either included with Windbg, made personally or downloaded from a site (like cmkd), and they are prefixed with a bang (!), like !analyze or !thread. I just called !errrec - which is an extension - a command to prevent confusion, but it's really technically called an extension.

There's just some types of crashdumps that are indecipherable because they actually provide details only accessible to those with private access to Windows code, like Windows development and support teams. The 0x1A crashes are often a prime example. The only item of much importance with a 0x1A bugcheck is Arg1, which is the subcode. If the subcode present is not mentioned in the list offered by the Windbg help manual for that particular bugcheck, then just disregard it and consider the crashdump worthless.

Even the best and brightest kernel debugging analysts can be easily stumped by a crashdump. Not every crashdump provided can offer relevant information. Some of them just happen too late and the culprit has already left the scene, or they just offer too little detail. The best step for a person doing debugging of this nature is to first make it so that a crashdump will provide something meaningful. Often this would mean turning on Driver Verifier, doing manual crashing, reading a kernel dump instead of a minidump, collecting more minidumps, etc. etc.. If what you have right now is making you scratch your head, STOP, and get more information. Driver Verifier is always the best start for this.

Yes, there's a lot of commands, and there's a lot of information, because we're dealing with a robust OS from the ground up, so learning about all of its intricate parts can kinda be necessary. I have recommendations on literature, sites and whatnot for research purposes, as well as studying tips (like running through Windbg help manual on each command) here, but it's gonna boil down to how much you're willing to learn on it.

Kernel debugging and crashdump analysis is not an entry-level profession. The people usually responsible for dealing with it are escalation engineers which are the highest echelon of computer engineering in the industry. However, it is also not an insurmountable obstacle. There is plenty of documentation and resources available to make the trek easier, but it's going to rely a lot on exposure and experience, which fortunately for here at SF is available in droves. Don't swallow more than you can stomach, and don't bite more than you can chew, and you should be able to make progress in understanding more and more about this field of study.

It's slow and methodical, but the benefits are great. Computer technicians and sysadmins that are able to do forensic troubleshooting like this are very, very few and far inbetween, and makes them stand head and shoulders over the rest of their peers in the industry. Knowledge like this will certainly not go unheeded, as it is a rare and treasured gem which few know of its value until they see it firsthand.

Glad to hear it, skipper. Though don't forget to actually enjoy doing it once in a while. I love seeing it as nerdy detective work, considering myself a bit of a data sleuth who scrutinizes the trail of clues to eventually find the suspect. If you work at it as just that - a trail of evidence - then that can help with your analysis. Sometimes trails end abruptly and you'll want to start back at the crime scene and follow another trail, or you just need to reevaluate the whole thing. Sometimes even the most obvious potential suspect can end up being the most misleading.

It's a rather fascinating venture, and seeing the pieces fit together is always a satisfying experience. Once you learn to deep dive further into a crashdump to find more specific yet relevant details, it becomes more and more exciting - like learning to play an instrument and getting to the point where you can play your favorite songs.

Of course, like playing an instrument, this isn't something for everyone. If all this stuff does is end up making you tear your hair out and down Advil like it's M&Ms, then perhaps you're better off leaving this to someone else and focusing on other fields of study that cater more to your aptitude and interests. Nobody will hold it against you on that. We already know how difficult it is, so to see someone try to begrudgingly work through it with no satisfaction would be a living hell for them!