Plagued by random BSoDs

TruePikachu · 02 Jan 2015

For quite some time, I've been getting random BSoDs. However, I have never been able to find any pattern between them, except for the possiblilty of memory corruption going on somewhere. Already checked my RAM with memtest86+ v4.20 (which I didn't know was technically outdated) and handled the main offenders (GFX and chipset should be fully up to date, audio is on the OEM-provided version, which I know is most likely not the source of the crashes).

I've cleaned up my minidump dir a number of times, I usually get PAGE_FAULT_IN_NONPAGED_ERROR, SYSTEM_SERVICE_EXCEPTION, and the like. However, most recent from a Verifier check, SPECIAL_POOL_DETECTED_MEMORY_CORRUPTION, searching the followup symbol name ( nt!MiCheckSpecialPoolSlop+9a ) got tons of results over to here; while the trace doesn't say much about the issue from what I can tell (everything is in the NT module), it does look like you guys know quite a bit more about the kernel architecture than I do (as well as how to use WinDbg - I'm personally a gdb guy, but that doesn't really help with this).

Due to the massive size of the current dump (1.68GB), I can't exactly provide it reasonably - 7z got it down to 115MB, but that is still quite big, and if it is more than the kernel in it, I'm not sure I can safely provide it. However, I can provide some of the stuff from WinDbg:
--SNIP--
I usually also have Saitek drivers and PPJoy also, but those both prevent Driver Verifier from working, and neither are the cause of the issues.

"EDIT": WinDbg's output, especially lmtsmn, threw this over the char limit - threw that into an attachment instead.

EDIT: Yes, I do have the MEMORY.DMP on file, and can issue WinDbg commands for it

Golden · 02 Jan 2015

Additional information is required.

1. Download the DM Log Collector application to your desktop by clicking the link below

DM Log Collector.exe

2. Run it by double-clicking the icon on your desktop, and follow the prompts.
3. Locate the .ZIP file that is created on your desktop, and upload it here in your next reply.

**cluberti** · 02 Jan 2015

Yes indeed. However, I can tell from the windbg output that free kernel pool (not sure if paged or nonpaged) is corrupt, meaning something is likely spraying into pool (versus a buffer overrun or underrun), so special pool given the current settings is probably not useful anyway. Best to start by making sure all drivers are fully up-to-date, as well as Windows patches and any software that uses kernel drivers (like antivirus or disk mounting utilities). After that, if things still crash, we do actually need that large dump file.

TruePikachu · 02 Jan 2015

@Golden: DM log is (and was) in the OP, the .zip
@cluberti: As far as I'm aware, drivers are fully up to date. The audio technically was on a newer version, but is currently on the OEM-provided known-stable (and since it is an IDT codec, one should be using the OEM version anyway).

The large dump file I had was the dump from Verifier - non-Verifier dumps are still large (on the order of ~600MB IIRC compared to 1.6GB), but I know are restricted to the kernel; those I can provide when they become availible. Verifier is currently disabled, so the next dump will be something like the PAGE_FAULT_IN_NONPAGED_AREA or such. (I am quite aware of the relative uselessness of the minidumps; don't they just include the bugcheck, registers, and stack?)

None of the optional system updates from Microsoft apply to the issue at all.

Am I correct in assuming that userland programs can't "spray" into the kernel's pages? I don't run any programs I would suspect write into the kernel directly (anything which might uses a driver).

Disk mounting hadn't been updated, but I've had it on for ages before the BSoDs began. I'll check for updates, but unless they make an assumption which no longer holds true, I don't believe they are the cause.

EDIT: Just for clarification, I use KeePass which might have some of my passwords in user memory at the times of crashes; this is the primary reason why I don't want to provide dumps which include userland unless absolutely necessary.

**cluberti** · 02 Jan 2015

TruePikachu said:

Am I correct in assuming that userland programs can't "spray" into the kernel's pages? I don't run any programs I would suspect write into the kernel directly (anything which might uses a driver).

Correct, at least not directly. Userland programs that have kernel-mode driver components (like antivirus/antimalware programs, for instance) could do it, but only via the driver. Spray is indicative of a problematic driver, almost always.

TruePikachu · 04 Jan 2015

Okay, first "natural" BSoD

Code:

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff8000374a22c, Address of the instruction which caused the bugcheck
Arg3: fffff880079cbe60, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.
(...)
STACK_TEXT:  
fffff880`079cc840 fffff800`0374a0a1 : fffffa80`087b3060 fffff880`079ccb60 00000000`0579cad8 fffff880`079cc950 : nt!PsLookupProcessByProcessId+0x50
fffff880`079cc880 fffff800`0374a2f3 : fffff8a0`14818410 00000000`0000041c 00000000`00010101 00000000`00010100 : nt!PsOpenProcess+0x15f
fffff880`079ccaa0 fffff800`03476e53 : fffffa80`087b3060 fffff880`079ccb60 00000000`00000000 fffffa80`057bf5e0 : nt!NtOpenProcess+0x23
fffff880`079ccae0 00000000`7754151a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0579ca48 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x7754151a

Since the full kernel dump, even compressed, is far too large to be attached, I have it located on my server at http://cdusto.selfip.com/7f_dump_00.zip

TruePikachu · 04 Jan 2015

Another natural, however this one is claiming a hardware issue

Code:

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c000001d, Exception code that caused the bugcheck
Arg2: fffff9600010de88, Address of the instruction which caused the bugcheck
Arg3: fffff880079e4ec0, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.
(...)
EXCEPTION_CODE: (NTSTATUS) 0xc000001d - {EXCEPTION}  Illegal Instruction  An attempt was made to execute an illegal instruction.

FAULTING_IP: 
win32k!AllocQEntry+a0
fffff960`0010de88 c4              ???
(...)
STACK_TEXT:  
fffff880`079e58a0 fffff960`00117b4a : fffff900`c207fbd0 00000000`00000010 00000000`00000001 00000000`00000001 : win32k!AllocQEntry+0xa0
fffff880`079e58d0 fffff960`0010c190 : fffff900`c20712b0 fffff880`079e5b60 00000000`00000001 fffff960`001098ca : win32k!DoTimer+0x4e
fffff880`079e5930 fffff960`0010c3b5 : 00000000`00000000 fffff800`000025ff 00000000`00000000 fffffa80`ffffffff : win32k!xxxRealInternalGetMessage+0x6c0
fffff880`079e5a10 fffff960`0010dd99 : 00000000`00000000 00000000`001cfd20 00000000`00000000 fffff800`034c8e53 : win32k!xxxInternalGetMessage+0x35
fffff880`079e5a50 fffff800`034c8e53 : fffffa80`05b07060 00000000`7efdb000 00000000`00000020 00000000`00000c5c : win32k!NtUserGetMessage+0x75
fffff880`079e5ae0 00000000`72d3fe3a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`001cdc28 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x72d3fe3a

Dump is at http://cdusto.selfip.com/7f_dump_01.zip

I personally doubt that it is a hardware issue (due to the passing of MemTest86+); it is also possible that spray hit a code section of memory.

I think I'll load up the huge dump from Verifier, check for where the spray starts, and look for anything which points to it; chances are, if I get hits, the module containing the pointer (excluding anything from Verifier, ofc) might be the culprit.

EDIT: I immediatly see traces of the spray from the Verifier dump; compared against Verifier's fill of 0xF1, 8 bytes at 8 byte intervals, starting at fffff980`5603c6c6, are instead 0xEF. What is the proper value at win32k!AllocQEntry+a0, if it isn't 0xC4?

TruePikachu · 12 Jan 2015

Bumping with another dump

Code:

MEMORY_MANAGEMENT (1a)
    # Any other values for parameter 1 must be individually examined.
Arguments:
Arg1: 0000000000041284, A PTE or the working set list is corrupt.
Arg2: fffff8a018a0a001
Arg3: 0000000000014f35
Arg4: fffff781c0000000
(...)
STACK_TEXT:  
fffff880`03124468 fffff800`03524727 : 00000000`0000001a 00000000`00041284 fffff8a0`18a0a001 00000000`00014f35 : nt!KeBugCheckEx
fffff880`03124470 fffff800`034bdbf9 : f3500001`1eb22963 ffffffff`ffffffff 00000000`00000000 fffff800`03648e80 : nt! ?? ::FNODOBFM::`string'+0x4ad3
fffff880`031244b0 fffff800`034bd1a1 : fffff8a0`00000000 00000000`00000001 fffff800`00000000 00000000`00000000 : nt!MiDeleteSystemPagableVm+0x179
fffff880`03124610 fffff800`035fe5b1 : fffffa80`00000020 fffff880`00000000 fffffa80`09ba2700 00000000`00018a0a : nt!MiFreePagedPoolPages+0x12d
fffff880`03124760 fffff800`03602c0b : fffffa80`05b0e820 fffffa80`09494bd0 00000000`00000000 fffffa80`08ec7cb0 : nt!MiFreePoolPages+0x2b1
fffff880`03124870 fffff800`036014f1 : 00000000`00000000 fffff8a0`198bc000 00000000`00000000 fffff800`0365d588 : nt!ExDeferredFreePool+0x34f
fffff880`03124900 fffff800`0348fcd2 : 00000000`00000000 fffff8a0`198bc810 fffffa80`74536d4d 00000000`00000000 : nt!ExFreePoolWithTag+0x411
fffff880`031249b0 fffff800`0376d9db : fffffa80`0c47e8a0 00000000`00000011 00000000`00088089 fffffa80`0c3cb010 : nt!MiDeleteSegmentPages+0x112
fffff880`03124a80 fffff800`0351a311 : fffffa80`0c47e8a8 00000000`00000001 00000000`00000000 00000000`00000631 : nt!MiSegmentDelete+0x7b
fffff880`03124ac0 fffff800`0351a1d5 : 00000000`00000000 00000000`00000080 fffffa80`04e4f990 fffffa80`00000012 : nt!MiProcessDereferenceList+0x131
fffff880`03124b80 fffff800`0376973a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!MiDereferenceSegmentThread+0x10d
fffff880`03124c00 fffff800`034be8e6 : fffff800`03648e80 fffffa80`04ee8b50 fffff800`03656cc0 00000000`00000000 : nt!PspSystemThreadStartup+0x5a
fffff880`03124c40 00000000`00000000 : fffff880`03125000 fffff880`0311f000 fffff880`03124380 00000000`00000000 : nt!KxStartSystemThread+0x16
(...)
FOLLOWUP_NAME:  Pool_corruption

~~I have the dump compressing right now, I'll throw it on my server when it is done.~~ http://cdusto.selfip.com/7f_dump_02.zip

TruePikachu · 14 Jan 2015

After a series of BSoDs in the past 36 hours (which all happened during idle time), I'm going to work on manually trying to figure out what driver is responsible, rather than looking at the dumps, which don't seem to be helping much. Can somebody confirm my assumptions please?

Drivers that were supplied with the system by the OEM (so my current audio driver) are not responsible (validated because I didn't always get the BSoDs)
Drivers supplied by Microsoft as a core part of Windows are not responsible (validated since not everyone running the OS gets the BSoDs, at least not this frequently)
Third-party drivers installed on the system `cWindows` are probably not responsible (there are at most a handful of drivers in common; cWindows is a 32-bit XP over here which hasn't had a BSoD for ages)
Third-party drivers installed on the systems `jLaptop` and `kLaptop` are probably not responsible (those two systems have Win7x64, and are by the same vendor as this system; however, I can't confirm that they don't get these problems)
Third-party drivers which have been updated might not be responsible (it is possible that updating my GFX, for instance, might still have the problem because it is an issue that wasn't fixed)
The driver for my 3D mouse is not responsible (this just narrows it down very slightly, but I just got the mouse this year, and the BSoDs began long before then)

EDIT: Driver list, as exported from DriverView, is at http://cdusto.selfip.com/cLaptop-drivers.txt . It is the list of the potential canidates, and will remain updated since editing it is not dependant on this system's stability

**cluberti** · 14 Jan 2015

downloading the dump file from 2 days ago right now. I would say that trusting drivers from the OEM is *usually* OK, although there's no way to be certain. Microsoft does not write drivers, they only push out drivers with security or stability updates after working with OEMs, so drivers gotten from Windows Update / Microsoft Update are no better (or worse) than what would come from the OEM directly.

In my experience, I generally start with antivirus software and software that emulates (or allows writing to) external devices. From there I look at video drivers, and after that I move down to audio, USB, and chipset drivers.