Solved Plagued by random BSoDs

[TruePikachu]

For quite some time, I've been getting random BSoDs. However, I have never been able to find any pattern between them, except for the possiblilty of memory corruption going on somewhere. Already checked my RAM with memtest86+ v4.20 (which I didn't know was technically outdated) and handled the main offenders (GFX and chipset should be fully up to date, audio is on the OEM-provided version, which I know is most likely not the source of the crashes).

I've cleaned up my minidump dir a number of times, I usually get PAGE_FAULT_IN_NONPAGED_ERROR, SYSTEM_SERVICE_EXCEPTION, and the like. However, most recent from a Verifier check, SPECIAL_POOL_DETECTED_MEMORY_CORRUPTION, searching the followup symbol name ( nt!MiCheckSpecialPoolSlop+9a ) got tons of results over to here; while the trace doesn't say much about the issue from what I can tell (everything is in the NT module), it does look like you guys know quite a bit more about the kernel architecture than I do (as well as how to use WinDbg - I'm personally a gdb guy, but that doesn't really help with this).

Due to the massive size of the current dump (1.68GB), I can't exactly provide it reasonably - 7z got it down to 115MB, but that is still quite big, and if it is more than the kernel in it, I'm not sure I can safely provide it. However, I can provide some of the stuff from WinDbg:
--SNIP--
I usually also have Saitek drivers and PPJoy also, but those both prevent Driver Verifier from working, and neither are the cause of the issues.

"EDIT": WinDbg's output, especially lmtsmn, threw this over the char limit - threw that into an attachment instead.

EDIT: Yes, I do have the MEMORY.DMP on file, and can issue WinDbg commands for it

 

[Golden]

Additional information is required.

1. Download the DM Log Collector application to your desktop by clicking the link below

DM Log Collector.exe

2. Run it by double-clicking the icon on your desktop, and follow the prompts.
3. Locate the .ZIP file that is created on your desktop, and upload it here in your next reply.

 

[cluberti]

Yes indeed. However, I can tell from the windbg output that free kernel pool (not sure if paged or nonpaged) is corrupt, meaning something is likely spraying into pool (versus a buffer overrun or underrun), so special pool given the current settings is probably not useful anyway. Best to start by making sure all drivers are fully up-to-date, as well as Windows patches and any software that uses kernel drivers (like antivirus or disk mounting utilities). After that, if things still crash, we do actually need that large dump file.

 

[TruePikachu]

@Golden: DM log is (and was) in the OP, the .zip
@cluberti: As far as I'm aware, drivers are fully up to date. The audio technically was on a newer version, but is currently on the OEM-provided known-stable (and since it is an IDT codec, one should be using the OEM version anyway).

The large dump file I had was the dump from Verifier - non-Verifier dumps are still large (on the order of ~600MB IIRC compared to 1.6GB), but I know are restricted to the kernel; those I can provide when they become availible. Verifier is currently disabled, so the next dump will be something like the PAGE_FAULT_IN_NONPAGED_AREA or such. (I am quite aware of the relative uselessness of the minidumps; don't they just include the bugcheck, registers, and stack?)

None of the optional system updates from Microsoft apply to the issue at all.

Am I correct in assuming that userland programs can't "spray" into the kernel's pages? I don't run any programs I would suspect write into the kernel directly (anything which might uses a driver).

Disk mounting hadn't been updated, but I've had it on for ages before the BSoDs began. I'll check for updates, but unless they make an assumption which no longer holds true, I don't believe they are the cause.

EDIT: Just for clarification, I use KeePass which might have some of my passwords in user memory at the times of crashes; this is the primary reason why I don't want to provide dumps which include userland unless absolutely necessary.

 

[cluberti]

Am I correct in assuming that userland programs can't "spray" into the kernel's pages? I don't run any programs I would suspect write into the kernel directly (anything which might uses a driver).

Correct, at least not directly. Userland programs that have kernel-mode driver components (like antivirus/antimalware programs, for instance) could do it, but only via the driver. Spray is indicative of a problematic driver, almost always.

 

[TruePikachu]

Okay, first "natural" BSoD

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff8000374a22c, Address of the instruction which caused the bugcheck
Arg3: fffff880079cbe60, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.
(...)
STACK_TEXT:  
fffff880`079cc840 fffff800`0374a0a1 : fffffa80`087b3060 fffff880`079ccb60 00000000`0579cad8 fffff880`079cc950 : nt!PsLookupProcessByProcessId+0x50
fffff880`079cc880 fffff800`0374a2f3 : fffff8a0`14818410 00000000`0000041c 00000000`00010101 00000000`00010100 : nt!PsOpenProcess+0x15f
fffff880`079ccaa0 fffff800`03476e53 : fffffa80`087b3060 fffff880`079ccb60 00000000`00000000 fffffa80`057bf5e0 : nt!NtOpenProcess+0x23
fffff880`079ccae0 00000000`7754151a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0579ca48 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x7754151a

Since the full kernel dump, even compressed, is far too large to be attached, I have it located on my server at http://cdusto.selfip.com/7f_dump_00.zip

 

[TruePikachu]

Another natural, however this one is claiming a hardware issue

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c000001d, Exception code that caused the bugcheck
Arg2: fffff9600010de88, Address of the instruction which caused the bugcheck
Arg3: fffff880079e4ec0, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.
(...)
EXCEPTION_CODE: (NTSTATUS) 0xc000001d - {EXCEPTION}  Illegal Instruction  An attempt was made to execute an illegal instruction.

FAULTING_IP: 
win32k!AllocQEntry+a0
fffff960`0010de88 c4              ???
(...)
STACK_TEXT:  
fffff880`079e58a0 fffff960`00117b4a : fffff900`c207fbd0 00000000`00000010 00000000`00000001 00000000`00000001 : win32k!AllocQEntry+0xa0
fffff880`079e58d0 fffff960`0010c190 : fffff900`c20712b0 fffff880`079e5b60 00000000`00000001 fffff960`001098ca : win32k!DoTimer+0x4e
fffff880`079e5930 fffff960`0010c3b5 : 00000000`00000000 fffff800`000025ff 00000000`00000000 fffffa80`ffffffff : win32k!xxxRealInternalGetMessage+0x6c0
fffff880`079e5a10 fffff960`0010dd99 : 00000000`00000000 00000000`001cfd20 00000000`00000000 fffff800`034c8e53 : win32k!xxxInternalGetMessage+0x35
fffff880`079e5a50 fffff800`034c8e53 : fffffa80`05b07060 00000000`7efdb000 00000000`00000020 00000000`00000c5c : win32k!NtUserGetMessage+0x75
fffff880`079e5ae0 00000000`72d3fe3a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`001cdc28 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x72d3fe3a

Dump is at http://cdusto.selfip.com/7f_dump_01.zip

I personally doubt that it is a hardware issue (due to the passing of MemTest86+); it is also possible that spray hit a code section of memory.

I think I'll load up the huge dump from Verifier, check for where the spray starts, and look for anything which points to it; chances are, if I get hits, the module containing the pointer (excluding anything from Verifier, ofc) might be the culprit.

EDIT: I immediatly see traces of the spray from the Verifier dump; compared against Verifier's fill of 0xF1, 8 bytes at 8 byte intervals, starting at fffff980`5603c6c6, are instead 0xEF. What is the proper value at win32k!AllocQEntry+a0, if it isn't 0xC4?

 

[TruePikachu]

Bumping with another dump

MEMORY_MANAGEMENT (1a)
    # Any other values for parameter 1 must be individually examined.
Arguments:
Arg1: 0000000000041284, A PTE or the working set list is corrupt.
Arg2: fffff8a018a0a001
Arg3: 0000000000014f35
Arg4: fffff781c0000000
(...)
STACK_TEXT:  
fffff880`03124468 fffff800`03524727 : 00000000`0000001a 00000000`00041284 fffff8a0`18a0a001 00000000`00014f35 : nt!KeBugCheckEx
fffff880`03124470 fffff800`034bdbf9 : f3500001`1eb22963 ffffffff`ffffffff 00000000`00000000 fffff800`03648e80 : nt! ?? ::FNODOBFM::`string'+0x4ad3
fffff880`031244b0 fffff800`034bd1a1 : fffff8a0`00000000 00000000`00000001 fffff800`00000000 00000000`00000000 : nt!MiDeleteSystemPagableVm+0x179
fffff880`03124610 fffff800`035fe5b1 : fffffa80`00000020 fffff880`00000000 fffffa80`09ba2700 00000000`00018a0a : nt!MiFreePagedPoolPages+0x12d
fffff880`03124760 fffff800`03602c0b : fffffa80`05b0e820 fffffa80`09494bd0 00000000`00000000 fffffa80`08ec7cb0 : nt!MiFreePoolPages+0x2b1
fffff880`03124870 fffff800`036014f1 : 00000000`00000000 fffff8a0`198bc000 00000000`00000000 fffff800`0365d588 : nt!ExDeferredFreePool+0x34f
fffff880`03124900 fffff800`0348fcd2 : 00000000`00000000 fffff8a0`198bc810 fffffa80`74536d4d 00000000`00000000 : nt!ExFreePoolWithTag+0x411
fffff880`031249b0 fffff800`0376d9db : fffffa80`0c47e8a0 00000000`00000011 00000000`00088089 fffffa80`0c3cb010 : nt!MiDeleteSegmentPages+0x112
fffff880`03124a80 fffff800`0351a311 : fffffa80`0c47e8a8 00000000`00000001 00000000`00000000 00000000`00000631 : nt!MiSegmentDelete+0x7b
fffff880`03124ac0 fffff800`0351a1d5 : 00000000`00000000 00000000`00000080 fffffa80`04e4f990 fffffa80`00000012 : nt!MiProcessDereferenceList+0x131
fffff880`03124b80 fffff800`0376973a : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!MiDereferenceSegmentThread+0x10d
fffff880`03124c00 fffff800`034be8e6 : fffff800`03648e80 fffffa80`04ee8b50 fffff800`03656cc0 00000000`00000000 : nt!PspSystemThreadStartup+0x5a
fffff880`03124c40 00000000`00000000 : fffff880`03125000 fffff880`0311f000 fffff880`03124380 00000000`00000000 : nt!KxStartSystemThread+0x16
(...)
FOLLOWUP_NAME:  Pool_corruption

I have the dump compressing right now, I'll throw it on my server when it is done. http://cdusto.selfip.com/7f_dump_02.zip

 

[TruePikachu]

After a series of BSoDs in the past 36 hours (which all happened during idle time), I'm going to work on manually trying to figure out what driver is responsible, rather than looking at the dumps, which don't seem to be helping much. Can somebody confirm my assumptions please?

• Drivers that were supplied with the system by the OEM (so my current audio driver) are not responsible (validated because I didn't always get the BSoDs)
• Drivers supplied by Microsoft as a core part of Windows are not responsible (validated since not everyone running the OS gets the BSoDs, at least not this frequently)
• Third-party drivers installed on the system `cWindows` are probably not responsible (there are at most a handful of drivers in common; cWindows is a 32-bit XP over here which hasn't had a BSoD for ages)
• Third-party drivers installed on the systems `jLaptop` and `kLaptop` are probably not responsible (those two systems have Win7x64, and are by the same vendor as this system; however, I can't confirm that they don't get these problems)
• Third-party drivers which have been updated might not be responsible (it is possible that updating my GFX, for instance, might still have the problem because it is an issue that wasn't fixed)
• The driver for my 3D mouse is not responsible (this just narrows it down very slightly, but I just got the mouse this year, and the BSoDs began long before then)

EDIT: Driver list, as exported from DriverView, is at http://cdusto.selfip.com/cLaptop-drivers.txt . It is the list of the potential canidates, and will remain updated since editing it is not dependant on this system's stability

 

[cluberti]

downloading the dump file from 2 days ago right now. I would say that trusting drivers from the OEM is *usually* OK, although there's no way to be certain. Microsoft does not write drivers, they only push out drivers with security or stability updates after working with OEMs, so drivers gotten from Windows Update / Microsoft Update are no better (or worse) than what would come from the OEM directly.

In my experience, I generally start with antivirus software and software that emulates (or allows writing to) external devices. From there I look at video drivers, and after that I move down to audio, USB, and chipset drivers.

 

[TruePikachu]

downloading the dump file from 2 days ago right now. I would say that trusting drivers from the OEM is *usually* OK, although there's no way to be certain. Microsoft does not write drivers, they only push out drivers with security or stability updates after working with OEMs, so drivers gotten from Windows Update / Microsoft Update are no better (or worse) than what would come from the OEM directly.

In my experience, I generally start with antivirus software and software that emulates (or allows writing to) external devices. From there I look at video drivers, and after that I move down to audio, USB, and chipset drivers.

OEM drivers which I have not had issues with in the past. As in the specific driver. For example, Accelerometer.sys was never updated, and there was a time when I didn't get all these BSoDs.

MS does have a few generic drivers which are in use - things like vga.sys and usbhub.sys. There are also kernel parts found by e.g. DriverView. But I'm not saying that drivers from manufacturers on winupdate are any more reliable; I'm saying that drivers which are a part of Windows's operation and which exist and are likely in use on just about every single system are probably not the cause of the issues.

AV is Security Essentials, haven't seen any kernel stability updates for that; it likely isn't the cause. None of my virtual disk managers have been updated on here for ages, Virtual Box is the only possible canidate in that category. Video, USB, and chipset are all up-to-date, and audio is the OEM original (I had reverted it to remove it as a canidate).

I'll get VBox updated, see if anything changes.

 

[cluberti]

IN the latest dump, something in a shared svchost created a page fault, and this triggered a paged pool lookup - it complains about memory corruption (PTE invalid), which really doesn't help much. I've been sidetracked today, but I'll try to dig deeper tomorrow.

 

[TruePikachu]

Updating VirtualBox to the latest version did not fix anything; I have experienced 2 BSoDs since updating (the first one terminating the string of uptime including the update, the second one not having the old driver loaded in memory at all).

 

[TruePikachu]

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: ffffffffc0000005, The exception code that was not handled
Arg2: fffff96000735e4d, The address that the exception occurred at
Arg3: fffff880025f4638, Exception Record Address
Arg4: fffff880025f3e90, Context Record Address
(...)
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: 
cdd!PresentWorkerThread+ae1
fffff960`00735e4d f7f1            div     eax,ecx
(...)
STACK_TEXT:  
fffff880`025f4870 fffff800`0371a73a : 00000000`040e3231 fffffa80`07ffa710 00000000`00000080 fffffa80`07fd1b30 : cdd!PresentWorkerThread+0xae1
fffff880`025f4c00 fffff800`0346f8e6 : fffff880`009eb180 fffffa80`07ffa710 fffffa80`06627060 00000028`00000000 : nt!PspSystemThreadStartup+0x5a
fffff880`025f4c40 00000000`00000000 : fffff880`025f5000 fffff880`025ef000 fffff880`025f43a0 00000000`00000000 : nt!KxStartSystemThread+0x16

http://cdusto.selfip.com/7f_dump_03.zip

 

[cluberti]

That call on the stack, cdd!PresentWorkerThread's next target is DirectX. That would point towards video drivers (and supporting software) in that dump. Note that most WDDM video drivers have both user-mode and kernel-mode components to make them work, so... that would fit the pattern of the previous dumps and kernel spray too....

 

[TruePikachu]

Just did a clean reinstall of the GFX (AMD's uninstaller for CCC and the display drivers, safe mode run of DDU set to not remove displays (monitors)*, reinstall of CCC, display drivers, and pretty much everything but the chipset. First sign is that CCC no longer crashes immediatly (which it had been since my less recent driver update), however I'm still on the same version numbers as reported by DriverView. While it might be the case that AMD doesn't always increment their version numbers, what is more likely is that I'm on the same drivers again.

*it is a pain to recalibrate color on all the displays I use when I dual-head - this is not related to the crashes, since they also occur during sessions when Windows has only used the primary display

 

[TruePikachu]

Quick update, no BSoDs since reinstalling the video drivers, but Bitcoin has been randomly crashing, so it is possible that there _might_ still be an issue, which hasn't hit the kernel yet due to random chance.

 

[cluberti]

You'd have to get a process dump of bitcoin crashing to know for sure. DebugDiag and a crash rule for the bitcoin process .exe can help in that regard.

 

[TruePikachu]

And a new BSoD. Too much of a headache right now to load up the dump in WinDbg, WhoCrashed report:[FONT=Segoe UI, Arial]
On Wed 1/28/2015 11:47:53 PM GMT your computer crashed
crash dump file:[FONT=Segoe UI, Arial] [ http://cdus[FONT=Segoe UI, Arial]to.selfip.com/7f_dump_04.zip ][/FONT][/FONT]
This was probably caused by the following module: [FONT=Segoe UI, Arial]ntkrnlmp.exe[/FONT] (nt!KeBugCheckEx+0x0)
Bugcheck code: 0x1 (0x774313BA, 0x0, 0xFFFFFFFF, 0xFFFFF88007EA4B60)
Error: [FONT=Segoe UI, Arial]APC_INDEX_MISMATCH[/FONT]
Bug check description: This indicates that there has been a mismatch in the APC state index.
The crash took place in the Windows kernel. Possibly this problem is caused by another driver that cannot be identified at this time.
[/FONT]

 

[TruePikachu]

Back to crashing the "usual" amount, couple of SYSTEM_SERVICE_EXCEPTIONs today, as well as Explorer crashing.