Solved Plagued by random BSoDs

[TruePikachu]

And last night I got another BSoD. It didn't dump (and it doesn't appear to have written to the log), it said something about memory issues. I don't recall the exact message, but if I see the message I'd reconise it.

 

[TruePikachu]

Ran the memory tester again for 20 hours, with the lid closed and internal cooling only; this way, if the issue was from RAM and thermal-related, I would get failures. Result ended up as 11 pass and 0 fail, while I've had the BSoDs occur in less time with less processor load. I'm convinced that whatever driver is causing this gave Windows a false positive for bad memory in the most recent BSoD.

 

[TruePikachu]

Quick update, just enabled Verifier again, set to only monitor the special pool (which is what triggered the previous Verifier dump). Hopefully this will have a minimal impact on preformance, compared to several of the other options.

EDIT: Quick side question, any way to get Windows to run a program with access to MEMORY.DMP on boot (not login)? I want to throw something quick together which will compress the MEMORY.DMPs automatically, or at least move them out of the way in case there is another crash before I grab the dump.

 

[cluberti]

No, because during boot Windows checks for the presence of a dump file header in pagefile.sys, and renames it memory.dmp (and recreates pagefile.sys if so) - assuming you're not using DedicatedDumpFile, of course. Note if you uncheck the "overwrite any existing file" in the dump options window, it will automatically rename .dmp files - it's not compression, but it will keep multiple .dmp files instead of constantly overwriting memory.dmp.

 

[TruePikachu]

BSoD, PAGE_FAULT_IN_NONPAGED_AREA:

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced.  This cannot be protected by try-except,
it must be protected by a Probe.  Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: fffff96000bd02a8, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff960000cd22a, If non-zero, the instruction address which referenced the bad memory
    address.
Arg4: 0000000000000002, (reserved)
(...)
STACK_TEXT:  
fffff880`02af0788 fffff800`035465e4 : 00000000`00000050 fffff960`00bd02a8 00000000`00000000 fffff880`02af08f0 : nt!KeBugCheckEx
fffff880`02af0790 fffff800`034c7cee : 00000000`00000000 fffff960`00bd02a8 00000000`40000100 fffff900`c0408a00 : nt! ?? ::FNODOBFM::`string'+0x43836
fffff880`02af08f0 fffff960`000cd22a : 00000000`0dc5fd20 fffffa80`09cb9630 fffff880`02af0b60 00000000`00022f78 : nt!KiPageFault+0x16e
fffff880`02af0a80 fffff960`000e2a2d : 00000000`000a05c0 00000000`fff5c000 00000000`0dc5fd20 00000000`fff5c000 : win32k!ValidateHwnd+0xda
fffff880`02af0ab0 fffff800`034c8e53 : fffffa80`09cb9630 00000000`0f44f1d8 00000000`00000000 00000000`0000c028 : win32k!NtUserGetProp+0x25
fffff880`02af0ae0 00000000`72a7feba : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0dc5e5f8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x72a7feba

Dump is at http://cdusto.selfip.com/7f_dump_05.zip

Still waiting on a new Verifier bugcheck.

 

[TruePikachu]

How does it rename the dump files? I just searched through the Windows/ tree, I don't see them, and there should be at least 2 from the past 24 hours...

c:\Windows>find | grep -i dmp$
find: `./Logs/SystemRestore': Permission denied
./LiveKernelReports/WATCHDOG/WD-20141129-2203.dmp
./LiveKernelReports/WATCHDOG/WD-20141214-2115.dmp
./LiveKernelReports/WATCHDOG/WD-20150103-1632.dmp
./LiveKernelReports/WATCHDOG/WD-20150105-1726.dmp
./LiveKernelReports/WATCHDOG/WD-20150108-2305.dmp
./LiveKernelReports/WATCHDOG/WD-20150116-1849.dmp
./LiveKernelReports/WATCHDOG/WD-20150124-1206.dmp
./LiveKernelReports/WATCHDOG/WD-20150125-1442.dmp
./LiveKernelReports/WATCHDOG/WD-20150130-0126.dmp
find: `./Logs/WindowsBackup': Permission denied
./MEMORY.DMP
find: `./PCHEALTH/ERRORREP/QHEADLES': Permission denied
find: `./PCHEALTH/ERRORREP/QSIGNOFF': Permission denied
./Minidump/020715-34133-01.dmp
./Minidump/020815-27768-01.dmp
./Minidump/020815-28516-01.dmp
./Minidump/020815-29016-01.dmp
./Minidump/020915-25802-01.dmp
./Minidump/021215-34944-01.dmp
find: `./Prefetch': Permission denied
./ServiceProfiles/LocalService/AppData/Local/CrashDumps/svchost.exe.1076.dmp
./ServiceProfiles/LocalService/AppData/Local/CrashDumps/svchost.exe.1980.dmp
./System32/com/dmp
find: `./System32/LogFiles/HTTPERR': Permission denied
./System32/LogFiles/WUDF/WudfHost_ext__1508.dmp
./SysWOW64/com/dmp
find: `./Temp/72ACDBDC-322C-447D-988B-A292E804BEC4-Sigs': Permission denied
find: `./Temp/SDIAG_3034da67-67e4-4a91-bbed-f5bb7d7602f9': Permission denied
./winsxs/amd64_microsoft-windows-icm-profiles_31bf3856ad364e35_6.1.7600.16385_none_f5547dd01f628131/wscRGB.cdmp
./winsxs/amd64_microsoft-windows-icm-profiles_31bf3856ad364e35_6.1.7600.16385_none_f5547dd01f628131/wsRGB.cdmp

(please note that I have the Cygwin find(1) in my %PATH% before the Windows find.exe)

 

[cluberti]

The file should be (by default) stored in %windir%\memory.dmp. When Windows crashes, the contents of memory (small, kernel, or full, depending on type) are written to the paging file. On reboot, there's a process that runs during system init that checks the paging file to see if it contains memory dump headers - if so, the paging file is renamed to memory.dmp (and potentially a number in parenthesis afterwards, if a file with that name already exists) and then a new paging file is created. Once that completes, Windows continues booting.

 

[TruePikachu]

Well, it doesn't appear to be doing that for some reason, and it is set to NOT overwrite the file. I might also note that the modification date is set to the 12th, so it would appear that it isn't writing the full dumps now. Setting it back to overwrite dumps.

 

[cluberti]

You might need to delete the original memory.dmp file if it existed before you checked that box for it to work...

 

[TruePikachu]

Just started playing X Rebirth, which is a memory and CPU hog. I say memory hog since running it tends to cause everything else to swap out.

Well, it just hung on a Wait:WrPageIn, and is unable to be killed. I have Process Explorer dumping it right now, but a quick search indicates that it might be a driver problem. However, I'm unable to get a debugger attached to the kernel (but I did just `bcdedit -debug on`), so I can't really track the issue right now. I'm additionally unable to get a stack trace via Process Explorer ("Error accessing thread"), so I'm pretty much stuck right now, and going to have to reboot manually.

I should have kernel debug availible afterwards, though, so if I get this hang again, it might be possible to track down the driver. Unless, of course, this issue just now arose because of memory corruption...

 

[cluberti]

If you're on x64, you need to set DisablePagingExecutive to 1 to get good stack information, for what it's worth.

 

[TruePikachu]

And now I'm starting to get "frequent" usermode crashes. A while ago, LCore.exe (mouse profiling software) crashed, it crashed again about an hour later (after restarting it), and Explorer crashed while I was dumping. The exception from the Windows error reporting was ACCESS_VIOLATION, for both LCore.exe crashes and Explorer. Just added Windows Error Reporting to my "Bugcheck" custom view (in Event Viewer), and I can confirm that these ACCESS_VIOLATIONS go on for quite some time.

 

[TruePikachu]

Random thought: is there some way to get the actual location on the RAM that a virtual address corresponds to? If this is some RAM failure, it is likely that everything would run into issues around the same block of RAM.

In other news, X apparently runs somewhat slowly partially since I'm on only 6GB of RAM, so I will probably see about upgrading to at least 2x4GB (replacing my 2GB stick). If there is a RAM issue on a single stick, it should be possible to isolate it with a bit of experimentation.
Just a side question, why should I have both sticks being the same capacity? I have 64MB+128MB in my main server box, something which adds up to 1.37GB in my game server box, and I have no idea what in a seperate Windows box; those three are rock-solid systems.

 

[TruePikachu]

Couple new dumps are up at http://cdusto.selfip.com/7f_dump_06.zip. I'm also getting symbol resolve issues on that pair of dumps

 

[TruePikachu]

Just a possible curiosity, I have my system in my custom "Display always on" power configuration, and it hasn't BSoD'd in the middle of the night when I leave it on (though note that some nights I put it to hibernation). Recently, I've only had to reboot the system when:
* Dial-up networking doesn't release (?) COM10 (BT link to my cell phone) properly when Bluetooth gets disabled (to note, it almost always releases the port correctly)
* Flash player causes the system to lock up (this one is confusing, since Flash doesn't have a kernel portion, and OS-reserved hotkeys (Super-L, Ctrl+Alt+Del, etc.) don't go through)
* Installing updates for Windows
* Swap blocks when X is unresponsive due to needing to swap

 

[TruePikachu]

Quick update, I'm tending to not get BSoDs now, which is actually a step backwards because now I'm just getting lockups.

I'm still biased towards it being part of the display driver; no crashes in the middle of the night if the display doesn't go into standby, and most lockups are on battery power, where the display hardware is likely running a seperate graphics adapter (I'm not entirely clear on how the 7520G works).

The current driver versions are dated 11/2014, so any issue in them would have persisted for some time.

 

[TruePikachu]

Got a swap hang from X, and it looks graphics related...

lkd> !thread fffffa800b7d2b50 16
THREAD fffffa800b7d2b50  Cid 178c.15bc  Teb: 000007ffb6544000 Win32Thread: fffff900c2c6bc20 WAIT: (WrPageIn) KernelMode Non-Alertable
    fffffa8006bae408  NotificationEvent
Not impersonating
DeviceMap                 fffff8a0057ecee0
Owning Process            fffffa800546e340       Image:         XRebirth.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      4802804        Ticks: 191092 (0:00:49:41.054)
Context Switch Count      182538         IdealProcessor: 0                 LargeStack
UserTime                  00:00:56.597
KernelTime                00:00:37.767
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Windows\system32\atiumd64.dll - 
Win32 Start Address atiumd64!XdxInitXopAdapterServices (0x000007feddb3886c)
Stack Init fffff88002bacc70 Current fffff88002bac4d0
Base fffff88002bad000 Limit fffff88002ba6000 Call 0
Priority 10 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5
GetContextState failed, 0x80004001
Unable to get current machine context, HRESULT 0x80004001
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`02bac510 fffff800`034810f2 : fffffa80`0b7d2b50 fffffa80`0b7d2b50 00000000`00000000 fffffa80`00000001 : nt!KiSwapContext+0x7a
fffff880`02bac650 fffff800`0348390f : fffff8a0`0845cd10 fffff880`03dd6a7f fffff880`00000000 fffff8a0`12d13c90 : nt!KiCommitThreadWait+0x1d2
fffff880`02bac6e0 fffff800`0341ec0c : fffffa80`07fc4000 00000000`00000009 00000000`00009000 fffff8a0`12d13c00 : nt!KeWaitForSingleObject+0x19f
fffff880`02bac780 fffff800`034ad3c4 : 00000000`00000000 fffff680`002465d8 fffffa80`038347e0 00000000`000000a0 : nt!MiWaitForCollidedFaultComplete+0x110
fffff880`02bac7e0 fffff800`03499bff : 00000000`00000001 00000000`00000000 00000000`00000000 fffffa80`0546e6d8 : nt!MiResolveTransitionFault+0x544
fffff880`02bac870 fffff800`034890b9 : 00000000`08edf720 ffffffff`ffffffff fffff880`02bacb60 fffff800`0347e8af : nt!MiDispatchFault+0x95f
fffff880`02bac980 fffff800`03479fee : 00000000`00000001 00000000`48cbb3a0 fffffa80`075e7901 00000000`4ab16e48 : nt!MmAccessFault+0x359
fffff880`02bacae0 000007fe`ddb3798d : 000007fe`dd61b14d 00000000`4ab16e48 00000000`4ab16e48 00000000`08edf919 : nt!KiPageFault+0x16e (TrapFrame @ fffff880`02bacae0)
00000000`08edf848 000007fe`dd61b14d : 00000000`4ab16e48 00000000`4ab16e48 00000000`08edf919 00000000`07336fb8 : atiumd64!XdxInitXopAdapterServices+0x38f97d
00000000`08edf850 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : atiumd64!DllMain+0xa14d

Since neither !thread nor !process are actually giving the IRP list (not implemented???), I currently have WinDbg searching for all IRPs for the hung thread.

I also got my symbol resolve issue fixed, so I'll look at some of my recent dumps when I have the chance.

EDIT: Terminating a process shouldn't prevent one from finding the IRP which is blocking the process from actually terminating, right?

 

[TruePikachu]

Okay, just confirmed that the issue is 100% not caused by the graphics driver. Did a full wipe of the "old" one (which was the newest version installed, Nov 2014 IIRC), installed the original OEM driver set, switched the power profile to one which puts the display in sleep, went to bed. In the morning, Firefox had crashed and "HPOSD" had crashed (HP On-Screen Display, handles indication for e.g. brightness and volume when setting them). Unplugged my mouse and brought the screen back into sleep manually (PowerShell script), came back about 4 hours later and I had a BSoD, SYSTEM_SERVICE_EXCEPTION (looks like an attempt to both read and write at ff96f900`c0859370:

win32k!xxxWindowHitTest+0x51:
fffff960`001688b1 ff4308          inc     dword ptr [rbx+8] ds:002b:ff96f900`c0859378=????????
Last set context:
rax=fffff900c08595e0 rbx=ff96f900c0859370 rcx=fffff900c08595e0
rdx=0000002f00000069 rsi=0000000000000001 rdi=0000000000000000
rip=fffff960001688b1 rsp=fffff880093519a0 rbp=0000000000000000
 r8=0000000000000000  r9=0000000000000001 r10=fffff960001af964
r11=fffff88009351980 r12=0000000000000011 r13=00000000001efd20
r14=0000000000000010 r15=0000000074c92450
iopl=0         nv up ei ng nz na pe nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00210282

 

[TruePikachu]

Uninstalled both imdisk and MagicDisc, in the one night of testing, I didn't get a BSoD, but Firefox still crashed. Research online indicates that MagicDisc could possibly be the culprit. Not marking solved until I can confirm the BSoDs are gone, though.

EDIT: They aren't.

 

[TruePikachu]

Just a quick update over to here, I recently (2-3 days ago) removed AODDriver2.sys, and while I haven't yet done comprehensive testing, a couple oddities disappeared. Specifically, KeePass doesn't seem to ask for my password anymore when I change attached displays, and I no longer have to cancel the fingerprint reader when typing my Windows password after resume from standby.

I'm actually slightly surprised that nobody noticed or thought anything about AODDriver, given the number of hits I've gotten from sevenforums relating to BSoDs when I searched recently...

EDIT: Not the cause of issues.