Several BSODs occurred last night, now seeing major performance drop

[KEGong]

System Specs

• Windows 7 Ultimate x64, Upgrade from a Windows 2K Install
• AMD Phenom II X4 955 @ 3.20GHz
• 8GB RAM
• AMD Radeon HD 6770
• 1TB Seagate Hard Drive
• Custom build completed in November 2011

Last night, I encountered my first BSOD since this build was complete. I had a series of maybe about half a dozen occur throughout the night, with each BSOD after the first occurring about 5 or 10 minutes after loading to Desktop. Safe Mode worked fine.

I ran Windows Memory Diagnostic overnight last night (6 passes, full tests) with no errors reported. Today, I noticed no further BSODs, but I found computer performance was noticeably degraded. Applications take a bit longer to load (sometimes a minute or two more), and games have very noticeable FPS drops and audio skips.

Reports attached.

Any ideas, guys? Thank you for your time!

 

[JMH]

These crashes were caused by memory corruption/exception (probably a driver).
Please run these two tests to verify your memory and find which driver is causing the problem.

* If you are overclocking anything reset to default before running these tests.
In other words STOP!!!

* If you have Raid update its Driver.



Memtest.

*Download a copy of Memtest86 and burn the ISO to a CD using Iso Recorder or another ISO burning program. Memtest86+ - Advanced Memory Diagnostic Tool

*Boot from the CD, and leave it running for at least 5 or 6 passes.

Just remember, any time Memtest reports errors, it can be either bad RAM or a bad motherboard slot.

Test the sticks individually, and if you find a good one, test it in all slots.

Any errors are indicative of a memory problem.

If a known good stick fails in a motherboard slot it is probably the slot.

http://www.sevenforums.com/tutorials/105647-ram-test-memtest86.html



Driver verifier

Using Driver Verifier is an iffy proposition. Most times it'll crash and it'll tell you what the driver is. But sometimes it'll crash and won't tell you the driver. Other times it'll crash before you can log in to Windows. If you can't get to Safe Mode, then you'll have to resort to offline editing of the registry to disable Driver Verifier.

So, I'd suggest that you first backup your stuff and then make sure you've got access to another computer so you can contact us if problems arise. Then make a System Restore point (so you can restore the system using the Vista/Win7 Startup Repair feature).

Then, here's the procedure:
- Go to Start and type in "verifier" (without the quotes) and press Enter
- Select "Create custom settings (for code developers)" and click "Next"
- Select "Select individual settings from a full list" and click "Next"
- Select everything EXCEPT FOR "Special Pool", "Force Pending I/O Requests" and "Low Resource Simulation" and click "Next"
- Select "Select driver names from a list" and click "Next"
Then select all drivers NOT provided by Microsoft and click "Next"
- Select "Finish" on the next page.

Reboot the system and wait for it to crash to the Blue Screen.
Continue to use your system normally, and if you know what causes the crash, do that repeatedly.
The objective here is to get the system to crash because Driver Verifier is stressing the drivers out.
If it doesn't crash for you, then let it run for at least 36 hours of continuous operation.

Reboot into Windows (after the crash) and turn off Driver Verifier by going back in and selecting "Delete existing settings" on the first page, then locate and zip up the memory dump file and upload it with your next post.

If you can't get into Windows because it crashes too soon, try it in Safe Mode.
If you can't get into Safe Mode, try using System Restore from your installation DVD to set the system back to the previous restore point that you created.

 

[zigzag3143]

7 of 11 crashes point directly to memory management.

Please follow JMH'S suggestions.

It is probably a driver but windows diagnostics doesnt stress the memory enough so you will have to run memtest as well.

 

[KEGong]

Thank you both for your prompt responses. I'll run memtest tonight to be sure.

 

[zigzag3143]

Thank you both for your prompt responses. I'll run memtest tonight to be sure.

Keep us in the loop and good luck.

 

[KEGong]

The BSODs have not happened since I started running Driver Verifier a few hours ago, and the performance issues have also mysteriously gone away. I'll leave Verifier running for the recommended 36 hours, and report back my findings.

 

[KEGong]

No BSODs, no performance issues, I stopped running Driver Verifier. Everything seems to be back to normal with no further intervention. Curious.

 

[KEGong]

Hello again everybody.

BSODs started again tonight. I woke the computer up from Sleep after being out all day, but the display remained blank. Restarted computer, got the first BSOD. After it rebooted from that crash, it crashed again with our familiar memory management code. Running Driver Verifier again now. Doesn't it sound like it's a display driver problem? That's the only driver I can think of that I've updated within the past month.

Please let me know if you would like me to post new dump files. I'll otherwise keep you all updated.

 

[KEGong]

Well, that was fast. After I restarted into normal boot with Verifier running, the computer immediately crashed, and restarted. Do you just want me to post the raw dump file, or should I run that Dump and System File Collection process again? Thanks.

 

[KEGong]

Hello again, attached are the minidump files from all the BSODs tonight, including ones where Verifier was running. Sorry for the post spam, I re-read JMH's instructions just now, and realized y'all were probably waiting for these.

 

[writhziden]

Please remove either: AMD Overdrive or AMD Vision Engine Control Centre. They use a driver called AODDriver2.sys that is known to overclock the graphics processor (GPU) and cause crashes.

AODDriver2	fffff880`089b2000	fffff880`089e3000	Mon Dec 19 00:30:03 2011 (4eeee7fb)	00018588		AODDriver2.sys

All verified crashes point to system files; this usually indicates Windows file corruption or a hardware fault. I also see a crash indicating hard disk problems.

• If you are overclocking any hardware, please stop.
  
  
• Run Disk Check with both boxes checked for all HDDs and with Automatically fix file system errors checked for all SSDs. Post back your logs for the checks after finding them using http://www.sevenforums.com/tutorials/96938-check-disk-chkdsk-read-event-viewer-log.html.
  For any drives that do not give the message:
  Windows has checked the file system and found no problems
  run disk check again as above. In other words, if it says:
  Windows has made corrections to the file system
  after running the disk check, run the disk check again.
  
  
• Run http://www.sevenforums.com/tutorials/1538-sfc-scannow-command-system-file-checker.html up to three times to fix all errors with a restart in between each. Post back if it continues to show errors after a fourth run or if the first run comes back with no integrity violations.
  
  
  
• Monitor temperatures during the following tests.
  
  Use the following programs to monitor the temperatures.​
  • Real Temp is a good CPU temperature monitor.
  • Speccy - System Information - Free Download will monitor all hardware temperatures.
  • HWiNFO, HWiNFO32 & HWiNFO64 - Hardware Information and Analysis Tools can be inaccurate for CPU temperatures, but is a good program for GPU temperature monitoring.
  • 
    Use FurMark: VGA Stress Test, Graphics Card and GPU Stability Test, Burn-in Test, OpenGL Benchmark and GPU Temperature | oZone3D.Net to test the graphics card GPU. Then use the |MG| Video Memory Stress Test 1.7.116 Download to test your graphics card memory. Let the memory test run for at least seven passes; the more the better.
    
    
  • Run http://www.sevenforums.com/tutorials/100352-hardware-stress-test-prime95.html to determine any hardware problems. Run all three tests for a few hours each. If you get errors, stop the test and post back here.
    
    
  • Follow the steps for doing a CPU stress test using IntelBurnTest
  
  
• Run the boot version of Memtest86+ paying close attention to Parts 2 and 3 of the tutorial. Also, in case Memtest86+ misses anything and comes up with no errors, run the extended version of the Windows Memory Diagnostics Tool for at least five passes. These you may want to run overnight since they take a long time to complete (run them an hour before bed each of the next two nights and check before going to sleep that they are still running).
  
  If you swap any memory components, follow these steps for ESD safety:
  1. Shut down and turn off your computer.
  2. Unplug all power supplies to the computer (AC Power then battery for laptops, AC power for desktops)
  3. Hold down the power button for 30 seconds to close the circuit and ensure all power drains from components.
  4. Make sure you are grounded by using proper grounding techniques, i.e. work on an anti-static workbench, anti-static desk, or an anti-static pad. Hold something metallic while touching it to the anti-static surface, or use an anti-static wristband to attach to the anti-static material while working.
  Once these steps have been followed, it is safe to remove and replace components within your computer.

 

[KEGong]

Hi Writhzeden, thanks for the response.

I'm now running CHKDSK again (I did it initially when I first saw this problem happening earlier this month) in case it missed anything the first time.

I also ran the boot version of Memtest86+ for 7 passes last week, with no problems detected. I posted about this to the thread the morning after about that result, but I guess I clicked the wrong post button or something, because the post isn't showing up in the thread (forum newbie problems, I guess). With that plus a few passes of the Windows Memory Diagnostic last, I was pretty sure it was not a RAM module problem then... I'll run Memtest again tonight to be sure, of course, but I figure more data is always good for analysis.

Since I don't use any of the AMD overclocking software, and since that's likely to have been updated in the Catalyst patch that came earlier this month, I'm hoping that getting rid of that will make a difference.

Next up after CHKDSK today will be SFC and then that hardware stress testing. I will report back with any anomalies.

 

[KEGong]

OK so, I thought this was notable: running SFC and I got another BSOD as it was doing it's thing. I got a CACHE MANAGER error. Here is another set of dumps, all from today. CHKDSK found no problems on my system drive, but now I'm running it on my other partitions too. I'll be out the rest of the day, so we'll see what happens when I get back.

 

[KEGong]

Got another one: running CHKDSK on my D partition from Safe Mode. I'm turning the computer off for the rest of the day while I'm out rather than letting it run tests without me here.

 

[writhziden]

Still see this on your system:

AODDriver2	fffff880`063bc000	fffff880`063ed000	Mon Dec 19 00:30:03 2011 (4eeee7fb)	00018588		AODDriver2.sys

Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [D:\Kingston\BSODDmpFiles\KEGong\minidump2\032512-17737-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: SRV*C:\SymCache*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Kernel Version 7601 (Service Pack 1) MP (4 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7601.17727.amd64fre.win7sp1_gdr.111118-2330
Machine Name:
Kernel base = 0xfffff800`02e66000 PsLoadedModuleList = 0xfffff800`030aa650
Debug session time: Sun Mar 25 12:50:34.033 2012 (UTC - 6:00)
System Uptime: 0 days 0:01:57.220
Loading Kernel Symbols
...............................................................
................................................................
..................................
Loading User Symbols
Loading unloaded module list
....
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck A, {fffffa8009e08010, 2, 1, fffff80002f994a6}

Probably caused by : memory_corruption ( nt!MiReleaseConfirmedPageFileSpace+86 )

Followup: MachineOwner
---------

1: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: fffffa8009e08010, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, bitfield :
	bit 0 : value 0 = read operation, 1 = write operation
	bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: fffff80002f994a6, address which referenced memory

Debugging Details:
------------------


WRITE_ADDRESS: GetPointerFromAddress: unable to read from fffff80003114100
 fffffa8009e08010 

CURRENT_IRQL:  2

FAULTING_IP: 
nt!MiReleaseConfirmedPageFileSpace+86
fffff800`02f994a6 480fb328        btr     qword ptr [rax],rbp

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  VERIFIER_ENABLED_VISTA_MINIDUMP

BUGCHECK_STR:  0xA

PROCESS_NAME:  WerFault.exe

TRAP_FRAME:  fffff8800afb9780 -- (.trap 0xfffff8800afb9780)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffffa8009cc8010 rbx=0000000000000000 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff80002f994a6 rsp=fffff8800afb9910 rbp=0000000000a00000
 r8=fffff8800afb9940  r9=0000000fffffffff r10=fffffa800b109728
r11=0000058000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
nt!MiReleaseConfirmedPageFileSpace+0x86:
fffff800`02f994a6 480fb328        btr     qword ptr [rax],rbp ds:9998:fffffa80`09cc8010=0000000000000001
Resetting default scope

LAST_CONTROL_TRANSFER:  from fffff80002ee22e9 to fffff80002ee2d40

STACK_TEXT:  
fffff880`0afb9638 fffff800`02ee22e9 : 00000000`0000000a fffffa80`09e08010 00000000`00000002 00000000`00000001 : nt!KeBugCheckEx
fffff880`0afb9640 fffff800`02ee0f60 : 00000000`0036b000 00000000`76f9e000 00000000`76af2000 00000000`00000000 : nt!KiBugCheckDispatch+0x69
fffff880`0afb9780 fffff800`02f994a6 : 00000000`00000001 00000000`00000000 fffffa80`09bf0c30 fffffa80`09bf0c30 : nt!KiPageFault+0x260
fffff880`0afb9910 fffff800`02f465c3 : 00a00000`00000000 00000000`00000000 00000000`00000003 00000000`0000001c : nt!MiReleaseConfirmedPageFileSpace+0x86
fffff880`0afb9990 fffff800`02eb2cf4 : fffffa80`0b109728 00000000`00000001 00000000`00000000 fffffa80`0000001c : nt! ?? ::FNODOBFM::`string'+0x210fa
fffff880`0afb9a10 fffff800`02eb394e : 00000000`00000001 fffffa80`0b109390 fffffa80`0b109390 00000000`00000001 : nt!MiDeleteAllHashMappings+0xe4
fffff880`0afb9a50 fffff800`031b781a : fffff8a0`0a9ba370 00000000`00000001 00000000`00000000 fffffa80`07bba780 : nt!MmCleanProcessAddressSpace+0xa2
fffff880`0afb9aa0 fffff800`031990d8 : 00000000`c0000005 00000000`00000001 000007ff`fffdd000 00000000`00000000 : nt!PspExitThread+0x56a
fffff880`0afb9ba0 fffff800`02ee1fd3 : fffffa80`0b109390 00000000`c0000005 fffffa80`07bba780 00000000`76f2c521 : nt!NtTerminateProcess+0x138
fffff880`0afb9c20 00000000`76f515da : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`001bf7a8 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x76f515da


STACK_COMMAND:  kb

FOLLOWUP_IP: 
nt!MiReleaseConfirmedPageFileSpace+86
fffff800`02f994a6 480fb328        btr     qword ptr [rax],rbp

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  nt!MiReleaseConfirmedPageFileSpace+86

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: nt

DEBUG_FLR_IMAGE_TIMESTAMP:  4ec79dd2

IMAGE_NAME:  memory_corruption

FAILURE_BUCKET_ID:  X64_0xA_VRF_nt!MiReleaseConfirmedPageFileSpace+86

BUCKET_ID:  X64_0xA_VRF_nt!MiReleaseConfirmedPageFileSpace+86

Followup: MachineOwner
---------

Pointing to the RAM. Still recommend running Memtest86+ again at your earliest convenience.

Also, I would like to check your RAM compatibility:

• Download and install CPU-Z and Upload screenshots of the CPU, Mainboard, Memory, and SPD tabs. In the SPD tab, upload an image of each slot.

 

[KEGong]

Just got home about 5 minutes ago. Will removing AMD Catalyst remove the driver? I'm having some questions with that since that's the only AMD software I have on my computer showing up in the uninstall list, and it's currently not stable enough for me to actually boot in long enough to uninstall it. Is there a workaround for removing that particular driver?

I will run Memtest overnight again since this seems to be a recurring issue. I need to go to bed soon, big work event tomorrow.

 

[KEGong]

So, as I was getting ready for bed, I noticed that Memtest this time started returning memory errors. I wonder why it wasn't caught the first night I did this earlier this month. Anyways, I'll test individual sticks and slot combinations either tomorrow or Tuesday.

 

[writhziden]

Memtest86+ sometimes needs to be run multiple times to catch errors. When RAM fails, it can be very finicky; sometimes it works fine, and other times it causes instability. That is just the nature of electronics. If you have even one error in Memtest86+, that is one too many and means a bad module is in the system and needs to be replaced.

 

[KEGong]

Thank you all for your advice, expertise, and analysis.

Alright, tested each stick individually in different slots. I have one borked module. I'm going to buy a replacement 8GB kit soon. Is there anything else folks think I need to check on before closing this case??

 

[writhziden]

Nothing I see. It is highly likely that it was just the bad module causing the problems. Let us know when you get the new kit how the system responds.