HardHang every night, no BSOD, not fixed in startup, common 0xC000000D

MarioHadEnough · 16 Aug 2012

Dear supporters, enough is enough...

The system should be stable, but every morning I find it frozen -- only alternative is a forced shut down. I tried everything I was comfortable with, and more. BIOS update was a nightmare in itself... anyway - you have here attached all the information from event viewer (I think), and info about the system.
Here is the software I run:
- Acronis for backup
- Norton 360 firewall and antivirus
- Intel rapid storage
- dreamweaver, acrobat 7.0, photoshop elements
- Teamviewer 5
- Epson wireless printer (workforce 840)
- MS office, skype, Nvidia, Firefox, etc...

I updated everything under the sun (drivers, sw, etc). The hangs happen with or without system loaded with running software. Sometimes overnight, sometimes in the middle of the day, with no clear reason or rime. It's just happening... 6 critical events in the last 7 days, PLUS 50 errors in the same period.

This morning, I downloaded and run Malwarebytes - nothing found. Norton also says I have nothing.

Then I downloaded and run Windows Debugger, and got a report I copy here below, using a memory.dmp file created after the last hang last night:

X*X*X*X*X*X*X*X StartOfTheReport X*X*X*X*X*X*X*X

Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Loading Dump File [C:\Users\Mario Sarli\Desktop\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available
Symbol search path is: srv*c:\symbols\websymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: Windows 7 Kernel Version 7601 (Service Pack 1) MP (8 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
Machine Name:
Kernel base = 0xfffff800`0305e000 PsLoadedModuleList = 0xfffff800`032a2670
Debug session time: Fri Jul 20 08:07:01.913 2012 (UTC - 4:00)
System Uptime: 0 days 23:57:09.646
Loading Kernel Symbols
...............................................................
Loading User Symbols
Loading unloaded module list
******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* Use !analyze -v to get detailed debugging information.
BugCheck 9F, {3, fffffa800d56da10, fffff80000b9c3d8, fffffa800cffde10}
Probably caused by : pci.sys
Followup: MachineOwner

X*X*X*X*X*X*X*X EndOfTheReport X*X*X*X*X*X*X*X

The big question is: What do I do next? [[[Fix pci.sys? (how?) Change something? What?]]]

Looking forward to having a stable system... I really need one

.
Best,
Mario

FredeGail · 20 Aug 2012

Hello Mario and welcome to Sevenforums.

BSOD Analyse

Suspecting Memory

Looking at your bugcheck it's likely that a memory corruption event was triggered. Though a driver could be causing the memory to be borked, but we're usually suggesting a memory test before moving forward. Do a scan with Memtest86+. Memtest is a scanner that'll check your sticks for errors.

Warning

Note that the below instructions may break your WARRANTY rules. If you're unsure check manuals, separated warranty papers, stickers on computer for secure permission.

To ensure that we'll know if it's the slots on the motherboard that's broken, or the sticks itself - we have a little procedure we'll recommend. Remove 1 stick, scan the other with Memtest in the current slot. After 7 passes, move the stick to another slot and scan, and so on and so forth until you've scanned all the sticks, and all the slots, one by one.

Memtest86+ - Advanced Memory Diagnostic Tool

Best Regards,
Frederik.

MarioHadEnough · 20 Aug 2012

Ok -- THANK YOU for the answer, FredeGail! I'm downloading and running the memtests (7 times, as indicated). Back when I bout the parts and assembled this machine, I did a memory test with memtest... so I expect the memory sticks to be ok, but I'll test it anyway. If I find anything wrong, I'll come back here immediately.

After the test, if the memory is fine, I'll leave the computer running overnight and will post the new crash report in the morning tomorrow, or as soon as it happens
Thank you once again,
Mario

FredeGail · 20 Aug 2012

You're welcome!

Best Regards,
Frederik.

Vir Gnarus · 20 Aug 2012

While you should go ahead and do the memtest, I want to add something which I found odd. You said you went heavy on updating drivers, but this latest crashdump shows your NIC drivers being from way back in January 2011. I also find it strange that the crashdump timestamp on the file itself is 07-25-2012, but the debug session time listed inside the crashdump is July 20, 2012. Very weird. Either way, see what you can do about your Realtek NIC drivers associated with your Gigabyte motherboard.

Also, do you have your power options set to put your system in any sort of sleep/power down state overnight? The system may be going into a type of hybrid sleep state that is unsupported by your motherboard/NIC card.

Analysts:

The latest crashdump given showed an issue that dealt with a device that was stuck for a long period of time performing a power IRP. This is basically I/O that's done with the device to change its power state. I went ahead and dumped the related power IRP to see what was up:

Code:

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_POWER_STATE_FAILURE (9f)
A driver has failed to complete a power IRP within a specific time (usually 10 minutes).
Arguments:
Arg1: 0000000000000003, A device object has been blocking an Irp for too long a time
Arg2: fffffa800d56da10, Physical Device Object of the stack
Arg3: fffff80000b9c3d8, nt!TRIAGE_9F_POWER on Win7, otherwise the Functional Device Object of the stack
Arg4: fffffa800cffde10, The blocked IRP

Debugging Details:
------------------

TRIAGER: Could not open triage file : C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\triage\modclass.ini, error 2

DRVPOWERSTATE_SUBCODE:  3

IMAGE_NAME:  pci.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  4ce7928f

MODULE_NAME: pci

FAULTING_MODULE: fffff88000f70000 pci

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

BUGCHECK_STR:  0x9F

PROCESS_NAME:  System

CURRENT_IRQL:  2

STACK_TEXT:  
fffff800`00b9c388 fffff800`031476c2 : 00000000`0000009f 00000000`00000003 fffffa80`0d56da10 fffff800`00b9c3d8 : nt!KeBugCheckEx
fffff800`00b9c390 fffff800`030e7e3c : fffff800`00b9c4c0 fffff800`00b9c4c0 00000000`00000000 00000000`00000001 : nt! ?? ::FNODOBFM::`string'+0x34050
fffff800`00b9c430 fffff800`030e7cd6 : fffffa80`0c7f7c58 fffffa80`0c7f7c58 00000000`00000000 00000000`00000000 : nt!KiProcessTimerDpcTable+0x6c
fffff800`00b9c4a0 fffff800`030e7bbe : 000000c8`c4dfd2d6 fffff800`00b9cb18 00000000`005457d1 fffff800`03253ca8 : nt!KiProcessExpiredTimerList+0xc6
fffff800`00b9caf0 fffff800`030e79a7 : 00000042`82a548c2 00000042`005457d1 00000042`82a548f6 00000000`000000d1 : nt!KiTimerExpiration+0x1be
fffff800`00b9cb90 fffff800`030d4eca : fffff800`0324fe80 fffff800`0325dcc0 00000000`00000002 fffff880`00000000 : nt!KiRetireDpcList+0x277
fffff800`00b9cc40 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000 fffff800`00b9cc00 00000000`00000000 : nt!KiIdleLoop+0x5a


STACK_COMMAND:  kb

FOLLOWUP_NAME:  MachineOwner

FAILURE_BUCKET_ID:  X64_0x9F_3_Rt64win7_IMAGE_pci.sys

BUCKET_ID:  X64_0x9F_3_Rt64win7_IMAGE_pci.sys

Followup: MachineOwner
---------
0: kd> !irp fffffa800cffde10 1
Irp is active with 3 stacks 2 is current (= 0xfffffa800cffdf28)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
Flags = 00000000
ThreadListEntry.Flink = fffffa800cffde30
ThreadListEntry.Blink = fffffa800cffde30
IoStatus.Status = c00000bb
IoStatus.Information = 00000000
RequestorMode = 00000000
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 00000000
UserEvent = 00000000
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 00000000
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = 00000000   
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = fffffa800cffde88
Tail.Overlay.Thread = 00000000
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = fffffa80146a5968
Tail.Overlay.ListEntry.Blink = fffff80003284b00
Tail.Overlay.CurrentStackLocation = fffffa800cffdf28
Tail.Overlay.OriginalFileObject = 00000000
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000    

            Args: 00000000 00000000 00000000 00000000
>[ 16, 3]   0 e1 fffffa8010d6e050 00000000 fffff80003321200-fffffa8012802be0 Success Error Cancel pending
           \Driver\RTL8167    nt!PopSystemIrpCompletion
            Args: 00016600 00000000 00000006 00000006
 [  0, 0]   0  0 00000000 00000000 00000000-fffffa8012802be0    

            Args: 00000000 00000000 00000000 00000000
0: kd> !error c00000bb
Error code: (NTSTATUS) 0xc00000bb (3221225659) - The request is not supported.

The IRP showed the responsible driver object and device object, that being related to Realtek NICs, showing up as the one hanging the IRP. The status of the IRP shows what appears to be that the IRP is responding that it does not support that kind of request (power IRP). What exactly is the IRP? It's a power IRP (Major function code 16), that's trying to do a remove device (Minor function code 3).

Now, correct me if I'm wrong, but I don't know any moment when a network controller on a motherboard or a PCI NIC card would ever want to be removed. So something tells me there's some sort of power state on the system that's going on that this device is incompatible with. I can try and verify this speculation by looking further in, but I'm not sure I have time at the moment. Suffice to say, something is trying to force this network controller device off Windows, and it's simply not accepting the request to do so, so it hangs up on that, never finishes, and the system crashes. That's at least my interpretation of it.

Btw, here's how I got the dirt on the driver object and its associated driver:

Code:

0: kd> !irp fffffa800cffde10 1
Irp is active with 3 stacks 2 is current (= 0xfffffa800cffdf28)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
Flags = 00000000
ThreadListEntry.Flink = fffffa800cffde30
ThreadListEntry.Blink = fffffa800cffde30
IoStatus.Status = c00000bb
IoStatus.Information = 00000000
RequestorMode = 00000000
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 00000000
UserEvent = 00000000
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 00000000
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = 00000000   
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = fffffa800cffde88
Tail.Overlay.Thread = 00000000
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = fffffa80146a5968
Tail.Overlay.ListEntry.Blink = fffff80003284b00
Tail.Overlay.CurrentStackLocation = fffffa800cffdf28
Tail.Overlay.OriginalFileObject = 00000000
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000    

            Args: 00000000 00000000 00000000 00000000
>[ 16, 3]   0 e1 fffffa8010d6e050 00000000 fffff80003321200-fffffa8012802be0 Success Error Cancel pending
           \Driver\RTL8167    nt!PopSystemIrpCompletion
            Args: 00016600 00000000 00000006 00000006
 [  0, 0]   0  0 00000000 00000000 00000000-fffffa8012802be0    

            Args: 00000000 00000000 00000000 00000000

0: kd> !devobj fffffa8010d6e050
Device object (fffffa8010d6e050) is for:
 InfoMask field not found for _OBJECT_HEADER at fffffa8010d6e020
 \Driver\RTL8167 DriverObject fffffa800fb15e70
Current Irp 00000000 RefCount 0 Type 00000017 Flags 00002050
DevExt fffffa8010d6e1a0 DevObjExt fffffa8010d6f8e0 
ExtensionFlags (0x00000800)  
                             Unknown flags 0x00000800
AttachedTo (Lower) fffffa800d56da10 \Driver\pci
Device queue is not busy.

0: kd> !drvobj fffffa800fb15e70 7
fffff80003284fb0: Unable to get value of ObpRootDirectoryObject
fffff80003284fb0: Unable to get value of ObpRootDirectoryObject
Driver object (fffffa800fb15e70) is for:
 \Driver\RTL8167
Driver Extension List: (id , addr)

Device Object list:
fffffa8010d6e050  

DriverEntry:   fffff880051921dc    Rt64win7
DriverStartIo: 00000000    
DriverUnload:  fffff880018cbcb0    ndis!ndisMUnloadEx
AddDevice:     00000000    

Dispatch routines:
[00] IRP_MJ_CREATE                      fffff88001823d20    ndis!ndisCreateIrpHandler
[01] IRP_MJ_CREATE_NAMED_PIPE           fffff880018ba490    ndis!ndisDummyIrpHandler
[02] IRP_MJ_CLOSE                       fffff88001823560    ndis!ndisCloseIrpHandler
[03] IRP_MJ_READ                        fffff880018ba490    ndis!ndisDummyIrpHandler
[04] IRP_MJ_WRITE                       fffff880018ba490    ndis!ndisDummyIrpHandler
[05] IRP_MJ_QUERY_INFORMATION           fffff880018ba490    ndis!ndisDummyIrpHandler
[06] IRP_MJ_SET_INFORMATION             fffff880018ba490    ndis!ndisDummyIrpHandler
[07] IRP_MJ_QUERY_EA                    fffff880018ba490    ndis!ndisDummyIrpHandler
[08] IRP_MJ_SET_EA                      fffff880018ba490    ndis!ndisDummyIrpHandler
[09] IRP_MJ_FLUSH_BUFFERS               fffff880018ba490    ndis!ndisDummyIrpHandler
[0a] IRP_MJ_QUERY_VOLUME_INFORMATION    fffff880018ba490    ndis!ndisDummyIrpHandler
[0b] IRP_MJ_SET_VOLUME_INFORMATION      fffff880018ba490    ndis!ndisDummyIrpHandler
[0c] IRP_MJ_DIRECTORY_CONTROL           fffff880018ba490    ndis!ndisDummyIrpHandler
[0d] IRP_MJ_FILE_SYSTEM_CONTROL         fffff880018ba490    ndis!ndisDummyIrpHandler
[0e] IRP_MJ_DEVICE_CONTROL              fffff880018ba020    ndis!ndisDeviceControlIrpHandler
[0f] IRP_MJ_INTERNAL_DEVICE_CONTROL     fffff880018ba490    ndis!ndisDummyIrpHandler
[10] IRP_MJ_SHUTDOWN                    fffff880018ba490    ndis!ndisDummyIrpHandler
[11] IRP_MJ_LOCK_CONTROL                fffff880018ba490    ndis!ndisDummyIrpHandler
[12] IRP_MJ_CLEANUP                     fffff880018ba490    ndis!ndisDummyIrpHandler
[13] IRP_MJ_CREATE_MAILSLOT             fffff880018ba490    ndis!ndisDummyIrpHandler
[14] IRP_MJ_QUERY_SECURITY              fffff880018ba490    ndis!ndisDummyIrpHandler
[15] IRP_MJ_SET_SECURITY                fffff880018ba490    ndis!ndisDummyIrpHandler
[16] IRP_MJ_POWER                       fffff8800186df80    ndis!ndisPowerDispatch
[17] IRP_MJ_SYSTEM_CONTROL              fffff8800188eee0    ndis!ndisWMIDispatch
[18] IRP_MJ_DEVICE_CHANGE               fffff880018ba490    ndis!ndisDummyIrpHandler
[19] IRP_MJ_QUERY_QUOTA                 fffff880018ba490    ndis!ndisDummyIrpHandler
[1a] IRP_MJ_SET_QUOTA                   fffff880018ba490    ndis!ndisDummyIrpHandler
[1b] IRP_MJ_PNP                         fffff880018c6110    ndis!ndisPnPDispatch

0: kd> lmvm Rt64win7
start             end                 module name
fffff880`0512e000 fffff880`05195000   Rt64win7 T (no symbols)           
    Loaded symbol image file: Rt64win7.sys
    Image path: \SystemRoot\system32\DRIVERS\Rt64win7.sys
    Image name: Rt64win7.sys
    Timestamp:        Thu Jan 13 06:57:19 2011 (4D2EE89F)
    CheckSum:         0006BB92
    ImageSize:        00067000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

MarioHadEnough · 20 Aug 2012

Hi All,
All memtests completed (each stick and each slot) and found nothing wrong. I'll leave the computer running overnight and will post the new crash report / memory dump file ... in the morning tomorrow, or as soon as it happens.

Vir Gnarus -- GREAT reply! Let me address a few parts of your post:
-- Drivers were updated by going into Device Manager, right click, and (ask windows to) search for new drivers. All resulted in the same message saying the drivers used are the latest. Gigabyte website had the same result.
-- If there is a better way to search for driver updates, please let me know.
-- Before posting, I downloaded and run the SF Diag Tool.exe, and uploaded the zipped file. I didn't check dates.
-- Is there a way to cleanup all crash-logs, so I can make sure I'm uploading the newest files next time?
-- I'll go back to gigabyte website and try again to find newer drivers for the network card.
-- Mobo supports sleep and hibernation. I used sleep and it crashed. I used 'always on' and it crashes just as much... but this way I can see the screen saver frozen. Don't know about hybrid states.
-- IF we are sure the problem is the network card (100% sure).... can I buy another card and solve this problem? If so, which one? (then you are all invited to a "destroy this ^&$#$% card" party...)

Thanks again,
Mario

Vir Gnarus · 21 Aug 2012

The best way to update drivers for anything is to do it manually: if you are using an OEM (Dell, HP, etc.), download/install from their website; if custom-built, download and install from each manufacturer (Gigabyte for your motherboard). Windows driver repository is updated very, very infrequently, so it only should be used to retrieve initial drivers not already provided with Windows installation in order to get things going at first before you go and update them yourself. This also pretty much applies to any driver CDs that come with hardware you purchase: toss them out and make your own updated driver CD(s).

Yes, you can clean up minidumps by cleaning out C:/Windows/Minidump.

Make sure your BIOS is also updated, as this can play a role in the whole sleep/hibernate compatibility. Also, are you using a network card, or just the network connection built-in to the motherboard?

MarioHadEnough · 22 Aug 2012

Hello Vir Gnarus... and team,
Yesterday I went to Mobo website and downloaded all applicable drivers, installed them... there was a network driver update issued just a few days ago - interesting!)... These updates included intel drivers and sound drivers as well.

This computer was assembled here (custom built) after multiple visits to Microcenter. I am using the network connection built-in the mobo, and my comment about a nework card was mistaken - I confused the Nvidia card with network card.

So, Yesterday I cleared all dmp files I found, from folders like minidump and firefox and others - trying to start fresh. The BIOS was updated the week before my first post, and caused extensive headaches -- until I found out the BIOS update had reset my drivers to IDE... upon changing them back to XHD (or whatever that acronym is for RAID enabled), all functioned normally, and crashed normally as it was the norm.

Since the drivers update yesterday, I left the system running overnight, and no crashes... (I'm liking it)... today I'll use it normally, and tonight I'll leave all regular programs I need running - Let's see what happens.

I'll post an update here ASAICA (((As Soon As It Crashes Again))). For now... extremely sincerely, THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU! THANK YOU!

No, seriously -- I really mean.... THANK YOU!
All the best,
Mario

Vir Gnarus · 22 Aug 2012

Sounds good, hope this all works out for ya. If you end up finding that this did take care of things, go ahead and mark this thread as solved for future reference.

MarioHadEnough · 23 Aug 2012

Good sign - no errors (no crashes either) in the last 24h.... I haven't seem an Event Viewer screen with so many zeros in a long time.
I'll come back and mark the thread as "closed" if we get to more than 7 days with no errors.
Thank you all once again.
Mario