BSOD in Wdf01000!FxIoQueue when computer goes to sleep (I think)

[sergeyn]

Hi.
Please help me to figure out what is causing the bsod. Even after motherboard and cpu and memory upgrade I still have same bsod (concluded based on memory dumps). This bugs me almost a year now. Posted the dump analysis at the bottom, but please let me know anything else you might need.

Thanks a lot!

7: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 000000000057ffeb, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff88000e3c899, address which referenced memory

Debugging Details:
------------------


READ_ADDRESS: GetPointerFromAddress: unable to read from fffff800036bb100
000000000057ffeb

CURRENT_IRQL: 2

FAULTING_IP:
Wdf01000!FxIoQueue:rocessPowerEvents+169
fffff880`00e3c899 483950e8 cmp qword ptr [rax-18h],rdx

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

BUGCHECK_STR: 0xD1

PROCESS_NAME: System

TRAP_FRAME: fffff88018e753b0 -- (.trap 0xfffff88018e753b0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000580003 rbx=0000000000000000 rcx=fffffa801bb713c0
rdx=fffffa801f5f5a38 rsi=0000000000000000 rdi=0000000000000000
rip=fffff88000e3c899 rsp=fffff88018e75540 rbp=fffff88018e755b8
r8=0000000000000000 r9=0000000000000000 r10=0000000000000000
r11=fffff88003374180 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na pe cy
Wdf01000!FxIoQueue:rocessPowerEvents+0x169:
fffff880`00e3c899 483950e8 cmp qword ptr [rax-18h],rdx ds:00000000`0057ffeb=????????????????
Resetting default scope

LAST_CONTROL_TRANSFER: from fffff800034831a9 to fffff80003483c00

STACK_TEXT:
fffff880`18e75268 fffff800`034831a9 : 00000000`0000000a 00000000`0057ffeb 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
fffff880`18e75270 fffff800`03481e20 : fffffa80`1e28e300 fffffa80`1bb13930 00000000`00000000 00000000`00000000 : nt!KiBugCheckDispatch+0x69
fffff880`18e753b0 fffff880`00e3c899 : fffffa80`00000000 fffffa80`188b7148 00000000`00000002 00000000`00000000 : nt!KiPageFault+0x260
fffff880`18e75540 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : Wdf01000!FxIoQueue:rocessPowerEvents+0x169


STACK_COMMAND: kb

FOLLOWUP_IP:
Wdf01000!FxIoQueue:rocessPowerEvents+169
fffff880`00e3c899 483950e8 cmp qword ptr [rax-18h],rdx

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: Wdf01000!FxIoQueue:rocessPowerEvents+169

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: Wdf01000

IMAGE_NAME: Wdf01000.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 5010aa89

FAILURE_BUCKET_ID: X64_0xD1_Wdf01000!FxIoQueue:rocessPowerEvents+169

BUCKET_ID: X64_0xD1_Wdf01000!FxIoQueue:rocessPowerEvents+169

Followup: MachineOwner
---------

 

[Vir Gnarus]

Looks like a driver is not performing a power IRP properly, and it appears it involves Remote Desktop USB hub operations. I see you have both a VM host application and Logmein present. I'm thinking these and/or perhaps other software I'm not aware of involving Remote Desktop is conflicting on each other. I recommend updating whatever related software you have, and perhaps even uninstall one or more of them to discover which one's causing the problems.

I'm also worried that there's a WHEA crash for one of the dump files. It's telling that during an undetermined write operation involving the CPU's L0 Data cache there was an issue. This happened while running some benchmark software (benchmark.exe). Typically this is a problem with an actual piece of hardware, but VM software can also trigger this, as well as software that comes with your motherboard. I recommend removing any software that came with your motherboard and only leaving drivers.

Right now this is all based on an educated guess from what I've discovered so far. If you want something better, you're going to either turn on Driver Verifier and provide me crashdumps triggered by it, or send me a kernel dump (MEMORY.DMP in Windows directory) by zipping it up and uploading to a site like Mirrorcreator.com.

Analysts:

Two crashdumps identical, in that it appears when attempting to perform some Power IRP using the WDF, it's bugging out. I can't tell what device/driver is involved directly without access to a kernel dump. However, I can get a general idea from the raw stack, provided it wasn't leftovers from some bygone operation.

7: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 000000000057ffeb, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff88000e3c899, address which referenced memory

Debugging Details:
------------------

TRIAGER: Could not open triage file : C:\Program Files (x86)\Windows Kits\8.0\Debuggers\x64\triage\modclass.ini, error 2

READ_ADDRESS: GetPointerFromAddress: unable to read from fffff800036bb100
GetUlongFromAddress: unable to read from fffff800036bb1c0
 000000000057ffeb Nonpaged pool

CURRENT_IRQL:  2

FAULTING_IP: 
Wdf01000!FxIoQueue::ProcessPowerEvents+169
fffff880`00e3c899 483950e8        cmp     qword ptr [rax-18h],rdx

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

BUGCHECK_STR:  0xD1

PROCESS_NAME:  System

TRAP_FRAME:  fffff88018e753b0 -- (.trap 0xfffff88018e753b0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000580003 rbx=0000000000000000 rcx=fffffa801bb713c0
rdx=fffffa801f5f5a38 rsi=0000000000000000 rdi=0000000000000000
rip=fffff88000e3c899 rsp=fffff88018e75540 rbp=fffff88018e755b8
 r8=0000000000000000  r9=0000000000000000 r10=0000000000000000
r11=fffff88003374180 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl nz na pe cy
Wdf01000!FxIoQueue::ProcessPowerEvents+0x169:
fffff880`00e3c899 483950e8        cmp     qword ptr [rax-18h],rdx ds:00000000`0057ffeb=????????????????
Resetting default scope

LAST_CONTROL_TRANSFER:  from fffff800034831a9 to fffff80003483c00

STACK_TEXT:  
fffff880`18e75268 fffff800`034831a9 : 00000000`0000000a 00000000`0057ffeb 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
fffff880`18e75270 fffff800`03481e20 : fffffa80`1e28e300 fffffa80`1bb13930 00000000`00000000 00000000`00000000 : nt!KiBugCheckDispatch+0x69
fffff880`18e753b0 fffff880`00e3c899 : fffffa80`00000000 fffffa80`188b7148 00000000`00000002 00000000`00000000 : nt!KiPageFault+0x260
fffff880`18e75540 fffff880`00e271dd : fffffa80`1bb712f0 fffff880`18e75688 fffffa80`0000001f 00000000`00000000 : Wdf01000!FxIoQueue::ProcessPowerEvents+0x169
fffff880`18e75600 fffff880`00e385c1 : fffffa80`1bae0010 fffff880`00e3e000 00000000`00000000 fffffa80`1bb712f0 : Wdf01000!FxIoQueue::DispatchEvents+0x41d
fffff880`18e75680 fffff880`00e383f0 : fffffa80`1bb712f0 0000057f`e448ed08 fffffa80`1bb71660 fffffa80`1bb71300 : Wdf01000!FxIoQueue::StopProcessingForPower+0x181
fffff880`18e756d0 fffff880`00e381a8 : 00000000`0000000f fffff880`18e75700 fffffa80`1bb71d80 fffffa80`1bb65860 : Wdf01000!FxPkgIo::StopProcessingForPower+0x22c
fffff880`18e75740 fffff880`00e38131 : 00000000`00000004 00000000`00000000 fffffa80`1ca2b350 fffffa80`1bb06d30 : Wdf01000!FxPkgPnp::PowerGotoDx+0x64
fffff880`18e75780 fffff880`00e35c3c : 00000000`0000031a fffff880`18e758b0 00000000`ffff7fff fffffa80`1bb65860 : Wdf01000!FxPkgPnp::PowerGotoDNotZero+0x9
fffff880`18e757b0 fffff880`00e36106 : fffff880`00ebaf60 fffffa80`1bb65a18 fffffa80`1bb65860 fffffa80`1bb65860 : Wdf01000!FxPkgPnp::PowerEnterNewState+0x1d8
fffff880`18e75910 fffff880`00e35e9f : 00000000`00000000 fffff880`18e759f0 fffffa80`1bb65a00 fffffa80`1bb65860 : Wdf01000!FxPkgPnp::PowerProcessEventInner+0x13e
fffff880`18e75980 fffff880`00e3f687 : 00000000`00000000 fffffa80`1bb65860 00000000`00000000 00000000`00000000 : Wdf01000!FxPkgPnp::PowerProcessEvent+0x1b3
fffff880`18e75a20 fffff880`00e33fea : 00000000`00000002 0000057f`e449afd8 fffffa80`1bb71e10 fffffa80`2122b902 : Wdf01000!FxPkgFdo::DispatchDeviceSetPower+0x117
fffff880`18e75a70 fffff880`00e2b9da : fffffa80`2122b910 fffffa80`2122b910 fffffa80`2122b910 00000000`00000000 : Wdf01000!FxPkgPnp::Dispatch+0x2aa
fffff880`18e75ad0 fffff880`00e2baa6 : fffffa80`2122b910 00000000`00000000 fffffa80`1bb71e10 fffffa80`1bb71e10 : Wdf01000!FxDevice::Dispatch+0x19a
fffff880`18e75b10 fffff800`0359fe95 : 00000000`00000001 00000000`00000000 fffffa80`1bb71e10 fffffa80`2122b9b8 : Wdf01000!FxDevice::DispatchWithLock+0xa6
fffff880`18e75b50 fffff800`03721ede : ffffffff`fa0a1f00 fffffa80`21295940 00000000`00000080 00000000`00000000 : nt!PopIrpWorker+0x3c5
fffff880`18e75c00 fffff800`03474906 : fffff880`03374180 fffffa80`21295940 fffff880`0337f0c0 00000000`00000246 : nt!PspSystemThreadStartup+0x5a
fffff880`18e75c40 00000000`00000000 : fffff880`18e76000 fffff880`18e70000 fffff880`18e758a0 00000000`00000000 : nt!KiStartSystemThread+0x16


STACK_COMMAND:  kb

FOLLOWUP_IP: 
Wdf01000!FxIoQueue::ProcessPowerEvents+169
fffff880`00e3c899 483950e8        cmp     qword ptr [rax-18h],rdx

SYMBOL_STACK_INDEX:  3

SYMBOL_NAME:  Wdf01000!FxIoQueue::ProcessPowerEvents+169

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: Wdf01000

IMAGE_NAME:  Wdf01000.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  5010aa89

FAILURE_BUCKET_ID:  X64_0xD1_Wdf01000!FxIoQueue::ProcessPowerEvents+169

BUCKET_ID:  X64_0xD1_Wdf01000!FxIoQueue::ProcessPowerEvents+169

Followup: MachineOwner

7: kd>[COLOR=Blue] !niemiro.rawstack[/COLOR]
dps fffff88018e75268 fffff88018e75ff8
fffff880`18e75268  fffff800`034831a9 nt!KiBugCheckDispatch+0x69
fffff880`18e75270  00000000`0000000a
fffff880`18e75278  00000000`0057ffeb
fffff880`18e75280  00000000`00000002

...

fffff880`18e753f8  00000000`00000000
fffff880`18e75400  00000000`00000000
fffff880`18e75408  00000000`00000000
fffff880`18e75410  fffff880`03374180
fffff880`18e75418  fffff880`00e3f938 Wdf01000!FxPkgPnp::_PowerPolDevicePowerDownComplete
fffff880`18e75420  fffffa80`1bb44490
fffff880`18e75428  fffff880`0169e868 [COLOR=Red]tsusbhub[/COLOR]!BdEvtIoStop    [COLOR=SeaGreen]< driver for Remote Desktop USB Hub[/COLOR]
fffff880`18e75430  00000000`00000000
fffff880`18e75438  00000000`00000000
fffff880`18e75440  00000000`00000000

...

Here's also that WHEA crash:

1: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

WHEA_UNCORRECTABLE_ERROR (124)
A fatal hardware error has occurred. Parameter 1 identifies the type of error
source that reported the error. Parameter 2 holds the address of the
WHEA_ERROR_RECORD structure that describes the error conditon.
Arguments:
Arg1: 0000000000000000, Machine Check Exception
Arg2: [COLOR=DarkOrange]fffffa8018db5028[/COLOR], Address of the WHEA_ERROR_RECORD structure.
Arg3: 00000000bf800000, High order 32-bits of the MCi_STATUS value.
Arg4: 0000000000000124, Low order 32-bits of the MCi_STATUS value.

...

1: kd> [COLOR=Blue]!errrec[/COLOR] [COLOR=DarkOrange]fffffa8018db5028[/COLOR]
===============================================================================
Common Platform Error Record @ fffffa8018db5028
-------------------------------------------------------------------------------
Record Id     : 01ce93a377abdfa4
Severity      : Fatal (1)
Length        : 928
Creator       : Microsoft
Notify Type   : Machine Check Exception
Timestamp     : 8/8/2013 21:42:59 (UTC)
Flags         : 0x00000000

===============================================================================
Section 0     : Processor Generic
-------------------------------------------------------------------------------
Descriptor    @ fffffa8018db50a8
Section       @ fffffa8018db5180
Offset        : 344
Length        : 192
Flags         : 0x00000001 Primary
Severity      : Fatal

Proc. Type    : x86/x64
Instr. Set    : x64
[COLOR=Red]Error Type    : Cache error
Operation     : Generic[/COLOR]
Flags         : 0x00
Level         : 0
CPU Version   : 0x00000000000306c3
Processor ID  : 0x0000000000000001

===============================================================================
Section 1     : x86/x64 Processor Specific
-------------------------------------------------------------------------------
Descriptor    @ fffffa8018db50f0
Section       @ fffffa8018db5240
Offset        : 536
Length        : 128
Flags         : 0x00000000
Severity      : Fatal

Local APIC Id : 0x0000000000000001
CPU Id        : c3 06 03 00 00 08 10 01 - bf fb da 7f ff fb eb bf
                00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00
                00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00

Proc. Info 0  @ fffffa8018db5240

===============================================================================
Section 2     : x86/x64 MCA
-------------------------------------------------------------------------------
Descriptor    @ fffffa8018db5138
Section       @ fffffa8018db52c0
Offset        : 664
Length        : 264
Flags         : 0x00000000
Severity      : Fatal

Error         : [COLOR=Red]DCACHEL0_WR_ERR (Proc 1 Bank 1)[/COLOR]
  Status      : 0xbf80000000000124
  Address     : 0x000000081e130500
  Misc.       : 0x0000000000000086

 

[sergeyn]

Thank you very much!
Indeed these crashes (except the WHEA) happened when I was in rdp session on this pc. WHEA crash is me trying to do some overclocking, I thought I've deleted that dump, but apparently not.
I'll first try to find a way to disable any USB over rdp. If that doesn't help, I'll try just logmein (disabling rdp) or just rdp, disabling logmein).

May I ask how did you figure out that that was a remote session stuff ? I'll also send you full dump whenever I encounter the bluescreen again.

Thank you so much again!

Best Regards,
SergeyN

 

[sergeyn]

Hello again.

Did few more experiments, even got a stable reprocase of the bsod. To make it bsod I simply remote login via rdp to the pc and make it go to sleep mode. happens 100% of the times. It doesn't bsod if you do it via logme in. Am I right that the full memory dump will let you see the name of the guy causing it ?

Thanks,
Sergey.

 

[Vir Gnarus]

If you read the Analyst section of my post you'll see I used Windbg to dump the memory contents of the raw stack for the thread that crashed at the time, and what showed up in the midst of it was a pointer leading to a function inside tsusbhub, which is the driver used for USB connections for Remote Desktop. Basically it's telling me that at one time that thread called into tsusbhub to do some work, which - given the rest of the contents of the stack - led me to believe it was indeed relevant to what crashed here. Also if you notice above it, there's Wdf01000!FxPkgPnp::_PowerPolDevicePowerDownComplete, which - along with the current callstack shown - fits the bill with this dealing with a sleep session, as it's telling a device to power down into sleep state. Again, I can't determine the device involved without at least a kernel dump, and Driver Verifier may crash the system at a more appropriate time where it may discover the driver responsible.

Btw, in case you are wondering, !niemiro.rawstack that I used is a custom made extension by someone I know over at Sysnative, which simplifies dumping the raw stack contents of a thread. To do so normally, you would get thread details, grab the Limit and Base addresses from it, which involve start and end points for the raw stack, respectively, and then dump it with dps using those addresses for range. Understand the output may be different than what I get with !niemiro.rawstack, as it may output more (most likely just extra nulls) because it expands the whole available stack range, whereas !niemiro.rawstack uses some special variables to find and dump the stack of only what's currently been used.

Example is below:

0: kd>[COLOR=Blue] !thread[/COLOR]     [COLOR=Green]< Dump details of current thread[/COLOR]
GetPointerFromAddress: unable to read from fffff80003715000
THREAD fffffa80188799d0  Cid 0004.0014  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0
Not impersonating
GetUlongFromAddress: unable to read from fffff80003654ba4
Owning Process            fffffa8018830840       Image:         System
Attached Process          N/A            Image:         N/A
fffff78000000000: Unable to get shared data
Wait Start TickCount      35477984     
Context Switch Count      1735           IdealProcessor: 0             
ReadMemory error: Cannot get nt!KeMaximumIncrement value.
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address nt!PopIrpWorker (0xfffff800035f9ad0)
Stack Init fffff880009ecc70 Current fffff880009ec380
Base [COLOR=DarkOrchid]fffff880009ed000 [/COLOR]Limit [COLOR=DarkOrange]fffff880009e7000 [/COLOR]Call 0
Priority 14 BasePriority 13 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffff880`009ec268 fffff800`034dd1a9 : 00000000`0000000a ffffffff`ffffffe8 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
fffff880`009ec270 fffff800`034dbe20 : fffffa80`188799d0 fffff800`034e0a7a ffffffff`ff676980 00000000`00000000 : nt!KiBugCheckDispatch+0x69
fffff880`009ec3b0 fffff880`00e61899 : fffffa80`00000000 fffff800`0368ad20 fffffa80`192d7800 00000000`00000000 : nt!KiPageFault+0x260 (TrapFrame @ fffff880`009ec3b0)
fffff880`009ec540 fffff880`00e4c1dd : fffffa80`1bd08a20 fffff880`009ec688 fffffa80`000000e6 00000000`00000000 : Wdf01000!FxIoQueue::ProcessPowerEvents+0x169
fffff880`009ec600 fffff880`00e5d5c1 : fffffa80`1bd16590 fffff880`00e63000 00000000`00000000 fffffa80`1bd08a20 : Wdf01000!FxIoQueue::DispatchEvents+0x41d
fffff880`009ec680 fffff880`00e5d3f0 : fffffa80`1bd08a20 0000057f`e42f75d8 fffffa80`1bd08d90 fffffa80`1bd08a30 : Wdf01000!FxIoQueue::StopProcessingForPower+0x181
fffff880`009ec6d0 fffff880`00e5d1a8 : 00000000`c000000f fffff880`009ec700 fffffa80`1bd08390 fffffa80`1bd17020 : Wdf01000!FxPkgIo::StopProcessingForPower+0x22c
fffff880`009ec740 fffff880`00e5d131 : 00000000`00000004 00000000`00000000 00000000`00000000 fffff880`01642323 : Wdf01000!FxPkgPnp::PowerGotoDx+0x64
fffff880`009ec780 fffff880`00e5ac3c : 00000000`0000031a fffff880`009ec8b0 00000000`ffff7fff fffffa80`1bd17020 : Wdf01000!FxPkgPnp::PowerGotoDNotZero+0x9
fffff880`009ec7b0 fffff880`00e5b106 : fffff880`00edff60 fffffa80`1bd171d8 fffffa80`1bd17020 fffffa80`1bd17020 : Wdf01000!FxPkgPnp::PowerEnterNewState+0x1d8
fffff880`009ec910 fffff880`00e5ae9f : 00000000`00000000 fffff880`009ec9f0 fffffa80`1bd171c0 fffffa80`1bd17020 : Wdf01000!FxPkgPnp::PowerProcessEventInner+0x13e
fffff880`009ec980 fffff880`00e64687 : 00000000`00000000 fffffa80`1bd17020 00000000`00000000 00000000`00000000 : Wdf01000!FxPkgPnp::PowerProcessEvent+0x1b3
fffff880`009eca20 fffff880`00e58fea : 00000000`00000002 0000057f`e43006b8 fffffa80`1bd177b0 fffffa80`1be9aa02 : Wdf01000!FxPkgFdo::DispatchDeviceSetPower+0x117
fffff880`009eca70 fffff880`00e509da : fffffa80`1be9aa50 fffffa80`1be9aa50 fffffa80`1be9aa50 00000000`00000000 : Wdf01000!FxPkgPnp::Dispatch+0x2aa
fffff880`009ecad0 fffff880`00e50aa6 : fffffa80`1be9aa50 00000000`00000000 fffffa80`1bd177b0 fffffa80`1bd177b0 : Wdf01000!FxDevice::Dispatch+0x19a
fffff880`009ecb10 fffff800`035f9e95 : 00000000`00000001 00000000`00000000 fffffa80`1bd177b0 fffffa80`1be9aaf8 : Wdf01000!FxDevice::DispatchWithLock+0xa6
fffff880`009ecb50 fffff800`0377bede : 00000000`00000000 fffffa80`188799d0 00000000`00000080 00000000`00000000 : nt!PopIrpWorker+0x3c5
fffff880`009ecc00 fffff800`034ce906 : fffff880`009ee180 fffffa80`188799d0 fffff880`009f90c0 00000000`00000000 : nt!PspSystemThreadStartup+0x5a
fffff880`009ecc40 00000000`00000000 : fffff880`009ed000 fffff880`009e7000 fffff880`009ec380 00000000`00000000 : nt!KiStartSystemThread+0x16

0: kd> [COLOR=Blue]dps [/COLOR][COLOR=DarkOrange]fffff880009e7000 [/COLOR][COLOR=DarkOrchid]fffff880009ed000[/COLOR]     [COLOR=Green]< Stacks grow *backwards*, so use Limit as start and Base as end[/COLOR]
fffff880`009e7000  ????????`????????
fffff880`009e7008  ????????`????????
fffff880`009e7010  ????????`????????
fffff880`009e7018  ????????`????????

...

If you'd like extra information to clarify things and improve, I have a thread here that has a lot of links that point to really good resources to get you started.


Anyways, you've checked to see if the system goes into sleep cleanly if not remoted into, correct? Just making sure this is specifically an RDC thing and not just a problem with a device sleeping overall.

 

[sergeyn]

Hello again,

I've made a full memory dump. Where can I PM you download details ?

Thanks!

 

[Vir Gnarus]

Just click my name and click "Send private message".

 

[sergeyn]

For the first time this bluescreen happened during normal reboot (a reboot after installing windows updates).

Same DRIVER_POWER_STATE_FAILURE .

0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_POWER_STATE_FAILURE (9f)
A driver has failed to complete a power IRP within a specific time (usually 10 minutes).
Arguments:
Arg1: 0000000000000003, A device object has been blocking an Irp for too long a time
Arg2: fffffa801b989870, Physical Device Object of the stack
Arg3: fffff80004e2a3d8, nt!TRIAGE_9F_POWER on Win7, otherwise the Functional Device Object of the stack
Arg4: fffffa80277a4ca0, The blocked IRP

Debugging Details:
------------------


DRVPOWERSTATE_SUBCODE: 3

IMAGE_NAME: umbus.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4ce7a695

MODULE_NAME: umbus

FAULTING_MODULE: fffff88005a50000 umbus

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0x9F

PROCESS_NAME: System

CURRENT_IRQL: 2

TAG_NOT_DEFINED_c000000f: FFFFF80004E30FB0

STACK_TEXT:
fffff800`04e2a388 fffff800`0373d8c2 : 00000000`0000009f 00000000`00000003 fffffa80`1b989870 fffff800`04e2a3d8 : nt!KeBugCheckEx
fffff800`04e2a390 fffff800`036d884c : fffff800`04e2a4c0 fffff800`04e2a4c0 00000000`00000000 00000000`00000001 : nt! ?? ::FNODOBFM::`string'+0x33af0
fffff800`04e2a430 fffff800`036d86e6 : fffff800`0387df20 00000000`00c92007 00000000`00000000 fffff880`03d89677 : nt!KiProcessTimerDpcTable+0x6c
fffff800`04e2a4a0 fffff800`036d85ce : 000001de`c167cf0c fffff800`04e2ab18 00000000`00c92007 fffff800`0384b368 : nt!KiProcessExpiredTimerList+0xc6
fffff800`04e2aaf0 fffff800`036d83b7 : 000000a3`a344cac1 000000a3`00c92007 000000a3`a344ca59 00000000`00000007 : nt!KiTimerExpiration+0x1be
fffff800`04e2ab90 fffff800`036c590a : fffff800`03848e80 fffff800`03856cc0 00000000`00000001 fffff880`00000000 : nt!KiRetireDpcList+0x277
fffff800`04e2ac40 00000000`00000000 : fffff800`04e2b000 fffff800`04e25000 fffff800`04e2ac00 00000000`00000000 : nt!KiIdleLoop+0x5a


STACK_COMMAND: kb

FOLLOWUP_NAME: MachineOwner

FAILURE_BUCKET_ID: X64_0x9F_3_tsusbhub_IMAGE_umbus.sys

BUCKET_ID: X64_0x9F_3_tsusbhub_IMAGE_umbus.sys

Followup: MachineOwner

 

[Vir Gnarus]

It's slightly different, but overall it's pretty much the same, in that the terminal server USB Hub driver was at least partly responsible.

I did not realize you were giving me complete dumps - they are huge! It'll take a while to scrutinize this.

 

[sergeyn]

>I did not realize you were giving me complete dumps - they are huge! It'll take a while to scrutinize this.
That's what I thought you were asking. Let me know if I can get something smaller, but more informative than 256kb dump.

 

[Vir Gnarus]

Well, I understand that, but I didn't realize you had that much RAM for a non-server system. This is a production system I take it?

 

[sergeyn]

no, but I do run some number-crunching applications, those take some ram.

 

[Vir Gnarus]

That's quite a bit! Anyways, I'll try looking through this at earliest convenience.

 

[sergeyn]

Thank you so much! I tried to do it myself yesterday, but I'm not a driver development expert and couldn't dig past what !analyze -v showed (i.e. - tsusbhup and umsys).

Do you consider looking into it directly at my machine (via some screen/control sharing)? I have wingdb installed, but not your fancy windgb extensions.

 

[Vir Gnarus]

I would prefer not, and the only way that would benefit would be to do a live kernel debug session, which won't happen unless my PC's connected to yours physically.

Honestly, there's only two extensions I've added to this Windbg, and that's Niemiro's rawstack extension he whipped up real quick at Sysnative (I think it's private), and CMKD, which has a couple of extensions.

 

[sergeyn]

Hello again,

I keep experimenting with this bluescreen. Basically what I've figured out so far is that whenever you RDP into the pc, then "I bluescreen on reboot/sleep" mode turns on and stays on even after you log out.

This is the last crash with some of my manual poking with windb (it contains FxIoQueue stuff). Can you comment on the following callstack ? I did some manual decodings (on the right), but I'm a noob in windb and kernel-space objects. All I know so far is !devobj, !drvobj, but none of the addresses in the callstack seem to be recognized. I wonder if there are other command I should try ?

fffff880`009ec3a8  fffff800`03677da0 nt!KiPageFault+0x260
fffff880`009ec3b0  00000000`00000002
fffff880`009ec3b8  fffff800`03682f4d nt!KeAcquireInStackQueuedSpinLock+0x8d
fffff880`009ec3c0  fffffa80`23e39a00
fffff880`009ec3c8  00000000`00000000
fffff880`009ec3d0  fffffa80`23818ac0
fffff880`009ec3d8  00001f80`0100c498
fffff880`009ec3e0  00000000`00000000
fffff880`009ec3e8  fffffa80`1be7cc30
fffff880`009ec3f0  fffffa80`1fe9dd88
fffff880`009ec3f8  00000000`00000000
fffff880`009ec400  00000000`00000000
fffff880`009ec408  00000000`00000000
fffff880`009ec410  fffff880`009b3180 --> 00000004`00001f80  fffffa80`188789d0 fffffa80`18886660 fffff880`009be0c0 00000004`00000000 fffff880`009ecc70 00000000`00000000 00000000`00000000 00000000`80050033 ffffffff`ffffffe8 00000000`00187000 00000000`000406f8 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 00000000`ffff4ff0 00000000`00000400 007f0000`00000000 fffff880`009be640 0fff0000`00000000 fffff880`009be6c0
fffff880`009ec418  fffff880`009ec498
fffff880`009ec420  fffffa80`1c015490 <-- Wdf01000!FxCallbackMutexLock::`vftable'
fffff880`009ec428  fffff880`15346868 tsusbhub!BdEvtIoStop
fffff880`009ec430  00000000`00000000
fffff880`009ec438  00000000`00000000
fffff880`009ec440  00000000`00000000
fffff880`009ec448  00000000`00000000
fffff880`009ec450  00000000`00000000
fffff880`009ec458  00000000`00000000
fffff880`009ec460  00000000`00000000
fffff880`009ec468  00000000`00000000
fffff880`009ec470  00000000`00000000
fffff880`009ec478  00000000`00000000
fffff880`009ec480  ffffffff`ffffffe8
fffff880`009ec488  fffff800`0377df2d nt!PopQueueQuerySetIrp+0x15d
fffff880`009ec490  fffffa80`191efe10
fffff880`009ec498  fffffa80`00000103
fffff880`009ec4a0  fffffa80`23e39ad0
fffff880`009ec4a8  00000000`00000000
fffff880`009ec4b0  00000000`00000000
fffff880`009ec4b8  fffff800`03829b80 nt!PopIrpLock
fffff880`009ec4c0  fffffa80`23818a00
fffff880`009ec4c8  fffffa80`191efe10
fffff880`009ec4d0  00000000`00000000
fffff880`009ec4d8  fffff800`0379969b nt!PoRequestPowerIrp+0x1ab
fffff880`009ec4e0  fffffa80`23e39ad0
fffff880`009ec4e8  00000000`00000002
fffff880`009ec4f0  00000000`00000000
fffff880`009ec4f8  fffff880`00ed3938 Wdf01000!FxPkgPnp::_PowerPolDevicePowerDownComplete
fffff880`009ec500  fffffa80`00000001
fffff880`009ec508  fffff880`009ec5b8
fffff880`009ec510  00000000`00000000
fffff880`009ec518  fffff880`00ed0899 Wdf01000!FxIoQueue::ProcessPowerEvents+0x169
fffff880`009ec520  00000000`00000010
fffff880`009ec528  00000000`00010203
fffff880`009ec530  fffff880`009ec540
fffff880`009ec538  00000000`00000000
fffff880`009ec540  00000000`00000004
fffff880`009ec548  fffff880`00ed376d Wdf01000!FxPkgPnp::PowerPolicySendDevicePowerRequest+0x79
fffff880`009ec550  fffffa80`191ed780
fffff880`009ec558  00000000`00000000
fffff880`009ec560  fffff880`00000004
fffff880`009ec568  fffffa80`1be7cb60
fffff880`009ec570  fffffa80`1be75010
fffff880`009ec578  fffff880`009ec608
fffff880`009ec580  00000001`1be7cb60
fffff880`009ec588  fffff880`00ed0b72 Wdf01000!FxIoQueue::ProcessPowerEvents+0x442
fffff880`009ec590  00000000`00000000
fffff880`009ec598  fffffa80`1c015490 <--Wdf01000!FxCallbackMutexLock::`vftable'
fffff880`009ec5a0  fffff880`15346868 tsusbhub!BdEvtIoStop
fffff880`009ec5a8  fffffa80`191ed780 <--Wdf01000!FxPkgFdo::`vftable'
fffff880`009ec5b0  fffff880`00f51290 Wdf01000!FxPkgPnp::m_WdfPowerPolicyStates+0x4e0
fffff880`009ec5b8  00000000`00000000
fffff880`009ec5c0  fffffa80`1be75010
fffff880`009ec5c8  00000000`00000000
fffff880`009ec5d0  fffff880`00f4ed40 Wdf01000!FxWmiIrpHandler::m_WmiDispatchTable+0x13a0
fffff880`009ec5d8  00000000`00000001
fffff880`009ec5e0  00000000`00000001
fffff880`009ec5e8  fffffa80`1be7cb60 <--Wdf01000!FxIoQueue::`vftable'
fffff880`009ec5f0  fffff880`009ec640 +10 us the stack (00000000`00000000)
fffff880`009ec5f8  fffff880`00ebb1dd Wdf01000!FxIoQueue::DispatchEvents+0x41d
fffff880`009ec600  fffffa80`1be7cb60
fffff880`009ec608  fffff880`009ec688 (up the stack arg of StopProcessingForPower) fffff880`00ed2000 Wdf01000!_FX_DRIVER_GLOBALS::WaitForSignal+0x48
fffff880`009ec610  fffffa80`000000e6
fffff880`009ec618  00000000`00000000
fffff880`009ec620  fffff880`009ec6d0 +21 up the stack (arg of StopProcessingEvent? ) fffffa80`1be7cb60 <--Wdf01000!FxIoQueue::`vftable'
fffff880`009ec628  00000000`00000001
fffff880`009ec630  00000000`00000001
fffff880`009ec638  fffffa80`1be7cb60 <--Wdf01000!FxIoQueue::`vftable'
fffff880`009ec640  00000000`00000000
fffff880`009ec648  fffffa80`1be7cea0
fffff880`009ec650  00000000`00000000
fffff880`009ec658  00000000`00000001
fffff880`009ec660  00000000`00000000
fffff880`009ec668  fffffa80`1be7ce00
fffff880`009ec670  fffff880`009ec6b0
fffff880`009ec678  fffff880`00ecc5c1 Wdf01000!FxIoQueue::StopProcessingForPower+0x181

Small update, one of the addresses near PopQueueQuerySetIrp was recognized with the following command:

4: kd> !irp fffffa80`23e39ad0
Irp is active with 3 stacks 2 is current (= 0xfffffa8023e39be8)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
 [  0, 0]   0  0 00000000 00000000 00000000-00000000    

            Args: 00000000 00000000 00000000 00000000
>[ 16, 2]   0 e1 [COLOR=DarkOrange]fffffa80191efe10[/COLOR] 00000000 00000000-00000000    pending
           \Driver\CompositeBus
            Args: 00014400 00000001 00000004 00000002
 [  0, 0]   0  0 00000000 00000000 00000000-fffffa8023818ac0    

            Args: 00000000 00000000 00000000 00000000
4: kd> dps fffffa80`23e39ad0

Then !devobj on that address

4: kd> !devobj  fffffa80191efe10
Device object (fffffa80191efe10) is for:
  \Driver\CompositeBus DriverObject [COLOR=Teal]fffffa80191ec060[/COLOR]
Current Irp 00000000 RefCount 0 Type 0000002a Flags 00002004
DevExt fffffa80191ec560 DevObjExt fffffa80191eff88 
ExtensionFlags (0x00000800)  DOE_DEFAULT_SD_PRESENT
Characteristics (0x00000100)  FILE_DEVICE_SECURE_OPEN
AttachedTo (Lower) fffffa8018874bb0 \Driver\PnpManager
Device queue is not busy.

4: kd> !drvobj [COLOR=SeaGreen]fffffa80191ec060[/COLOR]
Driver object (fffffa80191ec060) is for:
 \Driver\CompositeBus
Driver Extension List: (id , addr)
(fffff88000f20bf4 fffffa80191dc3f0)  
Device Object list:
fffffa80191efe10

I'm not sure how to decode device extensions?

 

[Vir Gnarus]

I've never got dev extensions to work either on newer system dumps. Instead I just check the devnode with !devnode, preferably dumping the tree so I can understand the relationship between the device objects on the devstack. Make sure to use !devstack as well! Oh, and add 7 to the end of !drvobj after the drive object address and it'll pump out extra details like the routine list. This will help you determine what function was called when passed that power IRP you're looking at for the CompositeBus.

For the list of major/minor function codes for IRPs, you can check the !irp help page on Windbg and it'll give you a list. Compare that with the [16,2] on the IRP and you can see it's a power IRP doing a SET_POWER. Sounds relevant to what we're dealing with. Also, type the !irp command again for that IRP, but add 1 to the end to get header details on the IRP. It may provide us a status code that can tell us a bit what's wrong.

Overall thanks for perusing this a bit. TBH, I probably won't be able to work on this until next week as this week (especially this weekend) has and is going to be crazy! I don't think I will have sufficient time to sit down and scrutinize your crashdump until then. Sorry for the wait, but it looks like you're doing well by yourself!

 

[sergeyn]

Hi,

Wondering if you have ever had a chance to go deeper into the issue, which is still present on my pc. Btw, full memory dumps stopped being generated. I wonder if it is because I've moved my page file to another partition?

Thanks.