BSOD Across Many identical systems. Need Dump analysis

Optimedica · 29 Oct 2012

Hello,

We have 45 systems that are all running on identical hardware:
Custom Computer
MB: Intel X58
Proc: Intel i7 HT enabled
Mem: 8GB DDR3 1066
Int. Sound
1 x 500GB 7200 RPM Sata
750W PS
Custom Matrox Video

For legal reasons we cannot run windows updates or make software changes on the systems regularly. All systems are running the base OS install and have no network or internet access.
recently on the systems we started getting Random BSOD's. I am unable to reproduce them even when exercising the same steps that caused it not 1 min before.
I have assesed the dump files as much as I can and was wondering if someone else can look at on for me?
The dump files is below from WinDBG.

Loading Dump File [C:\Users\acoleman\Desktop\Windows Dump Logs\597\102912-23025-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Code:

Symbol search path is: SRV*C:\SymCache*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Kernel Version 7600 MP (12 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7600.16385.amd64fre.win7_rtm.090713-1255
Machine Name:
Kernel base = 0xfffff800`02e0f000 PsLoadedModuleList = 0xfffff800`0304ce50
Debug session time: Mon Oct 29 09:57:23.210 2012 (UTC - 7:00)
System Uptime: 0 days 0:01:06.256
Loading Kernel Symbols
...............................................................
................................................................
.................................................
Loading User Symbols
Loading unloaded module list
............
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 1000007E, {ffffffffc0000047, fffff80002eaf7f8, fffff880035324f0, fffff88003532590}

Unable to load image \SystemRoot\System32\Drivers\EloUsbG2.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for EloUsbG2.sys
*** ERROR: Module load completed but symbols could not be loaded for EloUsbG2.sys
Probably caused by : EloUsbG2.sys ( EloUsbG2+d0f5 )

Followup: MachineOwner
---------

4: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

SYSTEM_THREAD_EXCEPTION_NOT_HANDLED_M (1000007e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003.  This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG.  This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG.  This will let us see why this breakpoint is
happening.
Arguments:
Arg1: ffffffffc0000047, The exception code that was not handled
Arg2: fffff80002eaf7f8, The address that the exception occurred at
Arg3: fffff880035324f0, Exception Record Address
Arg4: fffff88003532590, Context Record Address

Debugging Details:
------------------


EXCEPTION_CODE: (NTSTATUS) 0xc0000047 - An attempt was made to release a semaphore such that its maximum count would have been exceeded.

FAULTING_IP: 
nt!RtlRaiseStatus+18
fffff800`02eaf7f8 488b8424b8010000 mov     rax,qword ptr [rsp+1B8h]

EXCEPTION_RECORD:  fffff880035324f0 -- (.exr 0xfffff880035324f0)
ExceptionAddress: fffff80002eaf7f8 (nt!RtlRaiseStatus+0x0000000000000018)
   ExceptionCode: c0000047
  ExceptionFlags: 00000001
NumberParameters: 0

CONTEXT:  fffff88003532590 -- (.cxr 0xfffff88003532590)
rax=0000000000000000 rbx=00000000c0000047 rcx=fffff88003532590
rdx=0000000000000000 rsi=fffff88003532ba1 rdi=0000000000000001
rip=fffff80002eaf7f8 rsp=fffff880035324d0 rbp=fffff800030245f8
 r8=0000000000000000  r9=0000000000000000 r10=0000000000000000
r11=fffffa800b53d180 r12=0000000000000001 r13=fffff880009b2180
r14=0000000000000000 r15=0000000000000001
iopl=0         nv up ei ng nz na pe nc
cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00000282
nt!RtlRaiseStatus+0x18:
fffff800`02eaf7f8 488b8424b8010000 mov     rax,qword ptr [rsp+1B8h] ss:0018:fffff880`03532688=fffff80002eaf7f8
Resetting default scope

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0x7E

PROCESS_NAME:  System

CURRENT_IRQL:  0

ERROR_CODE: (NTSTATUS) 0xc0000047 - An attempt was made to release a semaphore such that its maximum count would have been exceeded.

LAST_CONTROL_TRANSFER:  from fffff80002edd7c2 to fffff80002eaf7f8

STACK_TEXT:  
fffff880`035324d0 fffff800`02edd7c2 : ffffffff`00000000 ffff0000`05ccbc61 fffff880`03532b68 fffff880`069e25ab : nt!RtlRaiseStatus+0x18
fffff880`03532a70 fffff880`069e70f5 : 0000057f`f5955ff8 fffff880`00000000 fffff880`03532b70 fffff880`03532b00 : nt! ?? ::FNODOBFM::`string'+0xf63c
fffff880`03532af0 0000057f`f5955ff8 : fffff880`00000000 fffff880`03532b70 fffff880`03532b00 fffff880`009b2180 : EloUsbG2+0xd0f5
fffff880`03532af8 fffff880`00000000 : fffff880`03532b70 fffff880`03532b00 fffff880`009b2180 fffff880`0155fa92 : 0x57f`f5955ff8
fffff880`03532b00 fffff880`03532b70 : fffff880`03532b00 fffff880`009b2180 fffff880`0155fa92 fffff880`03532ba8 : 0xfffff880`00000000
fffff880`03532b08 fffff880`03532b00 : fffff880`009b2180 fffff880`0155fa92 fffff880`03532ba8 fffffa80`0b53d180 : 0xfffff880`03532b70
fffff880`03532b10 fffff880`009b2180 : fffff880`0155fa92 fffff880`03532ba8 fffffa80`0b53d180 fffffa80`08d26500 : 0xfffff880`03532b00
fffff880`03532b18 fffff880`0155fa92 : fffff880`03532ba8 fffffa80`0b53d180 fffffa80`08d26500 00000000`00004e75 : 0xfffff880`009b2180
fffff880`03532b20 0000057f`f5955ff8 : 00000000`00101200 00000200`33003300 00000000`00000000 fffa800a`6aa37000 : NETIO!WfpStartTimerForLeftTime+0xb2
fffff880`03532b80 00000000`00101200 : 00000200`33003300 00000000`00000000 fffa800a`6aa37000 00000000`000000ff : 0x57f`f5955ff8
fffff880`03532b88 00000200`33003300 : 00000000`00000000 fffa800a`6aa37000 00000000`000000ff ffff0000`05ccbf61 : 0x101200
fffff880`03532b90 00000000`00000000 : fffa800a`6aa37000 00000000`000000ff ffff0000`05ccbf61 fffffa80`00000000 : 0x200`33003300


FOLLOWUP_IP: 
EloUsbG2+d0f5
fffff880`069e70f5 ??              ???

SYMBOL_STACK_INDEX:  2

SYMBOL_NAME:  EloUsbG2+d0f5

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: EloUsbG2

IMAGE_NAME:  EloUsbG2.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  4dcda5a5

STACK_COMMAND:  .cxr 0xfffff88003532590 ; kb

FAILURE_BUCKET_ID:  X64_0x7E_EloUsbG2+d0f5

BUCKET_ID:  X64_0x7E_EloUsbG2+d0f5

Followup: MachineOwner
---------

4: kd> lmvm EloUsbG2
start             end                 module name
fffff880`069da000 fffff880`069ff000   EloUsbG2 T (no symbols)           
    Loaded symbol image file: EloUsbG2.sys
    Image path: \SystemRoot\System32\Drivers\EloUsbG2.sys
    Image name: EloUsbG2.sys
    Timestamp:        Fri May 13 14:41:57 2011 (4DCDA5A5)
    CheckSum:         00025C09
    ImageSize:        00025000
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Dwarf · 29 Oct 2012

What do you mean by this?

Optimedica said:

For legal reasons we cannot run windows updates or make software changes on the systems regularly. All systems are running the base OS install and have no network or internet access.

If you mean that these are restrictions placed on the computers by your IT department, then you should take this issue up with them, especially if you are getting the same error on multiple machines (which would seem to indicate that something wants updating).

Optimedica · 29 Oct 2012

The computers are installed into a controlled device. All changes require a Verification and Validation prior to release.
Because of this we cannot just make changes when ever we want meaning this issue just came out of the blue with no software change. With this particular issue as far as IT is concerned I am the top level for support. This is why I am reaching out to gain additional review.

Britton30 · 29 Oct 2012

Can you remove and reinstall the EloUsbG2.sys? It is a driver for touch-screens.

What is meant that 45 systems are installed in a controlled device?

Optimedica · 29 Oct 2012

Yes we have tried the uninstall reinstall of the driver.
The computer is the primary component to a Medical device. There are 45 unit of the device installed in the field. The FDA controls the way we can address software related problems.

That being said I am a windows Sys admin and have done all the basic troubleshooting approaches based on the data presented in the Dump file.
What I do not know is how to interpret the hex data in the log files and determine if there is any useful non obvious data.

Layback Bear · 29 Oct 2012

Are all the computers acting this way? If so their/there is something that they are all doing or attached to. Do they have wifi ability and a location that could of sent them something. I'm thinking like a large medical center where/wheir a signal could of been sent by what ever and the signal was picked up using wifi. If all are infected their/there has to be something in common. Possibly a power surge.