Recurring BSOD Problem

ssu said:

Ive only tried the current OS. This computer was built around a month ago, and hasn't been overclocked. Are you saying the hardware is defective, or was it something I did (ive done nothing other than put it together basically)?

I'm 99% confident that your hardware exhibited a defect in 091609-18595-01.dmp. It is virtually impossible to tell (from a minidump) whether the cause is absolutely _broken_ hardware or something like insufficient cooling or a speck of dust lodged in a connector.

Personally, I'd start by running a memory tester for days if necessary - long enough to either find something or convince myself that I'm not going to find unreliable memory using that method.

chuckr said:

fffff880`014e7f4a 48896e20 mov qword ptr [rsi+20h],rbp
fffff880`014e7f4e 48897e08 mov qword ptr [rsi+8],rdi
fffff880`014e7f52 48897e10 mov qword ptr [rsi+10h],rdi
fffff880`014e7f56 48897e18 mov qword ptr [rsi+18h],rdi

The IP seems to count OK:
No instruction at 4b may be because of correct IP increment,
(+1 per byte, viz):
4a = 48 89 6e 20
4b is the '89',
4c is the '6e',
4d is the '20',

then,
4e 48 89 7e 08 <---< is appropriate, count-wise.

What strikes me (as quick first look) is the "Increment" of the
"LOOP" or "REPx" instruction, not proceeding logically:
Starting at +00h, then bumping up by +08h.....

Code:

0: kd> u fffff880`014e7f3a
ndis+0x1f3a:
fffff880`014e7f3a 8b6c2430 mov ebp,dword ptr [rsp+30h]
fffff880`014e7f3e 4885f6 test rsi,rsi
fffff880`014e7f41 0f8413840100 je ndis+0x1a35a (fffff880`0150035a)
fffff880`014e7f47 48893e mov qword ptr [rsi],rdi
fffff880`014e7f4a 48896e20 mov qword ptr [rsi+20h],rbp <===< *****
fffff880`014e7f4e 48897e08 mov qword ptr [rsi+8],rdi
fffff880`014e7f52 48897e10 mov qword ptr [rsi+10h],rdi
fffff880`014e7f56 48897e18 mov qword ptr [rsi+18h],rdi

The problem seems (to me) that this instruction shud be:

48897e00 mov qword ptr [rsi+00h],rdi

Perhaps something wrong, not in IP, but the 32-bit word itself...

Yep, just like DOS's "debug". Have rep

IDA has amazing graphical disassembly capabilities which are immensely useful when trying to understand other people's code - especially without access to "private" symbols. Between WinDBG/CDB/NTSD (all based on the same engine) and IDA, I haven't had any need to use another dedicated disassembler for a long time.

With regards to memtest and CPU stress test utils, I'm not well versed in those techniques. To me, the software's the interesting bit, and I'll happily defer to those with more knowledge when it comes to figuring out whether it's the third DIMM from the left or the APIC that's responsible for any given hardware weirdness.

As far as the IP is concerned, let's go a lot further back and start disassembly from there:

Code:

 
0: kd> u fffff880`014e7f10 L12
ndis+0x1f10:
fffff880`014e7f10 0041ff add byte ptr [rcx-1],al
fffff880`014e7f13 842c94 test byte ptr [rsp+rdx*4],ch
fffff880`014e7f16 0100 add dword ptr [rax],eax
fffff880`014e7f18 00ff add bh,bh
fffff880`014e7f1a 15b9310500 adc eax,531B9h
fffff880`014e7f1f 488bf0 mov rsi,rax
fffff880`014e7f22 4885c0 test rax,rax
fffff880`014e7f25 0f84aa010000 je ndis+0x20d5 (fffff880`014e80d5)
fffff880`014e7f2b 4885f6 test rsi,rsi
fffff880`014e7f2e 0f843a020000 je ndis+0x216e (fffff880`014e816e)
fffff880`014e7f34 448b642450 mov r12d,dword ptr [rsp+50h]
fffff880`014e7f39 4c8b6c2430 mov r13,qword ptr [rsp+30h]
fffff880`014e7f3e 4885f6 test rsi,rsi
fffff880`014e7f41 0f8413840100 je ndis+0x1a35a (fffff880`0150035a)
fffff880`014e7f47 48893e mov qword ptr [rsi],rdi
fffff880`014e7f4a 48896e20 mov qword ptr [rsi+20h],rbp
fffff880`014e7f4e 48897e08 mov qword ptr [rsi+8],rdi
fffff880`014e7f52 48897e10 mov qword ptr [rsi+10h],rdi

Going that far back, we can be almost certain that the disassembly has correctly "snapped to" by the time it reaches '4a. According to the in-memory version of the ndis.sys image, the instruction at '4a is 4 bytes long ("48896e20").

The problem with the IP is that it _should_ be at '4a or '4e, but certainly not '4b, because even if the in-memory NDIS image does not match the on-disk binary, the disassembly above shows the code that the machine ought to have been following.

chuckr said:

The problem seems (to me) that this instruction shud be:

48897e00 mov qword ptr [rsi+00h],rdi

Perhaps something wrong, not in IP, but the 32-bit word itself...

Quite possible. The in-memory image may be corrupted. We'd need somebody with a 7100 build (like the OP) to verify what's meant to be at that location. Bit-flips due to hardware issues may be responsible for both the IP misalignment and corruption of the binary image.

By itself, corruption of the in-memory image won't cause the IP to be misaligned though. However badly misrepresented the code may be in memory, perhaps due to software bugs, it's still meant to be precisely followed by the hardware, and the (R)IP should never be between two instructions.

Endian little hate I :)

Couldn't help myself - I had to hunt down a 7100 machine like the OPs and check that range. The NDIS.SYS in-memory image is intact in their case, at least in that location:

Mine:
.text:0000000000011F40 F6 0F 84 13 84 01 00 48 89 3E 48 89 6E 20 48 89 ÷¤ää
.Hë>Hën Hë

OPs:
0: kd> db fffff880`014e7f40 L10
fffff880`014e7f40 f6 0f 84 13 84 01 00 48-89 3e 48 89 6e 20 48 89 .......H.>H.n H.

I used WinDBG for the OPs and IDA for mine, just to confirm. Unsurprisingly, they agree on the disassembly. Perhaps the reference to RSI+20h and RBP is located there for pipelining reasons - some sort of structural or data hazard avoidance.

Thanks for a very interesting discussion :)

You really don't know what the OP has...
You're assuming a valid, operable 7100, like you have...

The aim was to check whether their in-memory version of the instruction at '4a matches ndis.sys on another 7100 box, or whether it was possibly mangled as you suspected (and I agree there was good reason to suspect that). Their minidump tells me some details regarding the OS version labelling and the specifics of their NDIS.SYS binary ('vertarget', '!lmi ndis.sys'), and the comparison to my own (presumed good) copy tells me that there is no discrepancy, at least in that area - the '4a instruction really is "mov [rsi+20h], rbp".

I believe the caveat here is:

Quote:
0: kd> .trap fffff880`031af4a0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.

The Windows trap frame struct ('dt _KTRAP_FRAME') doesn't necessarily get populated with the contents of every register - only those which are necessary for the operation of a particular type of trap handler: nt!KiTrap0E, nt!KiTrap0A... Hence, the warning above is spewed by the debugger as a generic reminder that at least some of the values may not be accurate. (R)IP is obviously essential though - so we can get back to where we left off when the handler's done - and it'll absolutely be saved every time. In the OPs case, the RIP value is undeniably populated and pointing close to where it should, albeit off-by-one byte:

0: kd> dt _KTRAP_FRAME fffff880`031af4a0 RIP
nt!_KTRAP_FRAME
+0x168 Rip : 0xfffff880`014e7f4b

(That's where the debugger gets its own 'kv' trap info of course.)

Not knowing exactly how "real-time" the "trap-frame" operates, I would venture to guess that the possibility exists that:
the CPU 'instruction-decoder' was in the midst of updating the IP register
(which may only increment +1 at a time, depending upon hw logic gates),

For what it's worth, I don't believe it works that way. From a software point of view, IP updates are an atomic operation, and the OS will never be able to watch the IP "count" incrementally between two valid instruction offsets.

If I were a professional, I'd 'Instruction_Breakpoint' at 7f3e, then poke around the registers (during real-time execution).
Then 'Single-step' and poke around some more.
Maybe setting a 'Data_Breakpoint' or two, at curious areas.

Without wishing to be seen to be buttering up a forum heavyweight, it's very obvious that you're very professional :)

I agree completely that if we had a setup with a reproducible fault and a kernel debugger hooked up it would be very interesting to 'bp <\'7f3e>', then 't' for a bit, watch if it's going to branch or not in that conditional jump, and 'ba <address>' (hardware breakpoint) on some of the underlying structures. I rather doubt this is reproducible though - if it's hardware, it may never happen again in the same way and in the same place.

That's where I would think differently:
the 'non-orthogonal' instruction set and the 'instruction-decoder' could well change the IP register with screwy counts:

Oh sure, software can directly walk the IP all over the place (the debugger does it all the time), but the point is that the code in question does nothing to directly shunt the IP one byte further along than it should be. Perhaps the thread was pre-empted and its context record was damaged, but the chances are very small that the stored RIP value merely got INCed (or similar).

Surely, there exists a hw problem.... If I were a betting man, the CPU!

Isolation requires in-person 'hand-holding', observation, and trusty 'tools'...

Yes, I certainly wouldn't be putting my money on a software flaw in this particular crash. I don't know whether it's the processor though.

Hope we don't both get the BanHammer, for getting this thread:

I'm prepared to run that risk for the joy I get from discussing these issues

chuckr said:

Didn't want this thing to 'die on the vine', especially since you went through the trouble of verifying the binary file 'ndis.sys'. Obviously, the OP has the correct code with his ndis.sys and the debug dump also indicates that.

Glad to talk about it any time :)

In reality I only verified the contents of the QWord starting at 0xfffff880`014e7f40 - the one that contains the '4a and '4e instructions, as well as the (non-existent) '4b instruction the box seemingly tried to execute at the time of the crash. I cannot be certain whether the OPs entire NDIS.SYS image is correct, mostly because it's not all recorded in the 270KB worth of minidump! (If we had a full kernel dump from the OP we could use the debugger's !chkimg extension to verify the whole thing against a known-good copy.)

chuckr said:

My aiming at the instruction: mov qword ptr [rsi+20h],rbp
was definitely with blinders on, assuming the +08h should have been in an orderly 'progression', starting with +00h. The bloody +00h is, in fact there, simply not indicated as such.

Sure, "[rsi]" is equivalent to "[rsi+00h]", if that's what you mean. That's just the way WinDBG chooses to present the disassembled mnemonics.

chuckr said:

Is it 'normal' for 64-bit code to be using Dword pointers?

I don't think I understand where you're seeing the 32-bit pointer(s)? They certainly shouldn't be any absolute 32-bit pointers because a jump to such an address would easily take us out of the kernel-mode address space on an x64/IA-64 machine and thereby cause a crash.

There's nothing wrong with 32-bit relative displacement though. Perhaps you're referring to this?

...
fffff880`014e7f3e 4885f6 test rsi,rsi
fffff880`014e7f41 0f8413840100 je ndis+0x1a35a (fffff880`0150035a)
fffff880`014e7f47 48893e mov qword ptr [rsi],rdi
...

That conditional jump at '41 uses a relative offset - relative to the current position. As per the Intel doc you quoted, the 'je' instruction includes a relative ("rel") 16/32-bit pointer:

0F 84 cw/cd JE rel16/32 Jump near if equal (ZF=1).

In the OP's case, the entire instruction is "0f8413840100". Hence, the leading "0f84" is the opcode and "13840100" is the relative offset in little-endian, so 0x18413 to me and you.

The starting point for that displacement is the IP as it would have looked after the jump instruction is fetched (note the atomic nature of the IP update):

fffff880`014e7f41 0f8413840100 je ndis+0x1a35a (fffff880`0150035a)
fffff880`014e7f47 48893e mov qword ptr [rsi],rdi

Adding the 'je' instruction's relative displacement to the IP yields:

0: kd> ? fffff880`014e7f47 + 0x18413
Evaluate expression: -8246315187366 = fffff880`0150035a

Let's now get the load address of NDIS.SYS as a starting point...

0: kd> lmm ndis
start end module name
fffff880`014e6000 fffff880`015d7000 ndis T (no symbols)

If we compare the displacement between the jump's destination and the NDIS load address:

0: kd> ? fffff880`0150035a - fffff880`014e6000
Evaluate expression: 107354 = 00000000`0001a35a

So the final jump destination would be NDIS+0x1a35a, as per WinDBG's calculation highlighted in pink up higher above. To me, it all seems to check out.

chuckr said:

Does your 7100 with that ndis.sys operate correctly?

Yes, to the best of my knowledge. It certainly doesn't BSOD.

chuckr said:

Since you like this stuff, is there any way that you could execute this "chunk" of code under 'controlled' circumstances?

Sure, no problem. I can run it under a kernel debugger and thus control its operation. If you like, you can drive. Let me know what breakpoints you'd like set and I'll paste back the results. Good geeky fun