Just so you're aware, Driver Verifier can easily cause slowdown on a system since it performs many more operations to institute checks on your drivers. So whenever it's active, expect the system to be a bit more sluggish.
I have found the exact issue, which is a solitary changed byte to change an instruction in the kernel code to a backwards compatible variation. Judging by this, I don't believe this is hardware we're dealing with but some driver playing foul. Most common instances of this situation involve any sort of sandboxing or VM (virtual machine) software. In addition, motherboard software, chipset drivers, and perhaps even the BIOS can be involved.
Unfortunately PatchGuard on Windows - the component that tracks kernel code for any unsuspected modifications - seems to only trap when it's come across a modification,
not from what caused the modification in the first place. This in all honesty is pretty stupid, but that's just how it is. So I'm afraid we're still unaware what exactly is causing this, but at least now we can trim it down to most suspect as software, with the aforementioned list in the previous paragraph being a good start. You'll want to do some late spring cleaning, which should involve cleaning off any motherboard software (all of it), updating your motherboard drivers and BIOS, and checking any software that has anything to do with sandboxing or VMs, such as certain antivirus software like Sandboxie (which should be uninstalled if present).
Another possibility, I venture to guess, is what triggered the modification could've been the processor bugged out somehow when trying to perform the 'correct' instruction, and some application responded by altering the code to make it backwards compatible with CPUs that would
typically bug out on said instruction, whereas this would've been just a one-time occurrence. I don't suspect this is the case as it all looks way too consistent for that, but you are still free to perform the hardware tests.
Analysts:
Found
this article on 109 bugchecks, and decided to go for it. Since the client is using Windows 7 SP1 and an AMD64 system, I downloaded and extracted the SP1 installation package and pulled out the modified
ntoskrnl.exe from it. Then I opened it with Windbg (not by "Open Executable" but by "Open Crash dump"). I already know the name of the symbol I'm looking for, the one that crashes all the time, and its offset, all of which is
ExpReleaseResourceForThreadLite+0x53, so I did an unassemble on it (with the
u command), and then did the same with the same block of code in the client's crash dump, and juxtaposed the two, as followed:
SP1
ntoskrnl.exe:
Code:
ntoskrnl!ExpReleaseResourceForThreadLite+0x53:
00000001`400829d3 4c897c2468 mov qword ptr [rsp+68h],r15
00000001`400829d8 0f[COLOR=Red]0d[/COLOR]4b18 [COLOR=Red]prefetchw[/COLOR] [rbx+18h]
00000001`400829dc f6431a80 test byte ptr [rbx+1Ah],80h
00000001`400829e0 0f85ba000000 jne ntoskrnl!ExpReleaseResourceForThreadLite+0x120 (00000001`40082aa0)
00000001`400829e6 41bf01000000 mov r15d,1
00000001`400829ec 48396b30 cmp qword ptr [rbx+30h],rbp
00000001`400829f0 0f8522010000 jne ntoskrnl!ExpReleaseResourceForThreadLite+0x198 (00000001`40082b18)
00000001`400829f6 488d7b30 lea rdi,[rbx+30h]
Client's crash dump:
Code:
nt!ExpReleaseResourceForThreadLite+0x53:
fffff800`0407c583 4c897c2468 mov qword ptr [rsp+68h],r15
fffff800`0407c588 0f[COLOR=Red]1f[/COLOR]4b18 [COLOR=Red]nop[/COLOR] dword ptr [rbx+18h]
fffff800`0407c58c f6431a80 test byte ptr [rbx+1Ah],80h
fffff800`0407c590 0f85ba000000 jne nt!ExpReleaseResourceForThreadLite+0x120 (fffff800`0407c650)
fffff800`0407c596 41bf01000000 mov r15d,1
fffff800`0407c59c 48396b30 cmp qword ptr [rbx+30h],rbp
fffff800`0407c5a0 0f8522010000 jne nt!ExpReleaseResourceForThreadLite+0x198 (fffff800`0407c6c8)
fffff800`0407c5a6 488d7b30 lea rdi,[rbx+30h]
There's a good bit of material on the difference between using the
nop instruction and its variations and
prefetch/prefetchw. This kinda sheds a bit of light on it generally speaking, but it's all more-or-less a deal with CPU compatibility. It's also the same kind of manipulation that's mentioned of in the Ntdebugging blog I referred to previously, however the function (and possibly the application involved) is different.