Solved Random BSOD in ntoskrnl.exe

[Tsunamijhoe]

Hey Guys

I was hoping you could help me. My new 20 drive windows 7 computer keeps BSOD on me at random times,but always when i play video off of it (computer is a video file server)

Enclosed are the files requested and gathered from your tool

I hope you can help me find the culprit

Thank you

 

[Tsunamijhoe]

Could anyone help please?

 

[koolkat77]

What is "new 20 drive" ?
Disable driver verifier.

• Driver Verifier - Enable and Disable

 

[Tsunamijhoe]

It is a 20x2TB harddrive file server.

Drive verifier was enabled to record if any driver was the reason of the BSOD

 

[koolkat77]

The dump file was blaming windows files. Which cannot be a true cause.
Disable verifier and upload the crash dumps you get.
You may enable verifier if required again, later.

 

[Tsunamijhoe]

Problem is, the bsod happens at random, and i have no way on knowing when a bsod will occur, however, it is ONLY the latest dmp file that had drive verifier enabled, all the other dmp files should hold the bsod info without verifier enabled..

I will disable verifier and then hope it wont be weeks until next bsod

 

[Tsunamijhoe]

OK, after turning off verifier,and running the server for about an hour,with absolutly no activity,meaning i wasnt accessing any files on the computer in question,it bsod´d again..

Enclosed are the latest dmp file,with verifier off

 

[koolkat77]

Very strange way of crashing. Anyway, some info from the dump file shows that it's crashing on the newest version of on board graphics. How old is this computer? Check for broken or bent CPU pins. Post a screenshot of the summary tab using Speccy and of your hard disk using crystal disk info:

• Speccy - System Information - Free Download
• CrystalDiskInfo - Software - Crystal Dew World
• http://www.sevenforums.com/tutorials/9733-screenshots-files-upload-post-seven-forums.html

kd> !irp fffffa8009912aa0
Irp is active with 4 stacks 1 is current (= 0xfffffa8009912b70)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
>[ 16, 2]   0 e1 fffffa8007b8a680 00000000 fffff88001bc57d0-fffffa800866c5a0 Success Error Cancel pending
	       \Driver\atapi	CLASSPNP!ClasspStartNextPowerIrpCompletion
			Args: 00000000 00000001 00000004 00000000
 [ 16, 2]   0 e1 fffffa800866c060 00000000 fffff88000e45f50-00000000 Success Error Cancel pending
	       \Driver\Disk	partmgr!PmPowerCompletion
			Args: 00000000 00000001 00000004 00000000
 [ 16, 2]   0 e1 fffffa800866cb90 00000000 00000000-00000000    pending
	       \Driver\partmgr
			Args: 00000000 00000001 00000004 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-fffffa8006e0b1d0    

			Args: 00000000 00000000 00000000 00000000
.................
kd> lmvm igdkmd64
start             end                 module name
fffff880`04ab8000 fffff880`04fd0960   igdkmd64 T (no symbols)           
    Loaded symbol image file: igdkmd64.sys
    Image path: \SystemRoot\system32\DRIVERS\igdkmd64.sys
    Image name: igdkmd64.sys
    Timestamp:        Thu Sep 27 01:49:04 2012 (50635C30)
    CheckSum:         0051C6B0
    ImageSize:        00518960
    Translations:     0000.04b0 0000.04e4 0409.04b0 0409.04e4

Did you reinstall Windows after making these changes? Does the BSOD occur in safe mode?

 

[Tsunamijhoe]

Hey koolkat

The cpu has no broken or bent pins, all the hardware are 5 months old or thereabouts.

I have NOT re-installed windows after turning verifier off,and i have no idea if it crashes in safemode,as i never run the rig in safemode,i guess i could try that,safemode with network enabled so i can access the files still..

I already use crystaldisk,it monitors all my 20 harddrives,as does Flexraid which emails me whenever there is a SMART error

I will report back once i run speccy.

And thank you very much for helping me

 

[Vir Gnarus]

Can you make sure your:

• Bios is up-to-date.
• Chipset drivers are up-to-date.
• Storage controller drivers/firmware are up-to-date.
• Backplane firmware is up-to-date.
• Integrated graphics drivers are up-to-date.

Either of those look to be a potential suspect here. However, right now it's just speculation on what info we have. We'll need you to turn on Driver Verifier again, let it crash, then send us a kernel dump (MEMORY.DMP in Windows directory). Obviously it'll be big, so archive it and upload to 3rd-party filesharing site (like Mirrorcreator.com).


Oh, and make sure your power options for your server are set to prevent any type of sleep, and make sure hibernate is disabled (type powercfg -h off in cmd prompt).



Analysts:


As koolkat discovered in the raw stack, integrated graphics controller and DirectX are present:

...fffff800`0453a8c8  fffff800`0453a970
fffff800`0453a8d0  fffffa80`08dfa010
fffff800`0453a8d8  fffff880`0456dc9e dxgmms1!VidSchDdiNotifyDpcWorker+0x36
fffff800`0453a8e0  00000000`00000000
fffff800`0453a8e8  fffffa80`08dfa7b8
fffff800`0453a8f0  fffff800`0453a920
[COLOR=Red]fffff800`0453a8f8  fffff880`04d7b0f8Unable to load image \SystemRoot\system32\DRIVERS\igdkmd64.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for igdkmd64.sys
*** ERROR: Module load completed but symbols could not be loaded for igdkmd64.sys
 igdkmd64+0x2c30f8[/COLOR]
fffff800`0453a900  fffff800`04534080
fffff800`0453a908  00000000`00000022
fffff800`0453a910  00000000`00000000
...

But strangely, the actual halted power IRP has something to do with storage:

0: kd> !irp fffffa8009912aa0 1
Irp is active with 4 stacks 1 is current (= 0xfffffa8009912b70)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
Flags = 00000000
ThreadListEntry.Flink = fffffa8009912ac0
ThreadListEntry.Blink = fffffa8009912ac0
IoStatus.Status = 00000000
IoStatus.Information = 00000004
RequestorMode = 00000000
Cancel = 00
CancelIrql = 0
ApcEnvironment = 00
UserIosb = 00000000
UserEvent = 00000000
Overlay.AsynchronousParameters.UserApcRoutine = 00000000
Overlay.AsynchronousParameters.UserApcContext = 00000000
Overlay.AllocationSize = 00000000 - 00000000
CancelRoutine = 00000000   
UserBuffer = 00000000
&Tail.Overlay.DeviceQueueEntry = fffffa8009912b18
Tail.Overlay.Thread = 00000000
Tail.Overlay.AuxiliaryBuffer = 00000000
Tail.Overlay.ListEntry.Flink = fffff80003028b00
Tail.Overlay.ListEntry.Blink = fffff80003028b00
Tail.Overlay.CurrentStackLocation = fffffa8009912b70
Tail.Overlay.OriginalFileObject = 00000000
Tail.Apc = 00000000
Tail.CompletionKey = 00000000
     cmd  flg cl Device   File     Completion-Context
>[ 16, 2]   0 e1 fffffa8007b8a680 00000000 fffff88001bc57d0-fffffa800866c5a0 Success Error Cancel pending
           \Driver\atapi    CLASSPNP!ClasspStartNextPowerIrpCompletion
            Args: 00000000 00000001 00000004 00000000
 [ 16, 2]   0 e1 fffffa800866c060 00000000 fffff88000e45f50-00000000 Success Error Cancel pending
           \Driver\Disk    partmgr!PmPowerCompletion
            Args: 00000000 00000001 00000004 00000000
 [ 16, 2]   0 e1 fffffa800866cb90 00000000 00000000-00000000    pending
           \Driver\partmgr (IopUnloadSafeCompletion)
            Args: 00000000 00000001 00000004 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-fffffa8006e0b1d0    

            Args: 00000000 00000000 00000000 00000000

Major code is 16 = Power IRP, minor is 2 = remove device. I'm not sure why it's trying to remove device. Maybe there's a bad drive that's compromising the integrity of the system? Not sure. Either way, as you can tell, all current IRP stacks are pending, which means they're waiting for something else to do its job before they can do theirs. That means some other IRP is holding things up. We cannot determine what that is without a kernel dump.

 

[Tsunamijhoe]

OK, i might know what it is if it is indeed related to device error..

I have had a known problem with my supermicro sata controller, where is dismounts drives after idle,sometimes it wont even mount certain drives (seems to always be drives 5 and 10) on boot up,could this be the cause?

A lot of people over at limetech (unraid) have reported this problem with the supermicro card,and it is annoying as hell,however, they report one should refrain from updating the firmware on the card,as problem will get worse.

Since i also use 5x2 port sata asrock cards i can mix and match drives and try and find a configuration where there are no conflicts,however i have been unsuccesful in this endevaour..

Mind you it can run for days before this happens,and then a reboot usually fixes it.

Also,i have completely removed the intel gfx driver,as i have no use for it,usually i run server headless,so i will try and see if that fixes anything before i do anything else

 

[Vir Gnarus]

Yes, now that I think about it, I have heard of some issues with unraid with Supermicro controllers, though I never gave it much thought since unraid did not fit what I was looking for in enterprise storage. Are you using the controller on a Supermicro mobo or a standalone PCI-E controller card? At least judging by the crashdumps it looks like you're using a controller card. What's the model for your mobo and/or card?

Either way, it looks like a conflict issue that's only going to be resolved by either Supermicro or limetech fixing the problem. There may even be the need to find another solution besides unraid/supermicro if this issue is going to persist.

 

[Tsunamijhoe]

Yes, now that I think about it, I have heard of some issues with unraid with Supermicro controllers, though I never gave it much thought since unraid did not fit what I was looking for in enterprise storage. Are you using the controller on a Supermicro mobo or a standalone PCI-E controller card? At least judging by the crashdumps it looks like you're using a controller card. What's the model for your mobo and/or card?

Either way, it looks like a conflict issue that's only going to be resolved by either Supermicro or limetech fixing the problem. There may even be the need to find another solution besides unraid/supermicro if this issue is going to persist.

See "my system specs" for hardware i use,it is accurate and up to date.

However,i do NOT use unraid, but flexraid,same difference i guess when it comes down to it.. Problem is, i bought the asrock mb because it had so many pci-e expansion slots,as i figured i would gradually add to my server,but as it turns out,i have max´ed out the server capacity in just 4 months,so i should probably have bought different hardware,but it is what it is..

Problem is, short of buying brand new controllers and mainboard, there isn´t any other 8 port sata controller on the market that i could find,with my setup i have 4xsata ports on mb, 10xsata ports on 5xasrock sata controllers, and 8 ports on the supermicro, i could go for a different 6 port controller,but havent found one..

you see my predicament?

I havent had BSOD since i removed the intel gfx driver though, but as topic says,it is totally random when it occurs, and thus i might have to wait a few more days/weeks to see if removing the driver that is part of the crash has fixed it

 

[Vir Gnarus]

Yes, I do. I've been reading more reviews on this particular Supermicro controller and it looks like it's very iffy especially with unRAID and flexRAID as well as poor Linux support (not that the Linux problem matters for you). If you wanted to go any higher it would mean getting an enterprise level card, which means getting an enterprise-level mobo and enterprise-level SATA drives. Obviously it can get quite more expensive.

People in the reviews were talking about unRAID and flexRAID and how it needed INT13 setting changed on the controller. I'm not sure if it involves your situation but I figured it'd be worth mentioning. Also some people were encouraging firmware update for the card, but if you said other were saying otherwise, then I'm not sure if it's worth the gamble. Worst case, though, would be replacing the card, so best to have a spare on hand in case the update bugs out this one.

I know you've been using CrystalDiskInfo and whatnot to keep a tab on your drives' health, but I do think it best to actually run some sort of diagnostics on em just in case. Not sure if Seatools would work but worth a shot. I am well aware of the odd behavior surfacing from a bad drive or two.

If you want you can give us that kernel dump and we can take a further look at it.

 

[Tsunamijhoe]

Thanks Vir

But buying another card of the same brand/make/model would be pure stupidity,as it is the card or firmware that is at fault,and no other affordable 8 port card is on the market (no way am i paying 300+$ for a controller),so i guess i will have to suffer until other manufacturers will make a similar card..

Also, it isn´t the disks,as it happens regardles of which of my 20 drives are connected to the supermicro,so unless all 20 drives including 4 brand new WD20EARX are faulty,it is the card.

Also, NO drives connected to the mb or to any of the 5 asrock sata controllers does ever exhibit this behavior..

In summary, i have deleted the gfx driver in question,and have uninstalled all non-essential software,i have just installed an older driver for the supermicro,as the one i was using wasnt specific for my card,but for the newer 2 card,although reports indicate that the driver i was using should work fine,and even allow 3+TB drives to be connected,but for the sake of trying everything,i installed the "correct" driver for my make/model for now.

I am gonna let it run a week and see if it BSOD,if not problem solved (removing the intel gfx driver),if it does indeed BSOD, i can make the dump files and upload them if need be

Last resort would be to remove all my WD20EARS (load cycle problem) drives from the supermicro,and either exchange them with blacks or the newer WD20EARX, which is NOT affected by the load cycle problem according to Western Digital customer service,and hope that will fix it..

Now i just need to be patient,and hope for the best

 

[Vir Gnarus]

You may not see another card like that for a good while in the consumer market, since a consumer level card with that much isn't really something with large demand. Anyone who's going for such is usually going more of a business level, which means they'd be willing to shell out $300 for a solid controller. Supermicro is not to blame, as itself is one of the best in the business, and I know that just from expert opinions and my own personal experience. But like any brand, reputable or not, if you dip into the discount bin, you're going to get discount quality.

Now I've actually been giving this some extra thought and re-reviewing it, and it just dawned on me. The load cycle issue with idle drives would be exactly what may be causing this. You see, built into drives is a time limit to discern if a drive is inactive or not and will put it into a low idle power state. This is to conserve energy, but is contradictory to RAID controllers which are constantly checking for drives to see if they are alive or not as a health check. If the drive does not respond to the call in a designated time frame it will deem that drive as bad and unload it. This isn't a problem for drives usually, but when a drive decides to become idle, pulling it out of the idle state to respond to the heartbeat will take just enough time to miss the threshold. I've seen this happen often with people who mix consumer drives with business/enterprise class controllers because the consumer drives have a smaller idle threshold to place itself into a low power state, whereas enterprise class drives have a much longer or even nonexistent threshold. A lot of newer consumer drives are also going this route because they're seeing that people are using RAID controllers a lot more often which are asking for a much faster response time than what their inert drive can respond too.

Now, this seems to correspond with what I interpret is going on with the bugcheck. The system is ordering the removal of a disk device, most likely because of it missing the heartbeat, and the disk is saying, "You can't do that, I'm alive!" So it is rejecting the request. That sounds very much inline with what you're experiencing with the load cycle problem.

I say, if the BSODs continue, definitely try out those WD20EARX drives. That may very well be the answer we're looking for.

 

[Tsunamijhoe]

Thx m8, and yes,i am gradually exchanging to WD20EARS drives for WD20EARX drives anyways, but all of the problem drives, i have run WDIDLE on to disable the head parking completely, and i have windows set at 20 mins for harddrive sleep timer,so the drives shouldnt go into an idle state,however WD has shipped 2 types of WD20EARX and i know contrary to what WD says,that WD20EARX-008FB0 also suffers from the head park,so unfortunatly i only own 3 of the brand new type with NO idle park the WD20EARX-00PASB0,but i still have to exchange 6 of the problem EARS type,and this will free the asrock controllers,so i can move the idle problem EARX Drives to the asrock controller,and only use the brand new on the supermicro card.

That said,i only run snapshot raid,not real time,so the headparking again shouldnt be a problem, but i dont know

 

[Vir Gnarus]

It doesn't matter what RAID or no RAID setup you have. You can even have a JBOD and it'll still bug out the controller if the heartbeat isn't responded too in quick fashion. Don't forget that the controller itself may be designed to expect enterprise drive configuration and not consumer, despite what the price and whatnot may seem (expected given that Supermicro is an enterprise hardware company).

 

[Tsunamijhoe]

I think i mght have fixed the problem, 2 days without a drive dismount,and no bsod.. which is much better than usual. i usually have at least 1 drive dismount a day,so none in 2 days is good.. However,too early to say if it is completely fixed..I will let it run a week and see if anything happens

 

[Vir Gnarus]

What'd you accomplish, just moving the WD20EARS drives to the mobo storage controller?