Windows 7: Unexpected bsod on server

scottgus1 · 24 Oct 2012

Our server, running Windows 7 Professional 64bit as host for several Virtualbox guests, had a bsod last night, during the backup routine. This is the first time such a thing has happened since the server was put into production this March 2012. The bsod occurred while all the guests were shut down and the main guest, SBS2003, was getting copied.
Could anyone please let me know what happened?

This text was in the after-reset-boot-up popup "Windows has recovered from an unexpected shutdown":

Code:

 
Problem signature:
Problem Event Name: BlueScreen
OS Version: 6.1.7601.2.1.0.256.48
Locale ID: 1033
Additional information about the problem:
BCCode: f4
BCP1: 0000000000000003
BCP2: FFFFFA80541ECB30
BCP3: FFFFFA80541ECE10
BCP4: FFFFF80002DE4510
OS Version: 6_1_7601
Service Pack: 1_0
Product: 256_1
Files that help describe the problem:
C:\Windows\Minidump\102412-14430-01.dmp
C:\Users\RnG\AppData\Local\Temp\WER-22152-0.sysdata.xml

This was in the Event Viewer System log, after the reset & reboot:
System, Error, BugCheck, event ID 1001

Code:

 
The computer has rebooted from a bugcheck. The bugcheck was: 0x000000f4 (0x0000000000000003, 0xfffffa80541ecb30, 0xfffffa80541ece10, 0xfffff80002de4510). A dump was saved in: C:\Windows\Minidump\102412-14430-01.dmp. Report Id: 102412-14430-01.

The zip contains the files from the Seven Forums Diagnostic Tool, as well as the two files mentioned in the after-reset-boot-up popup (these are in the bsod folder in the zip).

Thanks very much for looking into this!

scottgus1 · 24 Oct 2012

I noticed that the system events file in the zip only seems to go up to June this year. Here's an Event Viewer save with the last seven days' system events. The bugcheck error report shows at 10/24/2012 7:12:46 AM.

scottgus1 · 24 Oct 2012

I got the Windows Debugger installed, this is what I got first:

Code:

MODULE_NAME: csrss
FAULTING_MODULE: 0000000000000000 
DEBUG_FLR_IMAGE_TIMESTAMP:  0
Probably caused by : csrss.exe
Followup: MachineOwner
---------
10: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************
CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been
terminated.
Several processes and threads are necessary for the operation of the
system; when they are terminated (for any reason), the system can no
longer function.
Arguments:
Arg1: 0000000000000003, Process
Arg2: fffffa80541ecb30, Terminating object
Arg3: fffffa80541ece10, Process image file name
Arg4: fffff80002de4510, Explanatory message (ascii)
----in between are a bunch of "Your debugger is not using the correct symbols" errors ------
PROCESS_OBJECT: fffffa80541ecb30
IMAGE_NAME:  csrss.exe
CUSTOMER_CRASH_COUNT:  1
DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
BUGCHECK_STR:  0xF4
CURRENT_IRQL:  0
STACK_TEXT:  
fffff880`02dd0b08 fffff800`02e6c892 : 00000000`000000f4 00000000`00000003 fffffa80`541ecb30 fffffa80`541ece10 : nt+0x7f1c0
fffff880`02dd0b10 00000000`000000f4 : 00000000`00000003 fffffa80`541ecb30 fffffa80`541ece10 fffff800`02de4510 : nt+0x406892
fffff880`02dd0b18 00000000`00000003 : fffffa80`541ecb30 fffffa80`541ece10 fffff800`02de4510 fffff800`02dd7b94 : 0xf4
fffff880`02dd0b20 fffffa80`541ecb30 : fffffa80`541ece10 fffff800`02de4510 fffff800`02dd7b94 fffffa80`541ecb30 : 0x3
fffff880`02dd0b28 fffffa80`541ece10 : fffff800`02de4510 fffff800`02dd7b94 fffffa80`541ecb30 fffff800`02e18e8b : 0xfffffa80`541ecb30
fffff880`02dd0b30 fffff800`02de4510 : fffff800`02dd7b94 fffffa80`541ecb30 fffff800`02e18e8b ffffffff`ffffffff : 0xfffffa80`541ece10
fffff880`02dd0b38 fffff800`02dd7b94 : fffffa80`541ecb30 fffff800`02e18e8b ffffffff`ffffffff fffffa80`54417b50 : nt+0x37e510
fffff880`02dd0b40 fffffa80`541ecb30 : fffff800`02e18e8b ffffffff`ffffffff fffffa80`54417b50 fffffa80`541ecb30 : nt+0x371b94
fffff880`02dd0b48 fffff800`02e18e8b : ffffffff`ffffffff fffffa80`54417b50 fffffa80`541ecb30 fffffa80`5440b060 : 0xfffffa80`541ecb30
fffff880`02dd0b50 ffffffff`ffffffff : fffffa80`54417b50 fffffa80`541ecb30 fffffa80`5440b060 00000000`000001d4 : nt+0x3b2e8b
fffff880`02dd0b58 fffffa80`54417b50 : fffffa80`541ecb30 fffffa80`5440b060 00000000`000001d4 00000000`00000008 : 0xffffffff`ffffffff
fffff880`02dd0b60 fffffa80`541ecb30 : fffffa80`5440b060 00000000`000001d4 00000000`00000008 fffffa80`5440b060 : 0xfffffa80`54417b50
fffff880`02dd0b68 fffffa80`5440b060 : 00000000`000001d4 00000000`00000008 fffffa80`5440b060 00000000`00000000 : 0xfffffa80`541ecb30
fffff880`02dd0b70 00000000`000001d4 : 00000000`00000008 fffffa80`5440b060 00000000`00000000 fffffa80`54417b50 : 0xfffffa80`5440b060
fffff880`02dd0b78 00000000`00000008 : fffffa80`5440b060 00000000`00000000 fffffa80`54417b50 fffff800`02d97f74 : 0x1d4
fffff880`02dd0b80 fffffa80`5440b060 : 00000000`00000000 fffffa80`54417b50 fffff800`02d97f74 ffffffff`ffffffff : 0x8
fffff880`02dd0b88 00000000`00000000 : fffffa80`54417b50 fffff800`02d97f74 ffffffff`ffffffff 00000000`00000001 : 0xfffffa80`5440b060

STACK_COMMAND:  kb
FOLLOWUP_NAME:  MachineOwner
BUCKET_ID:  WRONG_SYMBOLS
Followup: MachineOwner
---------
10: kd> lmvm csrss
start             end                 module name

I then found out how to attach to Microsoft's symbol server and got this:

Code:

Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Users\RnG\Desktop\SF_24-10-2012\bsod\102412-14430-01.dmp]
Mini Kernel Dump File: Only registers and stack trace are available
Symbol search path is: SRV*f:\localsymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Kernel Version 7601 (Service Pack 1) MP (24 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7601.17835.amd64fre.win7sp1_gdr.120503-2030
Machine Name:
Kernel base = 0xfffff800`02a66000 PsLoadedModuleList = 0xfffff800`02caa670
Debug session time: Wed Oct 24 02:00:00.782 2012 (UTC - 4:00)
System Uptime: 0 days 23:44:04.781
Loading Kernel Symbols
...............................................................
................................................................
..........................
Loading User Symbols
Loading unloaded module list
..........
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck F4, {3, fffffa80541ecb30, fffffa80541ece10, fffff80002de4510}
Probably caused by : _
Followup: MachineOwner
---------
10: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************
CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been
terminated.
Several processes and threads are necessary for the operation of the
system; when they are terminated (for any reason), the system can no
longer function.
Arguments:
Arg1: 0000000000000003, Process
Arg2: fffffa80541ecb30, Terminating object
Arg3: fffffa80541ece10, Process image file name
Arg4: fffff80002de4510, Explanatory message (ascii)
Debugging Details:
------------------

PROCESS_OBJECT: fffffa80541ecb30
IMAGE_NAME:  _
DEBUG_FLR_IMAGE_TIMESTAMP:  0
MODULE_NAME: _
FAULTING_MODULE: 0000000000000000 
PROCESS_NAME:  ASCServiceCrea
BUGCHECK_STR:  0xF4_ASCServiceCrea
CUSTOMER_CRASH_COUNT:  1
DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT
CURRENT_IRQL:  0
LAST_CONTROL_TRANSFER:  from fffff80002e6c892 to fffff80002ae51c0
STACK_TEXT:  
fffff880`02dd0b08 fffff800`02e6c892 : 00000000`000000f4 00000000`00000003 fffffa80`541ecb30 fffffa80`541ece10 : nt!KeBugCheckEx
fffff880`02dd0b10 fffff800`02e18e8b : ffffffff`ffffffff fffffa80`54417b50 fffffa80`541ecb30 fffffa80`5440b060 : nt!PspCatchCriticalBreak+0x92
fffff880`02dd0b50 fffff800`02d97f74 : ffffffff`ffffffff 00000000`00000001 fffffa80`541ecb30 00000000`00000008 : nt! ?? ::NNGAKEGL::`string'+0x176d6
fffff880`02dd0ba0 fffff800`02ae4453 : fffffa80`541ecb30 fffff880`00000000 fffffa80`54417b50 00000000`00000000 : nt!NtTerminateProcess+0xf4
fffff880`02dd0c20 00000000`77be15da : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000000`0008e318 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x77be15da

STACK_COMMAND:  kb
FOLLOWUP_NAME:  MachineOwner
FAILURE_BUCKET_ID:  X64_0xF4_ASCServiceCrea_IMAGE__
BUCKET_ID:  X64_0xF4_ASCServiceCrea_IMAGE__
Followup: MachineOwner
---------

The first debug says it was csrss.exe, which as I understand will cause a system failure if it crashes. The second debug apparently doesn't know what caused the error. Any thoughts?

**koolkat77** · 24 Oct 2012

F4 is usually storage/hard drive related.

Upload a screenshot of your hard disk using crystal disk info:

Run Disk Check on your hard disk for file system errors and bad sectors on it:

Disk Check

Hard drive test both short and long with:

SeaTools for Windows

Also the DOS version of:

SeaTools for DOS

scottgus1 · 24 Oct 2012

Thanks for the idea, KoolKat!

Here's my disk screenshots. (I'll have to wait to use CrystalDiskInfo until the weekend when no one's working.) I have 3 RAID 1 mirrors, with 6 one-TB drives. I checked about SeaTools: what I saw says that program doesn't work on a RAID, so I'll need to swap out drives on the mirrors to test each individual drive on another computer; fortunately I have a spare 1-TB around here I can use.

I'll also do a consistency check on each mirror, maybe that might show something.

I'll report back...

Vir Gnarus · 25 Oct 2012

This solitary crash looks like it came from a program called Advanced SystemCare. This is one of those apps that are rather iffy at how they accomplish their tasks, so it's preferred that this program not be used. Big names in Windows development and troubleshooting like Mark Russinovich will tell you that registry cleaners and "performance apps" in general aren't reliable and can often cause more harm than good.

scottgus1 · 28 Oct 2012

Koolkat: I installed and ran CrystalDiskInfo and received a "Disk Not Found" error. Guess I should have checked their FAQs (FAQ): "CrystalDiskInfo does not support RAID disks and IDE (Parallel ATA) and Serial ATA disks connected external ATA controller." My 3 RAID1 mirrors, as reported in my System Specs, wouldn't show up.
Do you know if running chkdsk on a RAID1 is good, as in, have you ever done it or received good reports about doing it? I see conflicting reports on the web. I did consistency checks on all three mirrors and received good bills of health on all three. It'll take a bit of time to pop out each drive and rebuild the mirrors, so I might just wait to see if this error happens again. Maybe it's just a cosmic-ray-induced fluke.

Interesting thought, Vir. I'm the only person who runs anything on this computer, I doubt my boss even knows the password to log on without having to consult the notes I left him, and I'm sure he couldn't find them again without my telling him where they are. I know I've never run any form of speed optimization or registry cleaner on it, and certainly nothing called "Advanced Systemcare" - I had to google it to find out what it was

I also have Vipre Antivirus on it, and I scan every download manually, and haven't had any bad bugs get in.
Could you please let me know what indicated to you this software may have been run on this server?

Vir Gnarus · 29 Oct 2012

Easy: check the process name in the !analyze -v output. It'll show you the process that was currently running at the time of the crash in the latest thread context. The process that was running was ASCService Creator.exe (name partially trimmed in output), which googling will show it's part of Advanced SystemCare. Perhaps indeed it was one of those programs that was inadvertently installed through another application's installer.

scottgus1 · 07 Nov 2012

OK, I got what you're showing, Vir. I did a search on the computer and found the ASCServiceCreator.exe. Turns out it's part of the Intel Server control software which I can use to remotely monitor the RAIDs. So the program is OK, just seems to be named the same as that registry malware you warned against. And, since it's the RAID monitor, then it was hard-drive related, like Koolkat said.

Thanks to you both for the help! Sounds to me like a fluke. I'll report back if I get any further data. Thanks again!

Windows 7: Unexpected bsod on server

Unexpected bsod on server problems?