Unreproducable freezes

[j3anders]

I have a Windows Embedded Standard 7 (Basically Win7 with easily removable components and custom branding) installation that has frozen randomly on me a few times. We have three prototypes all running the same image, and have seen two freezes over the last month. We aren't sure if it was the same unit each time, but now we're keeping track of it.

The system locks up completely - the screen just freezes on whatever ti was doing. We had a mouse and keyboard hooked up to one and the cursor freezes and the numlock light doesn't toggle. There is nothing related to it in the event log after a reboot.

The non-windows drivers we are using (and were loaded at the time) are for four FTDI USB-UART converters, an Elo serial touchscreen and *cough*15DriversThatNationalInstrumentsInstalls*cough*

Any tips on how I can track down what is causing the problem? The issue is too rare to be able to test by uninstalling different drivers and waiting for a crash. Some of these things will be hooked up to hardware that can open a liquid nitrogen valve and is designed to run unattended, so random freezes are a bad thing.

 

[cybercore]

The non-windows drivers we are using (and were loaded at the time) are for four FTDI USB-UART converters, an Elo serial touchscreen and *cough*15DriversThatNationalInstrumentsInstalls*cough*

Could you clarify on what non-windows drivers you are using?

Freezes can also be caused by insufficient Power Supply, see if you have enough watt.

 

[soho1]

My experience with Windows 7 freezing is in ths post:
http://www.sevenforums.com/crashes-debugging/115707-freeze-whats-hang-up.html
See if the behaviour is similar. The secret patch worked for me.

 

[j3anders]

Install is 32-bit.
The FTDI drivers are here: Virtual COM Port Drivers
EloTouch (XPU4.8.7b.exe): Driver and file downloads - Elo TouchSystems - Tyco Electronics

Those are both WHQL-certified.

We also have everything that a National Instruments NI-DAQmx install throws in there. There are 15 drivers that it keeps loaded even when no NI hardware is being used. I don't know if they are certified or not. If you want to take a look at it, you can get the 1.1GB installer here: NI-DAQmx 9.1.7 - Windows 7 64 bit/7 x86/Server 2003 R2 (32-bit)/XP x86/Vista x64/Vista x86/Server 2008 R2 (64-bit) - National Instruments I installed only the NI-DAQmx component, and MAX because the installer forced me to. Afterwards I uninstalled everything but NI-DAQmx (apparently 40-some components) and turned off the services it installed that aren't related to the one piece of hardware we need the drivers for. (A dependency analysis showed that I only need 38mb out of the 1.1gb, but I haven't tried trimming it down further yet)

With those drivers installed everything works perfectly, except for the incredibly rare freezes.

Power shouldn't be an issue - the system is an Atom-based ITX board. The power supply is custom designed and puts out more than enough juice than the specs call for.

 

[j3anders]

soho1, thanks for the link but I don't know if it will help. I'm hoping to find a way to confirm what is causing the freezes. Installing the hotfix might work (if it lets me install it on WES7), but I've only seen the problem twice in about 3 months of runtime, so it will be tough to say whether the hotfix worked or if I've just been lucky.

 

[richc46]

Check this post
http://www.sevenforums.com/crashes-debugging/120940-windows-7-freezes-up.html#post1040730

 

[j3anders]

Check this post
http://www.sevenforums.com/crashes-debugging/120940-windows-7-freezes-up.html#post1040730

Thanks, the device manager is fine and the systems should be virus-free, one that froze has never been connected to a network. The only USB drive that has been plugged in scans as clean. There is a good chance antivirus won't actually run on it with the way WES7 is configured.

I haven't run memcheck on any device yet but I will try to sometime today.

I'll hook a thermocouple up to the heatsink on all three devices - the software they run should log the temp up till the moment they die and it will give us a decent test for our data-acquisition tools.

Unfortunately, I can't do safe mode/clean boot -- we're doing all sorts of testing with the prototypes and can't afford to have one out of commission for a few weeks, especially on a test that can't definitively prove that the freezing is fixed.

I'll report back to let you know if I learn anything from the ram/heat tests.

 

[j3anders]

Memcheck was clean - I was really hoping that it was the problem since it would be an easy fix. Yesterday we baked the system at 80C and it kept running without a problem, so it isn't looking to be heat related. It would have been fun to push farther but it seems like a bad idea to permanently damage one of our only prototypes. I haven't seen a freeze from any of them since the two I reported in my first post.

I didn't realize that the NI drivers were using compatibility shims, so I added support for that in my base image. I don't know if it will fix the problem or not. We'll be doing a small production run soon so hopefully more runtime will show that the issue is specific to the one unit we've seen freeze.

 

[barqy]

Just found another odd Windows 7 Embedded Standard issue.
Fully patched system as of August 2013...
Unplug USB Keyboard while not touching any keys: no hang
Unplug USB Keyboard while pressing "SHIFT" key: hang

A system with Windows 7 Embedded Standard from 2011 (no recent patches installed) does not hang.

So, seems like possibly a patch went sideways.

(Also an Intel Atom system with FTDI drivers)