Windows 7: External Hard Drive not safely removable

I have a Dell Vostro 400 using onboard Intel G31/G33 graphics. When I log into admin user and try to remove my Western Digital My Passport it gives me a message that it could not be removed because a program is using it. I ran Process Explorer and the processes running on the drive were all system processes. I tried running my antivirus on both the pc and the hard drive. I tried running WD Drive Utillities and did a quick self test and SMART test and both passed. I tried plugging other devices on the same port but they did not have problems. Also, if I insert the USB plug on a different port, use a different user, or plug in on my laptop it does work fine.

Note that this happens only sometimes and most of the times after I read or write on the drive it happens.
Should I worry about this or just shut down and remove the drive? It does not seem to be writing or reading anything on the drive because the indicator light is not blinking.

Externals giving that message is quite common. Many of us have had a similar problem. It is really just a pain and not a problem to be worried about. I would just turn it on when I need it and when you need to remove it, just shut down. Is your sata controller in AHCI mode? If it is, that is one of the features of AHCI, drives being hot swappable. I had a similar problem once even in AHCI mode (a BIOS setting), I installed Intel Rapid Storage driver and have not had the problem since. If you are not in AHCI mode, don't change it or your computer won't boot. There is a way to change it and if you need to know how, we will give you the link to a tutorial. But, unless you have an SSD, it really makes little difference.

Ok, so there isn't a big problem if I don't care about shutting down to take it out right?

Right. Sometimes, it just happens that way. I used to have to do it all time. It does not mean there is anything wrong with the hard drive.

Also, you may find this helpful. Read the bottom of the tutorial, it will give you an option to try.
Safely Remove Hardware - Eject Device

Thank you, so are you sure its safe? It has all my photos and I don't won't to corrupt anything.

I don't see a thing that could harm your drive. You are just resetting settings on the computer. It has nothing to do with the drive.

You do have your photos backed up, right?

No I mean my external hard drive has the photos. And I mean if it is safe to just eject the drive when I'm using it right?

And I did try that essenbe. Maybe I should try using sfc /scannow and let Windows check if there is any corrupted DLLs.