Windows 7: Front panel usb disconnects all devices when removing just 1 pen drive

I have 4 front panel USB ports which all work fine and i have double checked to make sure i they are connected correctly on MOBO.

The problem i have is, if i have say my PS3 control and my phone connected then use say a pen drive (but not limited to the pen drive). I add/delete what ever to and from pen drive then eject. On removing the pen drive one of the other devices disconnects.

I presume its the port that is connected on the same set of pins as the port the pen drive is connected to. Is this a problem that i have never noticed before on a PC or is it a problem that shouldn't be happening?

P.S. I'm sure i have most up to date drivers but will check again and let you know.

Thank you