Do you have Write Caching turned on? That will hold a drive until the buffer flushes. You can set USB drives to quick removal in Device Manager > Disk Drives. Right click the drive and select Properties. Click the Polices tab and chose between Performance (what you probably have now) and Quick Removal.
The thing is, you do have to make a choice.