"Safe to remove" really means that all cached writes have been written to the disk. When I encounter the "cannot remove ..." situation, I ensure that no open applications are obviously using the disk and there is no disk activity, then simply disconnect. I have never lost any data on an external drive, memory card or thumb drive from doing this.
I have heard people say they never even bother to attempt the "safe eject" and just disconnect the device. I think that is going a bit too far and they may get burned by that procedure.