Seems that there's always a handle on that device when I plug in.
The reason is probably because you once removed it when it was still "in use". Now the system remembers that the thing is "in use". Usually that should not happen because when you shut off, all handles should be removed. But who knows what happened here.
I would not worry about it. If you have no data in the buffers that could get lost, there is no danger. And buffer data is written out in no time (within seconds at most).