It will do that the first time you plug it into a new USB slot. XP did that too, if you unplug and replug in the same slot it shoult not go through that all over again. Sould be just a few seconds.
Though if there is somthing a little odd about that card reader that seems to require a special driver
that windows 7 does not have and never actually installs, then I could see that it might re-try every time before defaulting to something generic.
I see this a lot with thumb drives. There are a lot of them that try to be "special" and require a special driver whereas most of them just advertise as a generic USB mass storage device and have no problems.