If there is no reason to worry about doing a clean install on hardware swaps, then why be concerned with cleaning up old drivers
You can not be sure that you got every bit of old drivers entries removed completely as to not be of concern at some point.
Space is not really that big an issue any longer.
It is always advised, that in the long run, it is better to clean install when swapping Motherboard and CPU due to different chipsets etc.
If/when in the future one does have an issue, no one, I do not care who they are or where they come from or what their background is, can guarantee that driver
conflicts would absolutely not be the cause.
The only things not of real concern ,swaps of the following , RAM, HDD, Optical Drives, Sound Cards (usually, but not always) and Video Cards (usually, but not always), other misc devices.