Up to a point it matters. For example, if your computer hard drive is nearly at 100% capacity...so you will be dealing with a lot of extra fragmentation and such.
However, if you have a 500GB hard drive and are only using 35GB...It's not really going to change performance if you go from 460GB free to 430GB free of even 350GB free.
So, I would not go to any extraordinary lengths to keep my hard drive as clean as possible in hopes of seeing increased performance.