No. They are "stored" on the hard drive, but unless a file is needed it's never accessed by the hard drive. Thus it's not interfering with performance. That sort of goes along with the FALSE theory that constantly accessing the computer's memory (RAM) will wear it out.
However, a highly fragmented drive can very slightly slow down performance. If, for example a word processing document is fragmented and parts of the document in several locations on the hard drive, instead of just accessing one location to get the document, the drive must access several locations. Periodic defragmentation of the hard drive will take care of fagmented files. But even with a drive that is very fragmented with the speed of modern computers it may not even be noticed.