I tried stressing this drive to see if I can reproduce the freezes in any other way. I ran WinDirStat on the drive and it worked without freezing the computer. I then also ran Crystal Disk Mark without any issues as well. Then I ran them together without any problems.
Windows Backup should simply be reading the data from the drive like any other program so I'm not sure why stressing the disk with other programs doesn't produce the same issues. What's special about Windows Backup?