GPU having very weird fan usage fluctuations on idle.

So basically my gpu is the exact same one displayed in the video and is doing the same exact thing. (Randomly going to 4000RPM and back to 0 RPM for a few seconds)
Note that this only happens WITH drivers installed and only when not on load, when the gpu is on load the problem goes away and the fan acts normally.

Video: https://www.youtube.com/watch?v=k4TuvATjAT0

Fan graphs: https://imgur.com/a/iUffX

My specs:
GPU: GeForce GTX 750 Ti
CPU: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz
Memory: 8 GB RAM (7.94 GB RAM usable)
Current resolution: 1920 x 1080, 60Hz
Operating system: Microsoft Windows 7 Ultimate

You could download and run Speedfan (http://www.almico.com/speedfan.php). Speedfan is primarily to check the temperatures inside of the computer. But it also lets you set the fan speed.

Download and run Speedfan, but don't make any changes yet to the fan speed. At first, look at it to see what the temps are when the fan is running fast vs when the fan is running slow or not running. Then make a decision about what speed to set the fan on.

It may be that the computer runs hot until the GPU fan kicks in; in other words, the GPU fan may cool the GPU and the computer. If that is what is happening, then that could explain the erratic behavior of the computer's fan.

You can have the fan running all the time; that shouldn't hurt anything, except that the fan noise may bother you.

If I wasn't sure what speed to run the fan, I would start it off at 75% speed and then watch the temps. Then make adjustments from time to time, till the temps stay cool enough.

It's possible that your fan is going bad; but since it works without any problem once the GPU is on load, I doubt that this is the case.