I strongly advise you to get a anti-virus program. You never know where viruses may come from. Even harmless emails from your mom could contain something.
As long as you are physivally connected to the internet, you are not safe. The anti-virus programs only reduce the risk, never eliminate it.
If you get a keylogger, and become a victim of identity theft, you will kick yourself for a long time. The security is well-worth the performance impacts.
I recommend Microsoft Security Essentials, as other have recommended. It barely affects performance, and it should bring considerable peace of mind.
That said, you are on to something; common sense is by far the best security system. No anti-virus can take its place.