What kind of performance drops are you anticipating? Personally, I'd rather give up a couple hundred megabytes of RAM (out of the 8GB on my machine), or 10% of my CPU for real-time monitoring, than to go through the hassle of getting infected, using a previous system image, updating things, etc.
The good thing about AVs today is many of the recommended products like MSE, Avast, Avira, Comodo, etc are totally free. You could always try one for a week or two, see how it actually performs on your machine, and then decide if you want an AV or not. Just remember to use an official product uninstaller to remove all traces of the AV product before installing another one. Security Software | SingularLabs