/* EDIT */
I thought I should add this intro for context: There's a variety of opinions/approaches in the thread about backing up drivers. But to talk about the question of driver backups and approaches, the term "driver" (and just what software it includes) should be addressed first
/* End EDIT */
The term "driver" is used loosely. Strictly speaking, the term "driver" may refer to a "driver package" and sometimes it refers to a "Device Installation App". Here's the difference
Driver Package
A "driver package" is the set of files that allow higher level apps to talk to device hardware. (This what we usually think of as "the driver".) The driver package includes more then just software files. Driver package files include "meta data", things like:
> .cat (driver catalog) files. The cat files contain an inventory to all the pieces to the package as well as the driver's digital signature (that's how windows knows what's included and if anything has changed)
> .inf (Installation Setup Files). INF files instruct windows how to install the driver - including any add/modify/delete registry key changes required as part of the install
p.s. Based on what i've seen, control panel applets are usually included in the driver package as well
Tools like DoubleDriver (I use it myself

) backup the "driver package" set of files for each device installed on your computer.
Device Installation App
Here's where the confusion comes in.... When we download "drivers" from vendors, you're typically downloading the "Device Installation App". This app always includes the driver package BUT it can also include higher level software from the vendor. A good example is Catalyst Control Center (CCC). When you download your AMD graphics "driver" - you're downloading their Device Installation App which installs application level software (like CCC)
in addition to installing the driver package.
When you install using an EXE file, the EXE might only contain the "driver package" embedded within or it might contain both the driver package plus app level software.
When you install from an INF file, you're only installing the driver package.
Tools like DoubleDriver back up your driver package files. That guarantees you can at least get your hardware up and running from the backup. Then you can go looking for a Driver Installation App if remember there was any application software included in it that you wanted to re-install.
So saving your EXE is still a good idea. But IMO backing up your drivers (i.e. driver packages) is a must, first step