You have to ensure that you have 64-bit drivers
for all of your hardware that you have in your computer to properly use a 64-bit operating system. While a 64-bit system can run 32-bit software...it cannot use 32-bit drivers.
If you have 4GB of RAM or more..you need 64-bit to access all of it. From a performance perspective, unless you are using native 64-bit software applications...the likihood of seeing really noticeable differences is pretty slim.
At work, I run 32-bit on my laptop with 4GB of RAM...as I rely on the Checkpoint VPN client to remotely connect to our VPN server at work and that client doesn't support 64-bit. On my work desktop, I also have 4GB of RAM and run 64-bit there since I don't have to VPN in from that machine. Neither machine for the most part perform better or worse then the other.