I disagree that a SSD would be the best performance upgrade option for you at this starting point. No doubt SSDs boost performance (I personally see no reason for HDs when building new or upgrading). But SSDs only affect "drive" performance and "computer" performance is much more than just reading and writing to a drive. You see amazing boot times with SSDs compared to HDs, but once everything is loaded into RAM, the advantages (for most tasks) of a SSD diminish significantly - but ONLY IF you have a bunch of RAM. You don't.
You have only 4GB of RAM. That is, by far, your biggest (narrowest?) bottleneck right now. Since you already have 64-bit Windows (which can support more than 4GB), and your Gigabyte board will support 4GB sticks, I recommend you replace your current RAM with 2 x 4GB, upgrading to 8GB. Replacing all will ensure total compatibility.
You did not specify how your RAM is configured now. If you currently have 1 x 4GB, you might be able to add a second 4GB if you can find a matching stick. But again, to ensure compatibility, replacing all is best. Note most of us who have been doing this for awhile have a box or drawer full of perfectly good RAM because replacing all instead of adding was the best option. For around $50 for 2 x 4GB, this will BY FAR, give you the most bang for your money.
The other advantage to just upgrading your RAM is your current power supply will most likely support the increased demand for power so you will not need a new PSU. Note a graphics card can easily be the most power hungry devices in our computers, even more so than the CPU. So upgrading the graphics card often requires a more powerful PSU too.
You can replace your CPU, but that will cost considerably more than $50, and you will still have that 4GB bottleneck. A new CPU may also require a new PSU.
Replacing a motherboard (which may then require a new compatible CPU and new compatible RAM - and a new PSU to support them) will be much more expensive. But also a new motherboard, in terms of software licenses, is considered a new computer. That typically means you will need to buy a new Windows license too since the vast majority of Windows licenses are OEM/System Builder licenses and OEM are NEVER legally transferable to a new computer under any circumstances.