i'd rather have my memory working than not. as long as you do not have any problems, out of memory errors, freezing, bsod's etc., don't worry about memory useage. cpu useage is another story.
the old memory bugaboos from win95 and 98/me are no more. winxp on up handle memory differently and more usefully. as memory is needed a program is assigned it. when it's not needed anymore, it is not released in case it's needed later, as it's faster that way. if another program needs memory, it will however give it up.
free memory is memory that is unused and not working for you. why buy gigabytes of memory and then try your best not to use it? if you really must, go into the config (type 'about:config' without the quotes into the address bar) and change config.trim_on_minimize to true. when you minimize FF it will release any unneeded memory. it may then need to reapply for it when you restore it, which will take a bit more time.
do not get taken in by the hype from some software that 'frees up memory'. those programs and their overhead are not needed and are in fact counter-productive.
task manager does not give a complete picture of memory useage anyhow. try procexp.exe (process explorer) from sys internals (now part of microsoft), it's x86/x64 and gives a better breakdown, here's my firefox details:
