that's not normal. Outlook cannot possibly be eating 4 GB and all stuff loaded to RAM by super prefetch and left in RAM after you close an application is always ready to be deleted in case you need that space, and isn't shown as "occupied" in Task Manager.
Can you post a screenshot of Performance Monitor, with the tab about RAM use open, so we can see what is going on with our eyes as well?
to open Performance Monitor, open the Start menu and write "perfmon" (without the ") in the search box and click on the result, or from Task Manager's performance tab there is a button called Resource Monitor that starts it. This tutorial
explains how to post screenshots in this forum.