I'm not sure when this started happening since I usually do Shift+Delete. However, some things, like IE Favorites, will only go to the Recycle Bin. I only opened the Recycle Bin after all my programs were installed, so I'm not sure what's causing this issue.
Also, I already tried the tutorial posted above. It did nothing for me.
Here's a picture of the problem:
Also, I can still permanently delete stuff by clicking on "Empty Recycle Bin."