should updates be removed when you download new ones?

No leave the old updates alone
Each update is separate and should be used in conjuction with the other updates
When a SP is issued all previous updates are included but they will not duplicate.

Old Office Updates should be removed along with Office when it is uninstalled.

Use free Revo uninstaller in Advanced Mode to remove Office most cleanly. After running the Windows Uninstaller, don't restart when prompted but click Next and it will present a list of hidden Registry Keys for you to delete all the ones in bold, followed by a list of files.

The only case where you ever need to remove Windows Updates is if they cause a malfunction. You can then remove them in the Installed Updates panel, by System Restoring from Windows or from the booted WIn7 DVD or Repair CD. Every Update sets a Restore point, as does the Revo Uninstaller.