Hello jlusen, and welcome to Seven Forums.
Compatibility Mode has Windows 7 run the program as if it was running in XP.
XP Mode is XP virtual machine (a separate OS running in Windows 7 as a window) that runs inside the Windows Virtual PC program in Windows 7. Programs installed in the XP Mode virtual machine can have a shortcut in the Windows 7 Start Menu to be able to run the program as if it was running in the XP Mode virtual machine.
The pros and cons really depends on your needs and if the older XP program will run fine or not by just using Compatibility Mode. XP Mode will have no problem running XP programs, but requires you to install "Windows Virtual PC" and "XP Mode". Afterwards, install the XP program in XP Mode.
Hope this helps,
Shawn