A default program is the program that Windows uses when you open a particular type of file extension.
Open with allows you to change a file type to open with a different default program for just this one time or always using the context menu in Windows.
Removing
Open with from the context menu can make it easier to prevent all users on the computer from accidentally have a file extension type assigned a wrong default program to be opened with by default.