Control Panel > Internet Options > Delete button (under Browsing History).
A dialog box will open allowing you to check mark (select) Tempory Files, History, Cookies, etc for deletion. Or, click the Settings button and a dialog box called Temporary Internet Files and History Settings will appear. Click on View Files and you can see the (possibly thousands) of cookies being stored. From here you can select individual cookies for deletion.
If you don't want to go through a humongous listing of cookies by using the Settings button, but you want to clean up a bit more than the default Delete button gives you, you might give CCleaner a look. It's free. CCleaner - Optimization and Cleaning - Free Download