Do you want something like this ?
Actually I dont know how to add an album art in a video file ( I know how to do it for a music file), but there is a way to do the same thing, though a bit laborious . I am describing the steps below :
1. Download an album art for your video (I use google image search for it)
2. Make it square using any image editor. Dont crop, just resize, say 400x400 pixels
3. make a folder for your video, put the resized image in it, name it folder.jpg, and hide it.
4. You are done
Now look the explorer folder, add those to media player or media center, and see the change
If there is any trouble doing it, make me know