The album art is hidden as part of the "protected operating system files".
Organize > Folder and search options > View
Uncheck Show hidden files, folders, and drives
Uncheck Hide protected operating system files (Recommended)
You'll have to delete the album art from each folder that has it, for it to be totally removed from WMP.
You may want to clear the library of all the music and restart WMP once you're done.