It's not a Windows bug. On my machine I can create, manipulate, and delete files with a UFO extension to my heart's content, but then my machine doesn't have the same mix of type-specific add-on handlers and anti-virus filter drivers
As a suggestion, test what happens in safe mode.
As a separate test, try to temporarily uninstall (disabling isn't good enough) your anti-virus solution, if any, because it has the architectural capacity to cause this type of problem.
You're basically looking for the driver
or add-on which is interfering with the delete operations. A "vanilla" Windows machine doesn't even have any associations for *.UFO. It'll nuke those files happy-as