I experienced the exact same issue running on my ESXi 5.0 host (PE 1900, Dual Quad Core 5335) - blank screen on the Xbox 360 extender. I was using the ESX host for a number of other reasons and thought it might be cool to eliminate a media center PC as well. After reading this post I decided it wasn't worth it and moved on to other things.
This weekend, however, I tried installing Windows 8 Consumer Preview on the system and discovered that it wouldn't work without patching ESXi to patch2. I installed all of the latest patches. I also noticed that there was no audio device, and found a method of manually adding an HD Audio
driver to the vmx file. I also updated the VMware tools.
After this, I decided to try hooking up the vm to an XBox again -- and it worked! I got video and audio. Because I made all of these changes at once, I can't say which is needed, but they're all easy, free, and probably a good idea anyway.
That said, I'm left with one problem that I've actually seen before with one of my physical systems: when I skip, rew, or ffwd through recorded TV, it always takes a very long time --maybe 5-10 seconds before responding to the 30-second jump. The CPU and network graphs on the VM aren't spiking and I don't have a clue what it's using to do the seek of the file (remote over UNC path to WHS server).
Anyone have any idea why HD live tv (HD Homerun) and recorded wtv files would play smoothly, but jumps/rew/ffwd would experience delays?