That's usually governed by the drivers for the video board and the monitor. Do you have both loaded and updated? VMWare provides this function in the drivers, so that's where you should be focused in your search.
No, I mean if you are trying to use a resolution not listed, it's because one of the two driversdon't support it. Trying to add support in any way may leave you booting to a black screen. You need to go to the video card vendor and monitor vendor to get support for that resolution.