Hello there, well it's not necessary at all to change your video card to a Nvidia one, what issues are you experiencing, it could be driver related, it could be software compatibility related and it could be solved.
In the same category of the HD 5770 would be the GTS 250 or even the GTX 260 core 216!
But there's a catch! the HD 5770 fully supports DX11 and it has newer technology, plus... less power consumption.
Another Nvidia card could be the GTS 450! Fully supports DX11 too.. but let's solve your issues first before thinking of buying a new card