Have you un-installed the old video drivers... run driver sweeper and install the newest graphics drivers in that order after upgrading the card?
As well as disabled onboard graphics if there are any?

As mentioned, all the RAM should match.

However,what you can try for now:
if the new is faster or tighter timings you can try adjusting it all to the slowest speed of the bunch, and the loosest timings of the bunch.

I/E 2 sticks DDR2 667 5-5-5-15 and 2 are DDr2 800 4-4-4-12
Set them, all at 667 5-5-5-15

Although having them all match is by far the best route as already mentioned, and highly recommended.
You can try this to see if it helps, but I wouldnt leave it that way.