If you are getting these odd lines outside of Windows
ie during POST (or any area before
Windows starts loading ) it could be an issue with the card itself.
If it's only occurring inside
windows - then it may be
software, but from the symptoms, it sounds like hardware.
When you say you bought the computer, did you buy a pre-built machine or did you assemble it yourself?
If it was pre-built, was it also sold as 'pre-overclocked' ie the CPU, RAM etc.
For the memory scans, your friend may have been referring to memtest86+
This is a stand alone application that you runs outside of Windows and scans for memory errors.
You burn the .iso file to a bootable CD, boot from the disc and just let it run. Usually 8 passes is minimum/recommended to uncover any errors, unless it starts showing errors straight away. If it starts showing errors straight away, then it could be incorrect settings, a bad stick or the mobo itself.
(*NB memtest takes a while to run, so for a 'long run, it's best to do it overnight or while you are away from your PC)
You can use a tool like PowerIso
to mount and burn the .iso
If you could fill in the rest of the specs (especially what PSU and model of 5870 you using) that would help us a little too.