If you are getting these odd lines
outside of Windows ie during POST (or any area
before Windows starts loading ) it could be an issue with the card itself.
If it's only occurring
inside windows - then it
may be software, but from the symptoms, it sounds like hardware.
When you say you bought the computer, did you buy a pre-built machine or did you assemble it yourself?
If it was pre-built, was it also sold as 'pre-overclocked' ie the CPU, RAM etc.
For the memory scans, your friend may have been referring to
memtest86+ .
This is a stand alone application that you runs outside of Windows and scans for memory errors.
You burn the .iso file to a bootable CD, boot from the disc and just let it run. Usually 8 passes is minimum/recommended to uncover any errors, unless it starts showing errors straight away. If it starts showing errors straight away, then it could be incorrect settings, a bad stick or the mobo itself.
(*NB memtest takes a while to run, so for a 'long run, it's best to do it overnight or while you are away from your PC)
You can use a tool like
PowerIso to mount and burn the .iso
If you could fill in the rest of the specs (especially what PSU and model of 5870 you using) that would help us a little too.