I don't use OSX, but from what I've seen the OSX version of Windows Magnifier displays the same way with solid borders around the magnified area.
See: Preview's new "Magnifier" | Preview | Mac OS X Tips
Is there a specific use that you notice the dash lines or can you see them whenever you use Zoom?
In Mac OS 8 and 9 this feature was called CloseView
. In OSX it is called Zoom
Do you use any third-party drawing apps? Sometimes these can crossover and affect the main OS. Check all your settings especially any smoothing, make sure its turned on if you do have these types of programs and check the OSX Zoom and main display settings for smoothing.
If you increase the zoom factor; Do the dash lines get thicker or do they disappear? If thicker, It could be a screen resolution setting. If they disappear while zooming in or out they could be artifacts in whatever your trying to magnify.
This may be similar: Problem with Preview "Select" command - MacRumors Forums
I have seen a Mozilla Bug Report
if you use Firefox in OSX while using zoom, but it's marked resolved. Are you up to date? the latest OSX is: https://itunes.apple.com/us/app/os-x...15041082?mt=12
You may have to visit your favorite OSX Forum and pose your question there.
Here's what mine looks like: