Windows 7: Corruption in large file transfers to USB drives

[TrustMe]

The solution was posted by Bill2. He recommended using TeraCopy. The OP said it worked like a charm. Read post #34 in this thread.

 

[dishe]

TrustMe, I saw that before finding this forum and am already installing Teracopy. The problem is that's only a workaround and not a solution. Any application that uses the windows copy protocol under the hood will still corrupt data when transferring to an external, since Teracopy doesn't actually replace the Windows method.

For example, I use the excellent open source FreeFileSync to set rules and incrementally mirror a folder of working files to a folder on another USB drive every night. FFS uses the Windows copy protocol, which means while it has been copying my data to another drive, I could have been further corrupting it, resulting in corrupted "backups" all these years (thank goodness I never needed to restore from it!). There are backup apps which will check and verify data for you, but as far as I know most of them still rely on the windows method- you can't force them to use Teracopy's!

Then there are applications and scripts for truncating spanned files back together, as an example- sometimes they create a temporary file on the system drive to work, and then copy to the destination when finished. If I'm working off external drives, and the operation then copies my multi-GB stitched-together file, it is going to be using the Windows file copy protocol again.

Teracopy is a band aid that will help in some scenarios. Sadly, it isn't a solution! This thread has more updated info and possible causes, but it seems to still be largely unknown what causes it and what conclusively fixes it:
Files corrupted when copying to external usb3 hdd

 

[TrustMe]

Sorry, but you seem to know way more about this than I do. I use TeraCopy whenever I have to copy large files to my portable hard drive. I read it creates a hash before it starts to copy and checks the hash when it's finished. I don't do a lot of coping though.

As for why Windows copy protocol corrupts the file, I really don't know.

 

[dishe]

TrustMe, I appreciate the response either way.
Sadly, I'm having this problem on my work computer, and this is a very big problem for what I do.

I just can't believe such a major and re-creatable problem has been swept under the rug for years! How many people just didn't notice this was happening because they don't do MD5 checksums to verify data copies? I thought I had a fool-proof solution for my business, and now I feel like I've been living a lie! What about people who run major databases, like banks and government workstations? Would MS just tell them to use Teracopy? This could potentially be a very big problem, and teracopy isn't even a proper solution for all uses.

It seemed like they were getting pretty close to figuring out what happened between this and other threads I've been finding- some say changing the USB drive settings to "better performance" instead of the default fast disconnect helped. Others said that removing Microsoft Security Essentials got rid of it. Some suggested new RAM or a new PCI USB card fixed theirs, but its totally inconsistent since others said it didn't work for them. I hate it that I can't confirm any of these to be a solution.

Does Windows 10 solve it? I've been sticking to 7 because my legacy software is finally set up in a way that works for my job, and I'm afraid to touch anything in case new incompatibilities show up and create new headaches!

 

[TrustMe]

I do have a computer with Windows 10. A couple weeks back Microsoft came out with new Windows 10 ISO files (February update). I downloaded the Windows 10 Multiple Editions (Home and Pro) 32-Bit about 2.8GB and 64-Bit about 3.8GB. I also downloaded the Single Lanuage version both 32-Bit and 64-Bit. Not sure the size of Single Language.

I don't have TeraCopy on that computer and used Windows copy protocol. I copied all 4 ISO files to two different USB 3 portable hard drives. I checked the Sha1 hash on all 8 ISO files and there were no mistakes.

I know that is not much of a test but like I said, I don't do a lot of copying. Maybe you can setup a Windows 10 test computer and try it for yourself.

 

[dishe]

I have a laptop, tablet and netbook all running Win10 with the free upgrade. The problem is, I can't verify that these ever had the copy bug previously. Plenty of people in the technet thread weren't able to reproduce the error (which also means they weren't able to troubleshoot it), so it seems to only happen on some computers, with some sort of "perfect storm" scenario that no one has managed to figure out the cause of:
https://social.technet.microsoft.co...ied-to-external-ntfs-drives?forum=w7itproperf

What I'd like to see, is if someone who originally HAD this problem on a Windows 7 machine, then upgraded to 10 and had it go away? I'd take the plunge on my work computer if I knew it would fix it, otherwise it took me a long time to get my legacy applications installed and working properly even on Windows 7 so I don't want to touch it if it wouldn't fix it anyway.

 

[dishe]

I wonder, actually- has anyone confirmed that this bug doesn't apply to exFAT formatted drives?
I see NTFS mentioned a bunch of times, but I'm not sure if that's just because everyone effected only tested it on their NTFS drive, or if they found it didn't happen on non-NTFS drives?

exFAT has a lot of advantages as a cross-platform backup tool, in case I need to share files with a Mac and back. I wouldn't mind... if it would work, that is.