Windows 7 64-bit Corrupting Large Files Copied to External NTFS Drives

This past September (2011), I discovered one day after a series of unexplainable issues with a restored file, that a number of backup copies of large files on external USB-connected NTFS drives differed from the source versions still on my hard drive. I also discovered that I could reproduce the issue with newly-copied files.

Here are the specifics of the issue, following a series of experiments:


- On my system, copies of large files, files typically larger than 500MB, are corrupted (altered) roughly 30% of the time when copying them under Windows 7 64-bit to USB-connected NTFS-formatted external drives.


- No error occurs / no error message appears during the copy



- The file size of copy is always identical, whether or not data was altered during the copy process.



- File differences are confirmed via either the command-line "FC" command or a utility such as WinDiff


- The issue impacts copies made via the Windows GUI -OR- via command-line copy or xcopy


- The issue occurs with multiple external USB NTFS-formatted drives, no matter what make or model.


- Subsequent attempts to copy an affected file will ultimately yield an identical copy. This would seem to rule out interference by an external program such as an anti-virus program (and the only AV I am running is Microsoft Security Essentials)


- The USB drives involved pass error checks, and copies made to these drives on other (non W7) systems produce identical copies

- So far, the third party utility "TeraCopy" manages to consistently produce clean copies, and therefore is a temporary workaround. This utility apparently works because it, by default, bypasses the NTFS memory caching operation used by the Windows 7 OS...a caching system which I have so far found no way of disabling.

- The problem does not appear to impact relatively small files (1 to 100MB or so). I have not found any particular threshold, but I have seen the issue impact numerous files in the 500MB neighborhood.


- The problem seems to date at least to the version of Windows 7 that was in release as far back as the Fall of 2010, as I discovered corrupted backup copies of files dating back that far. Again, the files are corrupted with respect to the original copy...NOT with respect to file structure itself.

There is a thread on this issue in the MS Technet forum, and others are reporting identical experiences and specifics. Evidence at this point seems to be pointing to NTFS write-caching, which is used by the OS (with no apparent way to disable), but which is not used by the third-party TeraCopy unless one explicitly enables it.


I assumed that this problem would have been addressed by Microsoft at this point and resolved, but so far, it remains an issue. It's very frustrating to not be able to depend on the native copy operations of Windows 7, but at the moment, that's the case on my system, and I am not alone.

(P.S. It's worth noting that, even with "Optimize for Removal" set on an external drive, the write cache is still minimally utilized, in a "write-through" mode.)

Welcome to Seven Forums retronaut.
Try enabling Write Caching for better performance to see if that helps. You'll need to use the Safely remove hardware with it enabled.

Heh yes, if Safely Remove Hardware was not used after any copy across USB then there is always a chance of corruption. Maybe larger copies suffer from this more than smaller ones? Or were you sure that you always properly ejected the device before unplugging it or turning it off?

fseal said:

Heh yes, if Safely Remove Hardware was not used after any copy across USB then there is always a chance of corruption. Maybe larger copies suffer from this more than smaller ones? Or were you sure that you always properly ejected the device before unplugging it or turning it off?

My drives are set to Optimized for Removal, but I always use Safely Remove. This is an extremely surreptitious problem. File structure is sound, disks are free of errors, no errors are indicated or logged on copies or accesses, etc., etc.. Only when comparing copied files with the source (or encountering playback or other interpretive errors with the data in the copied file) does the problem reveal itself. It's also intermittent with respect to any particular copy operation, including multiple copies of the same file. For example, last night, I made three successive copies of the same 1.4GB file to a USB-connected external NTFS drive under Windows 7 64-bit. The first two copies compared successfully, but here are the results of the comparison of the third copy:

D:\tv\720p-mkv>comp test.mkv g:\test\test.mkv
Comparing test.mkv and G:\test\test.mkv...
Compare error at OFFSET 6660000
file1 = E8
file2 = 3
Compare error at OFFSET 6660001
file1 = 1D
file2 = 1
Compare error at OFFSET 6660002
file1 = 9F
file2 = EE
Compare error at OFFSET 6660003
file1 = 19
file2 = D6
Compare error at OFFSET 6660004
file1 = 9F
file2 = 44
Compare error at OFFSET 6660005
file1 = 60
file2 = 7C
Compare error at OFFSET 6660006
file1 = 65
file2 = 82
Compare error at OFFSET 6660007
file1 = C6
file2 = D7
Compare error at OFFSET 6660008
file1 = C0
file2 = F8
Compare error at OFFSET 6660009
file1 = 8F
file2 = 7E
10 mismatches - ending compare


By the way, several in the TechNet thread on this report identical symptoms, and one person there is observing this and can reproduce the issue on two different Windows 7 64-bit systems.

One person there tested extensively and could not generate faulty copies, and the only obvious difference was a different USB controller. However, another user indicates that his experience with the issue isn't limited to USB, and that he also sees it on externally-connected eSATA NTFS drives.