Windows 7: Windows 7 64-bit Corrupting Large Files Copied to External NTFS Drives

This past September (2011), I discovered one day after a series of unexplainable issues with a restored file, that a number of backup copies of large files on external USB-connected NTFS drives differed from the source versions still on my hard drive. I also discovered that I could reproduce the issue with newly-copied files.

Here are the specifics of the issue, following a series of experiments:


- On my system, copies of large files, files typically larger than 500MB, are corrupted (altered) roughly 30% of the time when copying them under Windows 7 64-bit to USB-connected NTFS-formatted external drives.


- No error occurs / no error message appears during the copy



- The file size of copy is always identical, whether or not data was altered during the copy process.





- File differences are confirmed via either the command-line "FC" command or a utility such as WinDiff


- The issue impacts copies made via the Windows GUI -OR- via command-line copy or xcopy


- The issue occurs with multiple external USB NTFS-formatted drives, no matter what make or model.


- Subsequent attempts to copy an affected file will ultimately yield an identical copy. This would seem to rule out interference by an external program such as an anti-virus program (and the only AV I am running is Microsoft Security Essentials)


- The USB drives involved pass error checks, and copies made to these drives on other (non Windows 7) systems produce identical copies

- So far, the third party utility "TeraCopy" manages to consistently produce clean copies, and therefore is a temporary workaround. This utility apparently works because it, by default, bypasses the NTFS memory caching operation used by the Windows 7 OS...a caching system which I have so far found no way of disabling.

- The problem does not appear to impact relatively small files (1 to 100MB or so). I have not found any particular threshold, but I have seen the issue impact numerous files in the 500MB neighborhood.


- The problem seems to date at least to the version of Windows 7 that was in release as far back as the Fall of 2010, as I discovered corrupted backup copies of files dating back that far. Again, the files are corrupted with respect to the original copy...NOT with respect to file structure itself.

There is a thread on this issue in the MS Technet forum, and others are reporting identical experiences and specifics. Evidence at this point seems to be pointing to NTFS write-caching, which is used by the OS (with no apparent way to disable), but which is not used by the third-party TeraCopy unless one explicitly enables it.


I assumed that this problem would have been addressed by Microsoft at this point and resolved, but so far, it remains an issue. It's very frustrating to not be able to depend on the native copy operations of Windows 7, but at the moment, that's the case on my system, and I am not alone.

(P.S. It's worth noting that, even with "Optimize for Removal" set on an external drive, the write cache is still minimally utilized, in a "write-through" mode.)

I've seen this problem. I downloaded a large zip file and copied it to an external drive. When it was done copying I opened the copy and discovered the corruption. I returned to the original file and discovered it was not corrupt. My second attempt worked fine. I seldom work with files this large so this is the only data point I have. I'll create some large archives and try it a few times over the next few days.

Welcome to Seven Forums retronaut.
Try enabling Write Caching for better performance to see if that helps. You'll need to use the Safely remove hardware with it enabled.

Heh yes, if Safely Remove Hardware was not used after any copy across USB then there is always a chance of corruption. Maybe larger copies suffer from this more than smaller ones? Or were you sure that you always properly ejected the device before unplugging it or turning it off?

My drives are set to Optimized for Removal, but I always use Safely Remove. This is an extremely surreptitious problem. File structure is sound, disks are free of errors, no errors are indicated or logged on copies or accesses, etc., etc.. Only when comparing copied files with the source (or encountering playback or other interpretive errors with the data in the copied file) does the problem reveal itself. It's also intermittent with respect to any particular copy operation, including multiple copies of the same file. For example, last night, I made three successive copies of the same 1.4GB file to a USB-connected external NTFS drive under Windows 7 64-bit. The first two copies compared successfully, but here are the results of the comparison of the third copy:

D:\tv\720p-mkv>comp test.mkv g:\test\test.mkv
Comparing test.mkv and G:\test\test.mkv...
Compare error at OFFSET 6660000
file1 = E8
file2 = 3
Compare error at OFFSET 6660001
file1 = 1D
file2 = 1
Compare error at OFFSET 6660002
file1 = 9F
file2 = EE
Compare error at OFFSET 6660003
file1 = 19
file2 = D6
Compare error at OFFSET 6660004
file1 = 9F
file2 = 44
Compare error at OFFSET 6660005
file1 = 60
file2 = 7C
Compare error at OFFSET 6660006
file1 = 65
file2 = 82
Compare error at OFFSET 6660007
file1 = C6
file2 = D7
Compare error at OFFSET 6660008
file1 = C0
file2 = F8
Compare error at OFFSET 6660009
file1 = 8F
file2 = 7E
10 mismatches - ending compare


By the way, several in the TechNet thread on this report identical symptoms, and one person there is observing this and can reproduce the issue on two different Windows 7 64-bit systems.

One person there tested extensively and could not generate faulty copies, and the only obvious difference was a different USB controller. However, another user indicates that his experience with the issue isn't limited to USB, and that he also sees it on externally-connected eSATA NTFS drives.

This is an insidious and very stealthy issue. For those of you with Windows 7 64-bit and who have NTFS-formatted external drives connected via USB, please try the following. You will need a file of random data (such as a movie file) of at least 1GB in size.

From command prompt, and assuming your (large) file is named test.fil, it's in the directory "test" and your external NTFS-formatted USB drive is G:...

c:> cd test (or to whatever directory holds your test file)

c:> copy test.fil g:\test.fil

c:> fc test.fil g:\test.fil

c:> copy test.fil g:\test1.fil (note the alternate name for the destination file)

c:> fc test.fil g:\test1.fil

c:> copy test.fil g:\test2.fil ( " )

c:> fc test.fil g:\test2.fil

Do you obtain consistently identical copies as verified by fc? It may take at least four or five tries in my case, but typically, within the first three tries, such a test copy produced on my system (a Dell Studio XPS 435T) will be faulty, yet it will have all the outward appearances of being an identical copy. "comp" and the utility "WinDiff" also confirm the alteration...

Several others in the TechNet forum on this topic have confirmed identical symptoms...with different Intel USB chipsets and with different USB drives. I see the issue with two different models and vintages of USB drives. One user responding on the thread does not observe errors. This is a huge mystery, but a serious problem which renders the default Windows 7 copy operation unreliable for some, including myself.

Thank you.

By any chance, are you trying to copy/move large files to an NTFS drive with compression for the partition or directory turned on?