This past September (2011), I discovered one day after a series of unexplainable issues with a restored file, that a number of backup copies of large files on external USB-connected NTFS drives differed from the source versions still on my hard drive. I also discovered that I could reproduce the issue with newly-copied files.
Here are the specifics of the issue, following a series of experiments:
- On my system, copies of large files, files typically larger than 500MB, are corrupted (altered) roughly 30% of the time when copying them under Windows 7 64-bit to USB-connected NTFS-formatted external drives.
- No error occurs / no error message appears during the copy
- The file size of copy is always identical, whether or not data was altered during the copy process.
- File differences are confirmed via either the command-line "FC" command or a utility such as WinDiff
- The issue impacts copies made via the Windows GUI -OR- via command-line copy or xcopy
- The issue occurs with multiple external USB NTFS-formatted drives, no matter what make or model.
- Subsequent attempts to copy an affected file will ultimately yield an identical copy. This would seem to rule out interference by an external program such as an anti-virus program (and the only AV I am running is Microsoft Security Essentials)
- The USB drives involved pass error checks, and copies made to these drives on other (non W7) systems produce identical copies
- So far, the third party utility "TeraCopy" manages to consistently produce clean copies, and therefore is a temporary workaround. This utility apparently works because it, by default, bypasses the NTFS memory caching operation used by the Windows 7 OS...a caching system which I have so far found no way of disabling.
- The problem does not appear to impact relatively small files (1 to 100MB or so). I have not found any particular threshold, but I have seen the issue impact numerous files in the 500MB neighborhood.
- The problem seems to date at least to the version of Windows 7 that was in release as far back as the Fall of 2010, as I discovered corrupted backup copies of files dating back that far. Again, the files are corrupted with respect to the original copy...NOT with respect to file structure itself.
There is a thread on this issue in the MS Technet forum, and others are reporting identical experiences and specifics. Evidence at this point seems to be pointing to NTFS write-caching, which is used by the OS (with no apparent way to disable), but which is not used by the third-party TeraCopy unless one explicitly enables it.
I assumed that this problem would have been addressed by Microsoft at this point and resolved, but so far, it remains an issue. It's very frustrating to not be able to depend on the native copy operations of Windows 7, but at the moment, that's the case on my system, and I am not alone.
(P.S. It's worth noting that, even with "Optimize for Removal" set on an external drive, the write cache is still minimally utilized, in a "write-through" mode.)
Here are the specifics of the issue, following a series of experiments:
- On my system, copies of large files, files typically larger than 500MB, are corrupted (altered) roughly 30% of the time when copying them under Windows 7 64-bit to USB-connected NTFS-formatted external drives.
- No error occurs / no error message appears during the copy
- The file size of copy is always identical, whether or not data was altered during the copy process.
- File differences are confirmed via either the command-line "FC" command or a utility such as WinDiff
- The issue impacts copies made via the Windows GUI -OR- via command-line copy or xcopy
- The issue occurs with multiple external USB NTFS-formatted drives, no matter what make or model.
- Subsequent attempts to copy an affected file will ultimately yield an identical copy. This would seem to rule out interference by an external program such as an anti-virus program (and the only AV I am running is Microsoft Security Essentials)
- The USB drives involved pass error checks, and copies made to these drives on other (non W7) systems produce identical copies
- So far, the third party utility "TeraCopy" manages to consistently produce clean copies, and therefore is a temporary workaround. This utility apparently works because it, by default, bypasses the NTFS memory caching operation used by the Windows 7 OS...a caching system which I have so far found no way of disabling.
- The problem does not appear to impact relatively small files (1 to 100MB or so). I have not found any particular threshold, but I have seen the issue impact numerous files in the 500MB neighborhood.
- The problem seems to date at least to the version of Windows 7 that was in release as far back as the Fall of 2010, as I discovered corrupted backup copies of files dating back that far. Again, the files are corrupted with respect to the original copy...NOT with respect to file structure itself.
There is a thread on this issue in the MS Technet forum, and others are reporting identical experiences and specifics. Evidence at this point seems to be pointing to NTFS write-caching, which is used by the OS (with no apparent way to disable), but which is not used by the third-party TeraCopy unless one explicitly enables it.
I assumed that this problem would have been addressed by Microsoft at this point and resolved, but so far, it remains an issue. It's very frustrating to not be able to depend on the native copy operations of Windows 7, but at the moment, that's the case on my system, and I am not alone.
(P.S. It's worth noting that, even with "Optimize for Removal" set on an external drive, the write cache is still minimally utilized, in a "write-through" mode.)
Last edited:
My Computer
At a glance
Microsoft Windows 7 Professional 64-bitIntel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2668 ...
- Computer Manufacturer/Model Number
- Dell Studio XPS 435T
- OS
- Microsoft Windows 7 Professional 64-bit
- CPU
- Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz, 2668 Mhz, 4
