Windows 7: Corruption in large file transfers to USB drives

[Pleco]

Have you tried a clean reinstall to see if the behavior continues? I'd clean reinstall without any tweaking or SFV to see the difference.

As a last resort you could also try a Repair Install, but if you have corruption in the SFV program or from tweaking it might let it back through.

Tips for getting a baseline purrfect reinstall - use the ones which apply: http://www.sevenforums.com/installation-setup/125874-re-install-windows-7-a.html#post1086729

I have already tried a complete fresh install on a separate hard drive but the same thing occurs. I have never done any tweaking on my Windows 7, I left it on default settings. As for SFV, I downloaded a fresh copy of Quicksfv each time so I doubt it was corrupted itself. I have been using Quicksfv for over a decade now and it has never let me down.

But how long have you used SFV on Win7? If you've reinstalled to the same problem, then I'd try running for awhile without it. In fact I would not use any file copying program or anything like it at all until you elminate this as causing this as it is otherwise unheard of in Win7.

You should also wipe your HD of all code before trying reinstall as Infection or other corruption cannot be ruled out yet.

I have been using Quicksfv on Windows 7 RC for half a year with absolutely no problems. Once I moved to Windows 7 Retail is when first saw this happen. Quicksfv is NOT a file copying program, I only use Windows explorer for copying files. All it does is create a separate text file with a CRC32 checksum or MD5 of the file.

Infection and corruption of the OS has been ruled out, if you were to read my later posts, I have already stated at least 3 times that I have used a separate hard drive to install Windows 7 on.

Here's some people with the same problem:

http://www.sevenforums.com/general-discussion/63459-files-corrupt-after-copy-usb-hard-disk.html

 

[Pleco]

I actually tried testing this myself on a video I had. 9 gigs of video copied over. Reported as corrupted. I played the video start to finish. There was not so much as one tiny sound hitch or out of place pixel in the entire video. If corruption occurred it was not anything that could be detected by me or the media player.

Thank you, finally someone who understands what I'm experiencing .

Some of the files I'm backing up are important database files, so I can't have any corruption in them at all. And opening them up to check every bit of information would take forever.

 

[unifex]

I will second what Maguscreed posted: I do routinely copy large video files (starting from 1.1GB up to 8GB) onto various USB media - USB drives and external hard drives. I have never seen any problem - the videos are perfectly playable before, after, and on the USB media (basically, I never watch any video on my PC, I just copy the files onto one large USB stick, plug it into my Blu-Ray player and watch on the big screen).

My question to the OP is, what are other signs of corruption apart from your using the Quicksfv software? Were your videos rendered unplayable or various artifacts introduced? Were iso files corrupted, meaning burning software would not recognize them? Same thing with compressed archives? Basically these are the large files types that come to mind. You keep referring to the Quicksfv results, but for example my personal experience and possibly that of many others is that we do copy large files through USB, see no ill effect whatsoever, but never "check" file integrity with special software, such as Quicksfv (of which I have frankly not even heard before). Maybe this program just gives you false results?

My two cents.

 

[unifex]

I actually tried testing this myself on a video I had. 9 gigs of video copied over. Reported as corrupted. I played the video start to finish. There was not so much as one tiny sound hitch or out of place pixel in the entire video. If corruption occurred it was not anything that could be detected by me or the media player.

Thank you, finally someone who understands what I'm experiencing .

Some of the files I'm backing up are important database files, so I can't have any corruption in them at all. And opening them up to check every bit of information would take forever.

I think you misinterpreted this post. I highlight what I see as key phrases.

Did you ever try any other software instead of Quicksfv?

 

[Bill2]

Try copying with a 3rd party tool like teracopy.

 

[Pleco]

I actually tried testing this myself on a video I had. 9 gigs of video copied over. Reported as corrupted. I played the video start to finish. There was not so much as one tiny sound hitch or out of place pixel in the entire video. If corruption occurred it was not anything that could be detected by me or the media player.

Thank you, finally someone who understands what I'm experiencing .

Some of the files I'm backing up are important database files, so I can't have any corruption in them at all. And opening them up to check every bit of information would take forever.

I think you misinterpreted this post. I highlight what I see as key phrases.

Did you ever try any other software instead of Quicksfv?

I have most of my large files compressed with WinRAR to make them smaller. When I run the Test function it fails and reports at least one corrupted file. Yes I have tried other programs like SFV Checker and RapidCRC.

 

[Phenom78]

did you check your ram? Maybe there is some bad block that is causing problems?

 

[Pleco]

did you check your ram? Maybe there is some bad block that is causing problems?

Yes, with Memtest86+ for more than 24 hours.

 

[unifex]

You still did not answer the basic question - did you see any other signs of corruption apart from those test results?

I understand that it might be time consuming to check that every time you copy a file, but once or twice - it's definitely worth it. You do realize that most people copy large files without any problem, but never run any tests to "confirm" it. If I copy a video on my USB stick and it plays without a hitch on my Blu-Ray and at the same time a test will report it as corrupted - I will simply discard the testing software as useless.

 

[Pleco]

You still did not answer the basic question - did you see any other signs of corruption apart from those test results?

I understand that it might be time consuming to check that every time you copy a file, but once or twice - it's definitely worth it. You do realize that most people copy large files without any problem, but never run any tests to "confirm" it. If I copy a video on my USB stick and it plays without a hitch on my Blu-Ray and at the same time a test will report it as corrupted - I will simply discard the testing software as useless.

Scroll up a bit and you'll see I already answered that I compressed some files with WinRAR and when I run the test function it fails and reports at least one file was corrupted and cannot be extracted. This confirms my data was corrupted on the external and can't be used. I used several other checking programs as well. My files are not videos, I can't play them with a video player. Neither are they ISO images I can burn and verify.

I always check the integrity of my files when I back them up to any external source, and have been doing so for the last 15 years. This is the first time I have run INto this problem that's not related to hardware defects since its ONLY occurs on Windows 7. Xp, Linux, and even OSX copies large files absolutely fine without a hitch.

You say testing software is useless? So how exactly do you test the integrity of your files then? What do you tell your boss when you were too lazy and felt there was no need to do a simple integrity check and you just lost months of work due to corrupted files and you can't meet the project deadline?

 

[gregrocker]

I think what we are saying is that maintaining file integrity isn't much of an issue that any of us have seen with Win7, so the actual problem might be the way the file checking program interacts with Win7 causing it's output to be possibly faulty.

Sometimes I copy multiple ISO's, extract, recompile dozens of times and have never had any file degradation with Win7. The only need to check file HASH was when the download source was questionable, but never after it got onto Win7.

What I would do next in your situation is go to the website for the file checking program and find out the best way to approach them - forum, email, chat, phone - then ask for a definitive answer to your problem.

 

[Pleco]

I think what we are saying is that maintaining file integrity isn't much of an issue that any of us have seen with Win7, so the actual problem might be the way the file checking program interacts with Win7 causing it's output to be possibly faulty.

Sometimes I copy multiple ISO's, extract, recompile dozens of times and have never had any file degradation with Win7. The only need to check file HASH was when the download source was questionable, but never after it got onto Win7.

What I would do next in your situation is go to the website for the file checking program and find out the best way to approach them - forum, email, chat, phone - then ask for a definitive answer to your problem.

I have tried at least 5 different CRC/md5 checking programs and all of them I have confirmed are fully compatible with Windows 7 64bit, even the testing function within both WinRAR and 7zip stated the files copied over were indeed corrupted. And corruption rate isn't 100% of the time, about 3/10 files will be corrupted on average.

Wouldn't this conclude the problem is with the OS itself? This isn't just happening on my own hardware, I had a friend and colleague test this on their own computers and ended up with the same issues.

Hopefully, the new service pack will fix this problem.

 

[unifex]

First of all, I think we agree here that we are dealing with a rather rare problem and thus we need to pinpoint exactly what's going on and what's the reason for it.

Here's what I would do. As far as I am concerned, there are two types of files - text files and binary files. Both types of files can be directly compared. Why don't you take a case where your testing software reports corruption and directly compare the copied ("corrupted") file to the original and see what the difference is. You can use Windiff, Merge, or other similar tools to do that. With text files it's probably easier, but I am not sure what kind of files you're talking about.

Could you make a test comparison, document what you did - how the files got corrupted in the first place - and post the actual difference between them? I will then try to reproduce the problem on my own machine.

 

[Pleco]

Try copying with a 3rd party tool like teracopy.

Thank you Bill2 for recommending Teracopy, I actually had the time to test this program yesterday and it works like a charm. Spent 3 hours copying files to external hard drives and not one single incident of corruption. It even has the option to do the CRC checking for me after completing the copy. I used my usual programs to check the CRC as well and they all passed.

 

[riley34]

I have a brandnew laptop (Asus G53JW), where the same usb problem occurs, data loss/corrupted data while copying/moving large files with win explorer from Win7x64 to an external NTFS formatted USB harddrive.

I found the behaviour accidentally while creating images of the preinstalled recovery partition with sizes bigger 10 GB. Splitting the files in smaller chunks of 1G didn't help either.

The problem was present from the beginning, with the preinstalled OS (Windows7x64 Home Premium). The two internal SATA drives have no problem when copying with explorer from one disk to the other.

I installed the laptop completely from scratch using an genuine MS installation dvd, later all updates until SP1 and every single most recent driver from the manufacturer without any problem, even the BIOS is updated. Everything works fine, just the old problem there - again. Until this time I avoided installing FW/Virus software that could interfere the usb transfers.

By trial and error I found, that XCOPY produces no errors on copies in a CMD Window by some reason, but only when invoked with the /J switch, it disables i/o buffering: "xcopy /J <sourcefile> <destination>".

BTW Linux/Knoppix copies with perfection even under its GUI.

Switching the drive's write cache on/off doesn't change the situation under Win7.
The hardware used by the laptop is a i7 740QM, HM55 Express Chipset and a NVIDIA GTX460M.

Another interesting thing is that the System offers 2 seperate USB Host controllers, one by intel and one by fresco with different drivers, but the problem appeared on all USB hardware.

I guess the reason is more an internal OS mechanism with the win explorer, maybe with buffers or process syncronisation rather than a hardware or driver failure.

But honestly, I have absolutely no clue which screw to turn in Win7, for example how can I switch off i/o buffering in general for external usb drives?

I hope that some of the experts here have an idea, thanks in advance.

Cheers
Riley

 

[radioman]

I'm also experience problems transferring large files to external usb hard drives.

My computer uses Windows 7 Home Premium 64 bits, my usb controllers are from Intel and the hard disk is a external 1Tb NTFS formatted.

The corruption for me started when the drive was about 50% of its capacity. The corruption consists in random blocks of zeros in the file.

These blocks of zeros are 20480 bytes in length. None of the files places at the first half of the disk were corrupted, but many on the second half was. Some files as small as 200 Mb were corrupted.

After intense hard drive checking and surface scan I discarded physical problems on the hard disk.

Using xcopy /j does not work, neither disable antivirus or 3rd party utilities.

I have reformated the hard drive to FAT32, and up to date, not a single byte corrupted. The main disadvantage is the need to split very large files, but I prefer this small oddity that loss of data.

For me it is clear there is a serious issue with external usb NTFS drives with Windows 7.

Hope this helps someone to find the problem.

 

[wanderlust76]

Hi everyone, I'm having the same problem and it's very annoying...
I experienced file transfer corruption with large files from two different laptops (both HP, a dv6000 and a dv5, with Windows 7 Ultimate x64) to two different external hard drives (a WD passport 1TB and an Iomega 500GB), both formatted NTFS.
When using Teracopy I'm not having crc errors.
Did anyone found a solution?

 

[Phone Man]

Are you running SP1. One of the Hot-Fixes included in SP1 deals with large file problems when writing to USB drives. Apparently the USB bus would time out and then restart.

Jim

 

[wanderlust76]

Yes, I updated to SP1 (and all the latest updates) both PC.

 

[dishe]

Hate to revive a thread as old as this one, but was there ever a solution found for this?

I work in video and have noticed frames missing/blank/corrupted in my editor sometimes. I play back the file directly in VLC, and I see there is a corrupted frame in that location. The thing is, the original file on the storage card media does not have this missing frame data at all. I downloaded an MD5 checksum application and compared the files to the original card. Sure enough, some files had matching checksums, and some did not (the ones that did not happen to be the ones with the corrupted data).

I thought maybe my card reader was going bad, but I just archived a large chunk of data from another USB drive, and lo and behold: same type of error popped up. I started thinking, maybe my USB drive is going bad? But CHKDSK and all other tests performed on it come back clean. I tried googling this problem, and imagine my surprise to have found this and many other threads discussing some sort of issue with Win 7 64 bit!

I'm beyond shocked that this has been an issue all along and I never knew about it. I blamed every other piece of hardware to no avail, when the problem was actually Windows itself!

Has anyone found a solution? I see in another thread someone mentioned his RAM had gone bad, but it is confirmed in this one that this isn't the issue for the OP.

...yes, I'm still running Win7 64 bit. I've got Win10 on some of my other personal computers, but my work machine has some legacy tools for media editing that isn't supported anymore. It took me a long time to set up properly and I don't want to fix what isn't broke by going to a new OS only to discover new problems. But not being able to trust Windows copy commands? This is a HUUUUGE ISSUE!