Quickest Way To Move 160GB?

[bobtran]

Bob, correct me here if Im wrong, but TAR isnt a windows file ... its Linux...

This process is also available for the windows platform although you would have been correct a few years ago. You can also get gzip and many other linux utilities for windows based boxes.

You will find a collection of GNU utilities (including gzip and tar) ported to Windows here:

Native Win32 ports of some GNU utilities

You can also use 7-zip to create a tarball (file ext .tar) or purchase a product called Wintar.

 

[bobtran]

Without in any way wishing to play down others' suggestions, I don't believe file compression or "cabification" is going to make any difference in your scenario (two local drives). In fact, it would take (far) longer to first compress 160GB into a tight ball, then copy, then decompress on the target drive. That strategy is valid under certain circumstances - very slow network links for example - but the "bandwidth" of the underlying hardware between the source and destination is not the primary bottleneck in this instance. Instead, what's important is the way your chosen copy tool breaks up data into segments and invokes OS functionality to copy it across.

I agree, avoid compression. It will take long to compress and uncompress than it would to just move them.

Not quite accurate...compressing data on the box prior to transmitting over data lines will be faster than sending uncompressed data over a usb or even broadband depending on size of file. I agree that for small files there will not be any savings (and it may take longer to compress/decompress) but as files get larger the savings multiply. compression can result in 30%-50% smaller files, (sometimes even more) unless the files have fixed field length date fields instead of variable length data fields, which can substantially reduce the time/bandwidth required to move a file. Running a compression or decompression natively is extremely fast unless you are using a really old box. Furthermore you don't need to decompress the files at storage end unless you want/need access to the contents there.

I do agree that the easy way is to send it native (The original post specified fastest way to send 160gb, not ease of process).

 

[H2SO4]

Not quite accurate...compressing data on the box prior to transmitting over data lines will be faster than sending uncompressed data over a usb or even broadband depending on size of file. I agree that for small files there will not be any savings (and it may take longer to compress/decompress) but as files get larger the savings multiply. compression can result in 30%-50% smaller files, (sometimes even more) unless the files have fixed field length date fields instead of variable length data fields, which can substantially reduce the time/bandwidth required to move a file. Running a compression or decompression natively is extremely fast unless you are using a really old box. Furthermore you don't need to decompress the files at storage end unless you want/need access to the contents there.

I do agree that the easy way is to send it native (The original post specified fastest way to send 160gb, not ease of process).

By way of manly experiment...

Source: [local drive] W:\MX\*.* (about 4.6GB worth of various media files)
Dest: [local drive] V:\
Machine: unadulterated Q6600 with 8GB of RAM running W7x64

First, a straight robocopy, uncompressed:

W:\>robocopy w:\mx v:\mx *.* /E /MT:8
...
Total Copied Skipped Mismatch FAILED Extras
Dirs : 35 35 0 0 0 0
Files : 831 831 0 0 0 0
Bytes : 4.683 g 4.683 g 0 0 0 0
Times : 0:21:53 0:02:09 0:00:00 0:00:16

Two minutes and nine seconds for the copy.
Effective throughput without compression: ~37 MBytes/sec.

================================================

Now using (command-line) 7-zip to first compress those same data with the default compression settings:

C:\Program Files\7-Zip>7z a -r w:\ w:\MX\

It took 27 minutes (frickin' thing doesn't do stats by default and I couldn't be bothered doing it again) to produce this archive:

W:\>dir .7z*
 Volume in drive W is Mirror
 Volume Serial Number is A4F5-EDC0
 Directory of W:\
16-Oct-09  12:07 PM     4,911,076,674 .7z
               1 File(s)  4,911,076,674 bytes
               0 Dir(s)  32,615,526,400 bytes free

Now copying that across to the second drive, again using Robocopy:

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         1         0         0         0
   Files :         1         1         0         0         0         0
   Bytes :   4.573 g   4.573 g         0         0         0         0
   Times :   0:03:21   0:00:50                       0:00:00   0:00:50

Total time taken: 27min + 0:50 = 27:50
Effective throughput with compression: ~2.8 MBytes/sec

The compression strategy would be worthwhile if we were traversing a "wet string" network link. For local-to-local copies it's counter-productive. My technique may be questionable, but even if the "with compression" results were an order of magnitude better they'd still fail to beat the uncompressed copy.

 

[bobtran]

Your Machine: unadulterated Q6600 with 8GB of RAM running W7x64

27 min to compress the 4.6gb file? I think something is wrong there.

My machine: also unadulterated Q6600 with 8GB of RAM running W7x64
You've got to love a good thing.

Using gzip compressed 7.5gb of ISO's in a little over 5 min (20% compression rate, I would expect better from basic documents but who knows). I will give you the fact that for 160gb native transfer will probably meet or exceed the speed of compressing and then transmitting to a usb/firewire attached drive. I'm actually used to doing Databases.

I bow to your proof although I think gzip is a better compression tool.
I DO use 7zip myself for normal zip/unzip and find it a good and worthy tool.

Thanks for making me check this out, having a hard time talking around my foot....I learned something tonight. I will have to check out the robocopy product although I get those speeds without it.

 

[A320]

Quickest way to move it you say....

Just lift up the HD from original spot, and place it down in new spot!

I know, sorry couldn't resist.

 

[H2SO4]

27 min to compress the 4.6gb file? I think something is wrong there.

Yeah, it was downright painful. I don't think it's broken though. 7zip just seems rather aggressive in its default compression, and the fact that the content was not particularly compressible probably didn't help.

... although I think gzip is a better compression tool.
I DO use 7zip myself for normal zip/unzip and find it a good and worthy tool.

I picked 7zip for the test mostly for its independence from Windows itself, and for its good reputation. I doubt any compression utility could do it 20 times faster, which is what it would take to approach the uncompressed throughput.

Thanks for making me check this out, having a hard time talking around my foot....I learned something tonight. I will have to check out the robocopy product although I get those speeds without it.

Thank you for an interesting discussion

Robocopy is built into the OS. That by itself is worth a lot to me, given my own typical usage patterns.

 

[jimbo45]

Hi all

Have I missed something here - but why not use something like Acronis - it will back up FILES as well as entire Disk Images (partitons).

1) start TI.
2) Select Data
3) click against folders you want to copy
4) backup to external HDD using with highest compression ratio
5) restore to target disk

job done -- won't take that long either.

Cheers
jimbo