Years long copy (numerous small files)

[sweiller]

Hi all,


look at the picture

I am making a backup of lots of small files (Android, Genie_timeline, PC systems, etc ...) from one internal hard disk to another one USB3. If I transfer large files the speed is about 150 MB/sec !!
Here with small files it changes from a few KB/sec, sometimes bit more ... I cannot wait 20 000 hours ~ 2 years !!!!
I use TotalCommander but TeraCopy is no better !
I tried to make grouped Zip files but this also will take a very very very long time !


Is there a known way to copy small files faster or not ?
Would Linux do better ?


BR

 

[torchwood]

Hi Br,

know of these


Xcopy
Robocopy
or macrium free

Start Using XCOPY to Speed up Transferring Files | TurboFuture


ROBOCOPY - Create Backup Script


Roy

 

[sweiller]

Hi,
Tried Xcopy /S /C /Y /D K:\ E:\DISK-K\
Was working - I don't know how fast - but stopped with error :
Insufficient memory
48473 file(s) copied


??
BR

 

[torchwood]

Hi
interesting article and fix


Xcopy: Insufficient Memory | Terminally Incoherent


Roy

 

[Alejandro85]

This is absolutely normal and expected, as your use case is among the worst possible situation for an operating system. Root problem is that the OS must ensure that the file system is consistent after each individual file, which means not only writing the file itself but also the system areas where files are indexed.

With large files, the data itself dominates the copy time, the drive bandwidth being the limiting factor. But with tons of tiny files, the bureaucracy of the filesystem produces lots of random writes (instead of sequential) which are much slower than writing a large block. Constantly going back and forth among those tasks noticeably slows the whole thing down, basically wasting the drive speed.
It's an inevitable fact that such operation will take a lot of time.

How many files are we talking about? Of what approximate size?
Using another tool may give a small boost and probably a better estimate (Windows Explorer is widely known for its non-sense time estimations). You could give some Linux a try at this, but I doubt that it could do noticeably better, the same problem still exists there.

 

[townsbg]

English is the official language here. Most here won't know what to make of that screenshot. You might try putting the files in a archive using either windows explorer, winrar, or 7z.

 

[sweiller]

Hi all,
Thanks.
I see there are no real fancy solutions.
Maybe will give another try to robocopy.
Making archives take also as I said a very long time.

 

[townsbg]

You might try doing a few folders at a time.

 

[pcplague]

If possible remove HD from case and connect internally, should be sata drive, so if you have a sata cable and a spare power connection in your pc case ...