Need a file copy recommendation with partial file resume

[robr]

I'm currently using Richcopy to move about 4TB worth of data from our colo facility (1000 miles away) to our office. And that data also gets routed overseas first before coming back (don't ask). Needless to say, connectivity while not bad, could be better.

The small files are zooming right along, the issue is the larger files (the ones that are hundreds of MB or even several GB).

From time to time, I briefly lose network connectivity for a second or two, when that happens, the files currently being copied abort and once connectivity is restored, Richcopy just moves on to the next files. So when Richcopy advertises that it 'resumes', it seems it is talking about resuming a file copy queue, and not resuming partially copied files. Those have to start over from the beginning.

Is there a tool similar to Richcopy with the functionality I'm looking for?

Thanks!

 

[Berton]

I'd look at some of the Download Manager choices on this list found by Google Search:
https://www.google.com/search?site=....0....0...1c.1.54.hp..3.20.2474.0.sI5DpbJXr1U

 

[robr]

Thanks download managers generally refer to httpbased downloads. I took a look at your search and that appears to be the case with the tools in the results.

 

[Alejandro85]

How do you perform the copy? Windows share over internet? FTP? Web server?
How are you doing it right now?

 

[robr]

It's just a straight up copy between two CIFS shares.

 

[Alejandro85]

I've used Teracopy to do large file transfer across Windows drives. I found it to be somewhat unreliable on the resume feature. Maybe give it a try and see if it actually fulfills your needs.

I would, however, consider using a different technology for such data amounts, specially over internet. Are you using any sort of VPN to connect the computers?

 

[robr]

There's a site to site VPN.

Teracopy does not appear to be able to resume a failed file from where it left off, I believe it restarts the file from the beginning (but I will test since I can't find any concrete info one way or the other). All these tools talk about resuming a failed copy, but none of them are clear that what they seem to mean by 'resume' is that if you have a lot of files queued, and there is a failure, it will retry the failed files and keep the queue going.

The only protocol I'm aware of that can actually resume a file is FTP, but I've found FTP to be significantly slower than a file copy. rsync will do it on *nix, not sure if that feature made it to the windows port but I'll check that tomorrow as well.

I may just have to fly down there with NAS in hand and do this on site. It's a once a year thing so if that's what I have to do, then so be it.

 

[Alejandro85]

FTP is more likely to be substantially faster than the built-in SMB, specially over internet.
Switching to it may be a good alternative. However, I would use SFTP instead for its added security. Many clients support resuming and it have tons of features besides file transfer. Using a web server can be done too, downloading with any ordinary browser, and maybe using all those "download managers" from previous posts.
Or what about BitTorrent? All clients natively support resuming and has a built-in checksum as a bonus. If there are a few computers sharing the same files it also may have significant speed gains.

 

[robr]

Security isn't a concern, everything is going over a VPN and adding another layer of encryption adds unnecessary overhead. FTP as a protocol has more overhead than SMB but I need reliability and ease of use over pure speed at this point. With the other technologies you mention, I don't know of any tooling that would work well to sync literally millions of files spread across multiple drives and directories. I'm open to recommendations as always.

So today I will be trying
- Teracopy
- rsync
- FTP