How to de-dupe 3 drives with some overlap, perhaps different versions?

[Riotubes]

Hi, this weekend I plan a fresh install of Win 7 Pro on a new SSD. The data will go on a 2nd HDD. After I install the OS, I'll need to port all data from 3 separate drives (legacy PC and 2 external HDs) to a new internal HDD in the new PC. Up to now, I have been backing up data manually and have lost track which files I've backed up, if they are separate versions, etc. With the new year and new PC I have committed to using windows backup or 3rd party app to do this automatically.

Is there a way I can send the data on all 3 drives through an app and have that app detect multiple versions so that I can delete duplicates...before loading the cleaned up data files onto my new PC?

Once accomplished, what is the best way to transfer the cleaned up data to the new HDD? Simply click, drag and drop onto the new drive or is there a program for this (video, flac, msoffice, app data files, etc.)?

Thanks!

 

[James Colbert]

I think you'd have to do this with 2 programs, one to copy the drives/partitions, the other to track down duplicates either before or after the Disk Copy/transfer.

I really can't recommend a duplicate file utility, but the free edition of Parttion Wizard will copy the drive/partition:

Compare Editions Partition Wizard

(this is a comparison chart that lists the features. Download links in the frame at left)

If you are not familiar with imaging programs and how to use them, the help files on the website should help. If not, post back.

Here is a google search for dup file finders. Be leary of the freeware in this category, as there seems to be a high incidence of malware in these free utilities.


Edit: Had to run out, but wanted to add: Partition copying is just one way to copy your data to your other drive(s), but, depending on how much data you have or what resources/knowledge you have, may not be the most convenient way.

You may want to remove the drives and connect them to the machine (via USB or whatever) that contains the new drive you wish to copy to and copy the files manually, or transfer them via network. Only you can decide what will work best for you. I myself find imaging quite convenient.

James

 

[ignatzatsonic]

I've never heard of a dupe finder for text type files (Word, Excel, etc), but they may exist.

I'd like to hear about that myself.

There are dupe finders that work quite well on pictures.

There are search tools that can quickly display all files that are exactly 145,675 KB in size, but you'd still have to make some type of manual judgment if they were in fact identical.

Your issue is complicated to the extent that substantially identical files have different names or slightly different sizes.

For the original transfer to the new HDD, you may as well use drag and drop. That is the simplest and most foolproof way to do it, although it might take a day or more, depending on amount and speed.

Thereafter, I would certainly use an automated program (of which there are many) that did NOT rely on imaging. Such programs only back up files that are new or have changed since the previous backup and for that reason typically take only seconds or a minute or two to run.

But you will be back in the toilet relatively quickly if you don't get a handle on folder structure and naming on your original file versions. An automated backup program relies purely on file name, directory, and file extension to distinguish between files. So if John.doc and Mary.doc have identical content, both will be backed up. If John.doc in folder X and John.doc in folder Y are identical, both will be backed up. So you'd end up with multiple copies again.

 

[Riotubes]

I wonder if syncback se or synctoy could be used to de-dupe duplicate files on different drives?

 

[MacGyvr]

Here you go, and it's free:

Find duplicate files with Auslogics Duplicate File Finder

 

[profdlp]

Here you go, and it's free:

Find duplicate files with Auslogics Duplicate File Finder

Great program.

 

[Riotubes]

Update: The Auslogics Duplicate File Finder works great! Just wondering if there is a way to scan to external HDDs both used for backup for the same PC...instead of seeking duplicate files...seek only those files that are on one drive but not the other? In other words, the vast majority of the files on these two HDDs are duplicates, and I want to search for only the "new" files.