Managing a large Photo/Video Collection

[garyguertin]

Background: Over the years I have collected somewhere between 60 and 100 thousand photos and thousands of hours of video in all kinds of formats (mostly jpg and muv). What's worse, doing backups and copies, I may have as many as 5 or 6 copies of any one photo or video. These are spread across two computers and dozens of CD's and DVD's.
Problem: I need to find a program that will go in and tell me not only how many copies of the same photo or video I have, but more importantly which is the best copy(ies) of/or the original, so that I can manage the files down to an original (or best copy) and one backup (next best copy). It needs to have batch capabilities because with that many photos/videos I could never do them one by one. Once I do it for the first time, it needs to be able to scan a CD or DVD of photos or videos and do the same job against the master file(s). Memory space and speed are not a problem because I'm getting a new computer with max memory and the fastest Intel chip.
Can anyone help me to find the best program to manage my collection? Thank You

 

[rap33042]

Welcome to SF, garyguertin!


The only one I've found that doesn't cost a ton of cash and that I use all the time is DupDetector.
It is designed solely for the purpose of finding duplicate Pictures.
I'm not sure if it is even supported anymore. It's one of those old, rare gems.

I used it on XP for years without a hitch; however, on Windows 7 it has a few display snags.
It still works, though- you just have to deal with them.

Anyway, here's the link: http://www.keronsoft.com/dupdetector.html

Edit: I just looked at mine and it is version 3.201. Google around and you'll find the latest version, I'm sure.
Incidentally, notice the date? 2005! Too bad because it's a great utility.

You can fine-tune the settings until you're happy with the results, otherwise, it will give you a lot of false matches.

Enjoy! and I hope this helps...

 

[garyguertin]

Rap33042, Thanks for the advice. I'll give it a try. In the meantime if you or anyone else has some other programs to try let me know Again, many thanks. GG

 

[mitchell65]

I think your main problem will be finding a program that can "decide" which of a multiple choice of files is the best. The parameters for this are endless especially in the realm of video clips. Have you got the same files on the PC's as are on one or more of your backup discs? If so this is going to be an added problem.

 

[garyguertin]

Mitchell65,
You are right, the only thing I can thank of is that everyfile (jpg, etc.) should have a date and some a time included. I would need a program that would put the same files (which it would determine) in an order by date. I could then erase all but the oldest two, which should be the original or first copy plus the next copy. Those should have the best quality. As to what's on CD's and DVD's, most of them (98%) are backup copies of files on my computer, but because I have changed computers about every 2 or 3 years I may have better quality files on those discs than I have on my current computer. You can see why I really need a good program to do this job and not just your normal dup file program. I have not tried keronsoft's DupDector, but I have a program called Duplicate Finder and Easy Duplicate Finder, which does an excellent jop of finding duplicate files but does not address quality or date of the file, it only shows location on your computer or DVD.. I will try DupDector, but if you or anyone has other programs to recommend, i'll try them. Thank You for your comments, GG

 

[ignatzatsonic]

I used to use a dupe finder program--maybe 6 to 10 years ago.

As I recall, all it could do is find identical file names and/or identical file sizes.

The user then had to do an eyeball inspection to confirm that the 3 files with the same name were in fact the same image---or that the files with the same sizes were in fact the same video.

Etcetera. I suspect you are going to be up against that situation.

Even if you found a program that could supposedly do what you need, would you trust it without manually confirming its opinion?

And I'd guess you are completely hosed if you have two identical images that have different file names. Or can software be savvy enough to know that Jones.jpg and Smith.jpg are otherwise identical---or that 2 files with the same name and extension are in fact completely unrelated??

If you find anything, let us know.

 

[rap33042]

DupDetector doesn't pay any attention to file names or file sizes unless you specify it in the options.
It can be set to 3 deletion types: Manual, Semi-auto, and Auto
It will detect flipped images, color vs gray-scale, and can be set to 1%-100% (mine's set at 96%) match and sort the list accordingly.
It will build a list from a directory, or two or more directories. It will save those lists for future use if you want.
It has several matching algorithms to choose from.
The list goes on...

It's display in Windows 7 is a bit quirky but the program works flawlessly.
I use it all the time and my only gripe is that it hasn't been updated in years.

I have about 80000 pictures and that number is still growing.
Anyone who needs to check through large numbers of pictures for duplications should really give it a try!

 

[ignatzatsonic]

OK; I will try DupeDetector out. I will only occasionally have use for it, but I can envision situations when it might be handy.

 

[garyguertin]

Thanks rap33042, I'll give the program a try. Again, if anyone knows of other programs let us know. Again, Thanks, GG

 

[ignatzatsonic]

I tried it. It works pretty well:

Choose get data tab and hit “build”. Navigate to chosen folder with browse. Choose OK. It will scan through all files in the chosen directory and its subdirectories. Takes less than 10 minutes for 10,000 files. When that finishes, choose “find dupes” tab. On that tab you can choose a percentage for the match algorithm.

I chose 96% per advice above.

It will then scan the scanned files, looking for file pairs that are 96% or better matches. After that, choose view images. You are shown each image pair, along with each picture’s path, size, and dimensions. You can choose to delete one of the pair or move on to the next pair.


In 3 or 4 cases out of a hundred or so, the matched pairs in fact were not matches at all or even close--entirely different subjects in fact.

But you can pick that up by eye. I assume choosing a percentage above 96% would reduce these "false positives".

I didn't see a way to bring up full size images of each of the images in the pair to take a real good look. I just used a second program to do that.

So, I'm happy---I deleted a hundred or so dupes and then uninstalled the app. I can reinstall it a year from now and run through the routine again.

 

[rap33042]

There's an option in the View menu to enlarge the dupes views; that's as good as it gets, though.
You're right about the percentage. If you raise it too much, you'll maybe miss actual duplicates.
It takes so little time to 'find dupes' once the list is compiled, that changing the percentages is negligible, time-wise. Just click the Find Dupes button again to see the change in the number of dupes it finds.

Glad you liked it and that it was helpful!
---

Remember I mentioned the GUI was quirky?
Well, on my screen I'm missing half the 'Back/Next' buttons (shown in screen capture).
Are you experiencing this same problem?
Do you know how I could eliminate this?

I have tried various options in Windows compatibility settings to no avail.

TIA!

 

[garyguertin]

Hi All, I tried what I thought was the DupeDetector we were talking about It was a sub-program of PC Unleashed Online. when I got to data and file duplicates, I didn't find any of the settings you talked about, just the opposite, it only asked me where I wanted to search and choose what kind of files I wanted to find dups.. I ran it, and like all the other programs I have used it just showed me duplicate files by their location, then I could mass erase the dublicates or select the ones to delete manually. There was no indications of date of files or any other way to determine which were the better quality files. First of all, did I have the same program you were talking about? If not where do I go to get the same program? Second of all, does anyone know where you can find a program which shows date of file or other info which could reflect quality of the file??? Thanks for your comments and Happy New Year

 

[FliGi7]

If you're into command line utilities, I use ssdeep fuzzy hashing to determine duplicates and possible duplicates. Fuzzy hashing allows you to determine not only if two files are the same or similar, but approximately how similar they are.

Fuzzy Hashing and ssdeep
Usage would be:> ssdeep -lrd \Directory\of\images

 

[garyguertin]

Thank You FliGi7. I'm not into command line utilities, but meanwhile I did some digging into some old programs I had and found an Ashisoft.com program called Duplicate Finder v3.5.3 which comes as close to my "wants" as I'm going to get. It allows you to do a byte by byte comparison and gives the date of each file. It has a lot of options and appears to be as complete as your going to get. The only disadvantage is you can try it for awhile free, but then have to pay around $30.00 for keeping it. I think this is my answer, but again if anyone has a better sugguestion let me know.