Solved Deleting mass files ending in specific characters

[COSenna]

Hello,

I have an external HDD formatted in FAT32 that has mass files that were duplicated. I would like to be able to delete the duplicate files which all seem to have been duplicated like this example:

This is for tens of thousands of songs and other file types in thousands of folders. Not only do I want to delete every files that ends in "1" "2" "3"... I also want to delete the added characters before and after each of the remaining files.

I am not the best at Google keywords so I found it quite hard to find a straight, simple answer on how to do this.

Thanks in advance for the help!

COSenna

 

[ignatzatsonic]

It's complicated a bit because the files are scattered across many folders.

Looking at your above list, which are the ones you would want to KEEP?

They all have the @ symbol. Which of those are the keepers?

Or maybe it doesn't matter, as long as you keep one version of all songs?

I notice that Wasn't Born To Follow isn't shown with an "@0" version. Or do you actually have an
"@0" version of ALL songs?

I've got an idea on how to do this, but need answers to the above questions.

Do it in two stages:

1: delete the duplicates

2: rename the survivors with a file-renaming program.

Regarding renaming files, suppose you have a file named:

235 Bob Dylan - Leopard Skin Pillbox Hat 54z.mp3

A file renaming program can easily chop off certain characters, leaving you with:

Bob Dylan - Leopard Skin Pillbox Hat.mp3

 

[COSenna]

Almost every song, and every file underneath a large file that was used to backup my old computer was duplicated 4 times. I only need one of each. Any of the files can be saved, although I was thinking of just saving all the files ending in "0".

If the songs are duplicated at all, then they were duplicated 4 times, and 4 times only, from what I've seen. I don't understand why about 90% were duplicated and the others were not...

Deleting the duplicates would take AGES. I originally had 70,000+ songs... now about about 4x that in duplicates? Yeah... I might just start my music library over to save some time/headache if that's the case.

Is there no program that I could DL/buy that would do this for me? Maybe some sort of batch cmd? It's hard to believe there isn't.

 

[ignatzatsonic]

Here's my idea. I've done similar procedures with this tool.

Get the "Everything" search tool from voidtools.com. It's a free download that you should have anyway. It's a fabulous drive search tool.

Run it. Configure it to index and search this external drive.

You search for a specific text string within the file name. It will cough up a list of files that match. You can then sort that list by clicking on the column headings.

If your files are named something like this:

Elvis - Hound Dog @3.mp3

then try @3.mp3 as a search term.

After you've sorted the list, scroll down to find a large group of duplicates that you know you don't want. There may be thousands of them in a row, so you can select the thousands all at once.

Highlight them all and hit the delete key.

They will be deleted from that hard drive.

Be careful, it's a powerful tool. Be sure you've highlighted only files you want to delete and are working on the correct drive.

After the first batch is deleted, then use other search terms, like maybe:

@2.mp3

@3.doc

@3.jpg

Or whatever term you think is descriptive and common to the unwanted duplicates.

Until you've exhausted all possibilities. I'm assuming the unwanted files have common characters and are identifiable by extension and something like @3.

 

[COSenna]

Sounds like a plan! Thank you kind sir, I'll give it a shot.

Edit... It seems as thought this program can only index NTFS, is that correct? The drive doesn't even show up in the program. :/

 

[ignatzatsonic]

Sounds like a plan! Thank you kind sir, I'll give it a shot.

Edit... It seems as thought this program can only index NTFS, is that correct? The drive doesn't even show up in the program. :/

I have no idea about NTFS only. Could be. Never tried other file systems.

Voidtools has an active forum. You could ask or perhaps they can point you to an alternative.

There are other ways to copy all of the files from multiple folders to single folder, but that won't do you any good because it's a copy, not a move. Your fouled up copies would remain behind.

Can you copy all files to some other drive that is NTFS and then maybe copy some back to the desired drive?

XXcopy is a powerful downloadable command line tool. I have minimal experience with it and have no reason to believe it can do a move rather than a copy. It can certainly copy files from multiple directories to a single directory with this command:

xxcopy source destination /sr

But again that won't help you.

All I can suggest is to pound Google.

It doesn't do you much good to get all your duplicates in a single folder if you can't then search them to differentiate between the ones you want to delete and the keepers.

Off the top of my head, if it were me, I'd try to get them on an NTFS drive and use Everything.

That tool can make a database of tens of thousands of files in a few minutes. It can then search that database for a given search term almost instantaneously. I'd guess you could do the entire job in under an hour.

I'd assume some heavy duty command line guru could come up with a tool, command, or program that would work, but that's beyond me.

 

[Berton]

It seems as thought this program can only index NTFS, is that correct? The drive doesn't even show up in the program.

Since it's a Windows-based application it most likely is limited to NTFS and maybe FAT32, can't work with Mac OS X and Linux formatting.

 

[billmcct]

I agree with ignatzatsonic Everything search works on fat32 drives and is very fast. The instructions he gave in post 4 are correct. You also can use wildcards too such as "*@3*".

Everything Search Engine

Bill

 

[cjkuhlenbeck]

Or you could avoid buying/installing additional software for a built in function.

In search, use

System.FileName:~="@1"

That will search a folder for all files containing @1. You can change it to

System.FileName:~="@2"

and

System.FileName:~="@3"

for the others. Or install the apps, i dk your choice.

EDIT: In case anyone is interested, here's a full list of the search commands - Advanced tips for searching in Windows

DOUBLE EDIT: You can also do a combo search:

System.FileName:~="@1" OR System.FileName:~="@2" OR System.FileName:~="@3"

This searches for any files containing: @1, @2, or @3. From there just scroll to make sure it didn't catch anything you want to keep, then CNTL+A~Delete

 

[billmcct]

/Not only do I want to delete every files that ends in "1" "2" "3"... I also want to delete the added characters before and after each of the remaining files./

I was assuming the 05,06,07, etc in front was the track numbers. Once you have deleted all the files you want deleted you will need to use a renaming utility to remove the characters you don't want in the names such as bulk rename utility.
Both utilities mentioned are free.

Download - Bulk Rename Utility

Bill

 

[ignatzatsonic]

COSenna:

Look at this link:

Indexes

which says, among other things:

By default, "Everything" indexes all local fixed NTFS volumes.

Everything will automatically index all fixed NTFS volumes.

Use folder indexing to include mapped network drives, CDRoms, DVDRoms, FAT and FAT32 volumes.

And here's what the folder indexing link says:

Folder indexing allows Everything to include any folder in the Everything index.

Indexed folders can go offline and remain in the Everything index.

For example, with folder indexing you can index:

Network shares or mapped network drives.
FAT32 and other volumes.
Any physical folder.

Folder indexing does not require administrative privileges or the Everything service.

Why is indexing so slow?

Folder indexing uses the same approach as the Windows search.

This can be a lot slower than NTFS indexing.

Everything can take a couple minutes to scan a folder and all its subfolders and files.

How do I add a folder to the Everything index?

In Everything, from the Tools menu, click Options.
Click the Folders tab.
Click Add....
Select a folder to add to the Everything index.
Click OK.
Click OK.

 

[cjkuhlenbeck]

Maybe I'm a sucker for not wanting to install an app that you'll only use once and then leave it there forever on the chance that you'll need it again. Creating unnecessary registry keys and the likes. But I'm going against the grain here and say batch renaming can also easily be done in powershell, and I believe I could write a pretty simple batch file to do it in command prompt.

Seems kinda silly to me like an app to clone a disk when it's a standard function you can do in 5 parts of windows, and in some cases easier than the third party apps. But there's millions out there for just that purpose. Or, in your specifc case, why have an app generate an index of a drive when you can just pony off the windows index. And most of those apps do the operations in cmd in the background anyways.

But what works best for you is the best choice. If you're more comfortable with some freeware apps, use those.

 

[billmcct]

Well, frankly I personally use Everything search exclusively. I have also disabled Windows indexing. It sits in the right click menu and is many many times faster than Windows search, and where windows search will show you all sorts of irrelevant files in the results, Everything only returns relevant results. As the OP I have 1.2 TB of music files in almost 7000 folders and when all of a sudden about 5000+ of those folders decided to duplicate the 80000+ files in them I was able to use Everything to fix the problem in minutes not hours. Glad I had it on my system.
Secondly not everyone knows how to write batch files so nuff said.

Bill

 

[COSenna]

So I moved all the music and files to a NTSC drive and used "Everything" to delete all the duplicates. Now as for renaming, I downloaded the Bulk renaming utility but I don't think it is very "bulk" in renaming. An indexing renaming tool is really what I need.

After doing a google search I ran into this program...

Métamorphose : A File and Folder Renamer - Powerful batch renamer

Does that look like what I would need? I really wish the "everything" program could rename, I'd be completely done in about 10 min!

 

[ignatzatsonic]

See post 11. You didn't have to move those files to an NTFS drive. Everything will search FAT32 by folder.

I know nothing about Metamorphose. On the face of it, it looks like it would work.

Not sure what you mean by an "index renaming tool".

I can vouch for Flexible Renamer, here:

The Portable Freeware Collection - Flexible Renamer

It's portable, you run it directly off the exe file rather than installing it.

There's another good one called "Name It Your Way", but I don't know where to get it nowadays. The downloadable file is named niyowins.exe. You might be able to find it with that search term.

I urge you to make copies of all files you intend to rename before starting. Renaming tools are powerful and something could go wrong on thousands of files in one operation that you could not easily undo, although the better ones have "undo" functions.

 

[COSenna]

I went ahead and moved it all to the new drive before checking the forum and seeing I could've done it in FAT32. Its probably a good thing however, in case something goes horribly wrong!

What I meant by indexing was pretty much what the Everything program can do, which is look for all files in all folders and make a list of them. This way I could rename a massive amount of files instead of going under each folder and highlighting everything inside and renaming them in their individual albums. Even with all the duplicates eliminated this thought of this makes me cringe.

This may be asking a little much from a free renaming tool, but maybe as someone had said, a batch cmd? I have little to no experience in such though so any help would be greatly appreciated!

 

[ignatzatsonic]

Give examples of your folder structure and what your remaining files now look like. Several examples.

In my experience, a renaming tool works recursively down into sub-folders.

So--imagine this folder structure:

MP3 is the parent. Inside it are genre folders, like rock, blues, country. Inside the genre folders are artist folders like Johnny Cash, Elvis, Muddy Waters.

So, for individual files, you might have:

MP3/country/Johnny Cash/034Johnny Cash - I Got Stripes^^^^333388.mp3

MP3/blues/Muddy Waters/442MuddyWaters - I'm Ready454.mp3

MP3/rock/Elvis/223Elvis Presley - Mystery Train565.mp3

A renaming app can operate on all 3 of those songs at the same time to eliminate the unwanted characters, leaving you with the likes of Elvis Presley - Mystery Train.mp3.

The app can cut characters from any position in the file name, whether at the beginning, end, or interior. You may have to issue a series of commands, one after the other. The first to chop from the tail, the second to chop from the beginning, the third to chop from somewhere in the interior, etc.

Still not quite following why you think this would not work for you.

I'm assuming the remaining files are poorly named in some recognizable pattern or patterns---e.g. some are "wrong" in this way and some are "wrong" in that way. Attack this pattern with this command and attack that pattern with that command, separately.

 

[COSenna]

All of my folders are laid out like this:

MyMusic/Artist/Album/Song Name

I just found however that there are a ton of songs that start like this:

01-01 [email protected]3
01-02 [email protected]3
...
20-01 [email protected]3
20-02 [email protected]3...

And a lot like this:

1-01 [email protected]3
1-02 [email protected]3
...


But most like this...

01 [email protected]3
02 [email protected]3


I have a feeling I am screwed.

 

[ignatzatsonic]

I'm still not clear on the precise difference between what you have and what you want.

List some actual true and complete file names as they show in Explorer. Not what they are "like".

Is the 01, 02, 1-01 stuff at the beginning actually part of the file name? Or not.

Or is the only thing wrong just the trailing @0 just before the dot mp3? That can certainly be eliminated.

For instance, consider:

01-01 [email protected]3

How much of that is a filename per se? Do these MP3s have tags? What do you want that example to be instead? 01-01 Songname.mp3? Songname.mp3? Something else?

Confused still.

 

[COSenna]

Here is a screen shot of a section opened with Everything:

The 3-01, 3-02, etc., is what I do not want.

Where I highlighted "2am" is the name of the actual song. Therefore there are many songs which did not get the added numbers to the beginning.

The songs with 27, 28, 29 in front of them are incorrect as well.

With the name editing program I used (Bulk Rename Utility,) I was not able to find any option except delete per se, the first two, or last three characters in the name of the file. Since the first characters in any given file are not of the same amount then I cannot use this option.

If I were to delete the first two characters of every file then I would have the song "2am.mp3" be renamed to "m.mp3"