Copying multiple files in a list, from one folder to another...

No the "large master list" is just a folder that stores all our files. I create the txt file from a BOM that is export out to excel and save as txt after doing some filtering.

I will try your code here in a little bit and let you know. I appreciate the help.

UsernameIssues this code worked and i updated the path as recommended. Make it very easy to navigate. The only problem that i had was, i have to make sure the txt file names are exact with the extension. 12345.XYZ

But i can manage that, thanks again for you help and i will be digging into scrips now to better understand them as i can see lots useful options.

Glad that it worked for you.

Yep, scripts can help a lot on repetitive tasks.

If the filtering that you do before making the text file list is not too complicated, then the script might be able to do that for you: e.g the script would only copy files that met your criteria. If that can be done, then you won't need to make a text file list.

If the source folder is always the same, then there is no need for the script to ask you to select it each time... just hard code that info into the script. Replace these two lines...

Code:

$SourceFolder = FileSelectFolder("Select the source folder.", "c:\temp\s")
If @error Then Exit

...with this single line:

Code:

$SourceFolder = "c:\temp\s"

...replacing "c:\temp\s" with the desired network path.

If the destination folder is created each day (perhaps using a date/time info), then the script can do that for you too (without you having to manually create the destination folder and then navigating to it when the script runs).

Yes for what I want to use this for my source folder is always the same, so I hard coded it in.

Filtering is a little complex but doesn't take just a min to create with Excel.

As for my destination folder I created a generic folder, once the scrip runs then i can manual move the files to the destination named folder.

Question, can a scrip check the "master list folder" and replace what is newer in the destination folder (test folder 1)? This will help with revisions that are saved into the "master list folder", Also could it check sub folders (test folder 1, test folder 2, test folder 3)?

It can be done, but I would change to a batch file for this (instead of AutoIt).

I've asked another forum member to stop by this thread. I'm not that confident with making that type of batch files.

Hello. UsernameIssues has asked of me to “stop by”, so here I am, with the batch file of course… I’ll be providing your assistance, Goofybt, from now.

goofybt said:

Question, can a scrip check the "master list folder" and replace what is newer in the destination folder (test folder 1)? This will help with revisions that are saved into the "master list folder"

Firstly, I’m not too sure of what to make of this. If you simply mean that the script should perform an overwrite copy to the destination location, that’s a non-problem. If not, perhaps you’d like to clarify your situation for me.

goofybt said:

Also could it check sub folders (test folder 1, test folder 2, test folder 3)?

As a popular feature, my batch script does support searching through subfolders for files (use /s switch).

Goofybt, try using the script below. I’ve been working on it for some time now, but I can’t guarantee it will work with your situation right now as network locations are involved and I haven’t tested it with one yet. Unlike AutoIt, network locations are one thing to consider when writing in batch and must be coded especially to support it.

If you’re unfamiliar with the command line, you can just edit the defaults near the top of the script and double click the file to invoke the script that way.


FindAndCopyFromList.bat

Code:

:: FindAndCopyFromList.bat | v2.0 | 11 SEP 2016
:: Created by Pyprohly @ https://www.sevenforums.com/member.php?u=406309

:: FindAndCopyFromList.bat
:: FindAndCopyFromList.bat filelist [-c:action] [-f] [-s] [-a] [-q]
:: FindAndCopyFromList.bat filelist source -c:"DELETE" [-f] [-s] [-a] [-q]
:: FindAndCopyFromList.bat filelist source -c:"LIST" [-f] [-s] [-a] [-q]
:: FindAndCopyFromList.bat source destination filelist [-c:action] [-f] [-s] [-a] [-q]
@echo off
setlocal EnableDelayedExpansion

:::.defaults
set "filelist=.\list.txt"
set "source=%~dp0"
set "destination=.\New Folder"
set "action=COPY"
set "force=FALSE"
set "transverse_subfolders=FALSE"
set "adapt_current_directory=FALSE"
:::

::. ************ DOCUMENTATION ************
::?.FindAndCopyFromList.bat v2.0 documentation
::?.
::?.SYNOPSIS
::?.	Locate and perform an action on the set of files specified in a given
::?.	newline-separated text file list.
::?.
::?.INSTRUCTIONS
::?.	Use this tool in command line mode by specifying a source folder, a
::?.	destination folder, a newline-separated text file comprised of file
::?.	names, and a predefined action as arguments, or pass only a filelist
::?.	and an action. Argument take precedence over the defaults.
::?.
::?.	Alternativly, omit specifying any arguments and simply execute this
::?.	this script as is to assume all the defaults in place of arguments.
::?.
::?.SYNTAX
::?.    FindAndCopyFromList.bat
::?.    FindAndCopyFromList.bat filelist action [-c:action] [-f] [-s] [-a] [-q]
::?.    FindAndCopyFromList.bat filelist source -c:"DELETE" [-f] [-s] [-a] [-q]
::?.    FindAndCopyFromList.bat filelist source -c:"LIST" [-f] [-s] [-a] [-q]
::?.    FindAndCopyFromList.bat source destination filelist [-c:action] [-f] [-s] [-a] [-q]
::?.
::?.PARAMETERS
::?.    source      If -s is not specified, the directory of which the script
::?.                should search files for, else this will act as the location
::?.                of where the script is to begin looking for files.
::?.
::?.    destination    The location of where files are to be sent when found. If
::?.                the -f switch is specified, this location will be created if
::?.                it does not exist. The location may be relative. If the -a
::?.                switch is used, the relative location will be relative to the
::?.                path of the current item being processed rather than the
::?.                current working directoy.
::?.
::?.    filelist    A text file containing a newline-separated list of file names.
::?.                File names listed in this file must not be followed by a path
::?.                and each entry must include an extension as matches are
::?.                processed exactly. Lines beginning with a semicolon (;) in
::?.                this file will be ignored.
::?.
::?.    action      The operation to perform on a located file. This may be one
::?.                of the following values: "COPY", "MOVE", "DELETE", or "LIST".
::?.                The "LIST" action may be used to verify the items that will
::?.                be processed, as well as those that won’t be, without
::?.                affecting files. If "LIST" or "DELETE" is specified, the
::?.                destination parameter will become meaningless and
::?.                non-mandatory.
::?.
::?.OPTIONS
::?.    -f          Overwrite files in the destination directory should file name
::?.                conflicts occur. Destination directories will be created if
::?.                they do not exist. Meaningful only with COPY or MOVE actions.
::?.                This switch is synonymous to the "force_action" default.
::?.
::?.    -s          Search through subdirectories for file name matches. If
::?.                multiple matches of the same file name are found, all
::?.                occurrences will be reported and no action will be taken.
::?.                This switch is synonymous to the "transverse_subfolders"
::?.                default.
::?.
::?.    -a          Before an action is performed on an item, change the working
::?.                directory to that of the item. This option is useful when
::?.                specifing a relative path for the destination location. E.g.,
::?.                if MOVE and "New folder" is specified for the action and
::?.                destination respectively, located files will be moved into
::?.                their own "New folder" directory (if exists. Consider using
::?.                -f switch) instead of them all being moved into a single
::?.                ".\New folder" location in the source location. Note, this
::?.                option does not aid againt the multiple file occurrence error
::?.                attainable when using -s. This switch is synonymous to the
::?.                "adapt_current_directory_before_action" default.
::?.
::?.    -q          Suppress warning when using DELETE action.


goto :main

:initialise_paramters
	if defined ARG_1 (
		set "filelist=%ARG_1%"
	)

	if defined ARG_2 (
		set "source=%ARG_2%"
	)

	if defined ARG_3 (
		set "destination=%ARG_3%"
	)

	if defined ARG_K_F (
		set "force=TRUE"
	) else (
		if defined FORCE (
			if /i "%FORCE%"=="TRUE" (
				set "force=TRUE"
			) else (
				set "force=FALSE"
			)
		)
	)

	if defined ARG_K_S (
		set "transverse_subfolders=TRUE"
	) else (
		if defined TRANSVERSE_SUBFOLDERS (
			if /i "%TRANSVERSE_SUBFOLDERS%"=="TRUE" (
				set "transverse_subfolders=TRUE"
			) else (
				set "transverse_subfolders=FALSE"
			)
		)
	)

	if defined ARG_K_A (
		set "adapt_current_directory=TRUE"
	) else (
		if defined ADAPT_CURRENT_DIRECTORY (
			if /i "%ADAPT_CURRENT_DIRECTORY%"=="TRUE" (
				set "adapt_current_directory=TRUE"
			) else (
				set "adapt_current_directory=FALSE"
			)
		)
	)

	if defined ARG_K_Q (
		set "suppress=TRUE"
	) else (
		set "suppress=FALSE"
	)

	set "action_valid=FALSE"
	if defined ARG_K_C (
		for %%I in (C M D L COPY MOVE DELETE DEL LIST INV) do (
			if /i "%ARG_V_C%"=="%%I" (
				set "action=%%I"
				set "action_valid=TRUE"
			)
		)
	) else (
		for %%I in (C M D L COPY MOVE DELETE DEL LIST INV) do (
			if /i "%ACTION%"=="%%I" (
				set "action=%%I"
				set "action_valid=TRUE"
			)
		)
	)
	if "%ACTION%"=="C" set "action=COPY"
	if "%ACTION%"=="M" set "action=MOVE"
	if "%ACTION%"=="D" set "action=DELETE"
	if "%ACTION%"=="DEL" set "action=DELETE"
	if "%ACTION%"=="L" set "action=LIST"
	if "%ACTION_VALID%"=="FALSE" set "action=INV"
goto :eof

:arg_parser
	set "tempfile_args="%TEMP%\%~nx0_arg-parser.tmp.js""
	if not exist %TEMPFILE_ARGS% exit /b 2
	for /f "tokens=1,* delims==" %%I in ('set ARG_ 2^>NUL') do (set "%%~I=")
	set /p ARG_0=<%TEMPFILE_ARGS%
	>%TEMPFILE_ARGS% echo Args=WScript.Arguments;for^(p_count=0,n_count=0,i=0;i^<Args.Length;i++^){item=Args.Item^(i^);if^(item.slice^(0,1^)=='/'^|^|item.slice^(0,1^)=='-'^){colon_index=item.indexOf^(':'^);equal_index=item.indexOf^('='^);min=Math.min^(colon_index,equal_index^);max=Math.max^(colon_index,equal_index^);index=min==-1?max:min;arg_name=index^>-1?item.slice^(1,index^):item.slice^(1^);WScript.Echo^('ARG_K_'+arg_name+'=TRUE'^);if^(index^>-1^){WScript.Echo^('ARG_V_'+arg_name+'='+item.slice^(index+1^)^)}n_count++}else{WScript.Echo^('ARG_'+(p_count+1)+'='+item^);p_count++}}WScript.Echo^('ARG_COUNT='+i^);WScript.Echo^('ARG_POSITIONAL_COUNT='+p_count^);WScript.Echo^('ARG_SWITCHES_COUNT='+n_count^)
	for /f "delims=" %%I in (' cscript /nologo %TEMPFILE_ARGS% %ARG_0% ') do (set %%I)
	del %TEMPFILE_ARGS%
	set "tempfile_args="
goto :eof

:arg_parser__validate_args
setlocal
	if defined ARG_4 echo This script takes 3 positional arguments at most but %ARG_POSITIONAL_COUNT% were given&& exit /b 1
	set /a or=0
	if not defined ARG_3 if defined ARG_2 if /i not "%ARG_V_C%"=="LIST" set /a or+=1
	if not defined ARG_3 if defined ARG_2 if /i not "%ARG_V_C%"=="LIST" set /a or+=1
	if %OR% gtr 0 echo Parameter set cannot be resolved using the specified parameters&& exit /b 1

	set /a paramter_set=-1
	if defined ARG_3 (
		set /a paramter_set=0
	) else if defined ARG_2 (
		set /a paramter_set=1
	) else if defined ARG_1 (
		set /a paramter_set=2
	) else if not defined ARG_1 (
		set /a paramter_set=3
	)
	if %PARAMTER_SET% equ -1 echo Parameter set cannot be resolved using the specified parameters&& exit /b 1

	set switch_optional_counter=0
	if defined ARG_K_F set /a switch_optional_counter+=1
	if defined ARG_K_S set /a switch_optional_counter+=1
	if defined ARG_K_A set /a switch_optional_counter+=1
	if defined ARG_K_Q set /a switch_optional_counter+=1
	if defined ARG_K_C set /a switch_optional_counter+=1
	set /a switch_counter=switch_optional_counter
	if %SWITCH_COUNTER% neq %ARG_SWITCHES_COUNT% echo Unexpected named argument was given&& exit /b 1
endlocal
goto :eof

:copy_file ItemFullName Destination [Overwrite]
setlocal
	if not exist "%~1" exit /b 53
	if not exist "%~2"\* exit /b 76
	if "%~3"=="TRUE" (
		copy "%~1" "%~2" >NUL 2>&1 || exit /b 17
	) else (
		if exist "%~2\%~nx1" exit /b 58
		copy "%~1" "%~2" >NUL 2>&1 || exit /b 17
	)
endlocal
goto :eof

:move_file ItemFullName Destination [Overwrite]
setlocal
	if not exist "%~1" exit /b 53
	if not exist "%~2"\* exit /b 76
		if "%~3"=="TRUE" (
		move "%~1" "%~2" >NUL 2>&1 || exit /b 17
	) else (
		if exist "%~2\%~nx1" exit /b 58
		move "%~1" "%~2" >NUL 2>&1 || exit /b 17
	)
endlocal
goto :eof

:delete_file ItemFullName Destination
setlocal
	if not exist "%~1" exit /b 53
	if not exist "%~2"\* exit /b 76
	del "%~1" /f /q >NUL 2>&1 || exit /b 17
endlocal
goto :eof

:help
:usage
setlocal EnableDelayedExpansion
	for /f "tokens=1,* delims=." %%1 in (' findstr "^::\?\." "%~f0" ') do (
		if "%%2"=="" (
			echo(
		) else (
			set "line=%%2"
			echo(!LINE:FindAndCopyFromList.bat=%~nx0!
		)
	)
endlocal
goto :eof

:main
(echo(%*)>"%TEMP%\%~nx0_arg-parser.tmp.js"
call :arg_parser
if defined ARG_K_? call :help & exit /b 0
call :arg_parser__validate_args >&2 || exit /b 1

call :initialise_paramters

if "%ACTION%"=="INV" (
	>&2 echo Invalid value for action named argument was specified&& exit /b 448
)

if "%ACTION%"=="DELETE" if "%SUPPRESS%"=="FALSE" (
	set/p=Warning: the selected mode will permanently delete items. Are you sure you want to continue? <NUL
	choice
	if errorlevel 2 exit /b
	echo(
)

set destination="%DESTINATION:"=%"
set "destination=%DESTINATION:\"="%"
set "destination=%DESTINATION:"=%"
if "%ADAPT_CURRENT_DIRECTORY%"=="FALSE" (
	if not exist "%DESTINATION%"\* >&2 (
		if "%FORCE%"=="TRUE" (
			if exist "%DESTINATION%" (
				echo Destination path could not be created&& exit /b 75
			) else (
				md "%DESTINATION%"
			)
		) else (
			echo Destination path not found&& exit /b 76
		)
	)
)

set source="%SOURCE:"=%"
set "source=%SOURCE:\"="%"
set "source=%SOURCE:"=%"
if not exist "%SOURCE%"\* >&2 echo Source path not found&& exit /b 76

pushd %SOURCE%

set "tempfile=%TEMP%\tmp%RANDOM%_%~n0__#.txt"
if "%ACTION%"=="LIST" (
	set "tempfile_found=!TEMPFILE:_#=FOUND!"
	set "filestream_found=>>"!TEMPFILE_FOUND!""
	copy NUL "!TEMPFILE_FOUND!" >NUL
	set "tempfile_none=!TEMPFILE:_#=NONE!"
	set "filestream_none=>>"!TEMPFILE_NONE!""
	copy NUL "!TEMPFILE_NONE!" >NUL
	set "tempfile_foundmulti=!TEMPFILE:_#=MULTI!"
	set "filestream_foundmulti=>>"!TEMPFILE_FOUNDMULTI!""
	copy NUL "!TEMPFILE_FOUNDMULTI!" >NUL
) else (
	set "filestream_none=>&2"
	set "filestream_foundmulti=>&2"
)

if %TRANSVERSE_SUBFOLDERS%==TRUE (
	set "transverse=/s"
) else (
	set "transverse="
)

set /a items_copied=0
for /f "useback delims=" %%I in ("%FILELIST%") do (
	set /a item_times_found=0
	for /f "tokens=*" %%J in ('
		robocopy "%SOURCE%" . "%%~I" %TRANSVERSE% /is /it /l /ns /nc /ndl /np /njh /njs /r:0 /w:0
	') do (
		set /a item_times_found+=1
		set "item_fullname=%%~fJ"
		set "item_path=%%~dpJ"
	)

	if !ITEM_TIMES_FOUND! equ 0 (
		if "%ACTION%"=="LIST" (
			%FILESTREAM_NONE% echo(%%~I
		) else (
			%FILESTREAM_NONE% echo Item '%%~I' could not be located
		)
	) else if !ITEM_TIMES_FOUND! gtr 1 (
		if "%ACTION%"=="LIST" (
			for /f "tokens=*" %%J in ('
				robocopy "%SOURCE%" . "%%~I" %TRANSVERSE% /is /it /l /ns /nc /ndl /np /njh /njs /r:0 /w:0
			') do %FILESTREAM_FOUNDMULTI% (
				echo(%%~J
			)
			%FILESTREAM_FOUNDMULTI% echo(
		) else (
			%FILESTREAM_FOUNDMULTI% echo Mulitple instances of '%%~I' found:
			for /f "tokens=*" %%J in ('
				robocopy "%SOURCE%" . "%%~I" %TRANSVERSE% /is /it /l /ns /nc /ndl /np /njh /njs /r:0 /w:0
			') do %FILESTREAM_FOUNDMULTI% (
				echo     '%%~J'
			)
			%FILESTREAM_FOUNDMULTI% echo   No action taken
		)
	) else if !ITEM_TIMES_FOUND! equ 1 (
		if "%ADAPT_CURRENT_DIRECTORY%"=="TRUE" (
			pushd !ITEM_PATH!

			if "%FORCE%"=="TRUE" (
				if not exist "%DESTINATION%" md "%DESTINATION%"
			)
		)

		if "%ACTION%"=="COPY" (
			call :copy_file "!ITEM_FULLNAME!" "!DESTINATION!" "%FORCE%" && (
				set /a items_copied+=1
				echo Copied file '!ITEM_FULLNAME!'
			) || (
				if !ERRORLEVEL! equ 58 (
					>&2 echo File '%%~I' already exists in destination
				) else (
					>&2 echo Error copying file '%%~I'
				)
			)
		) else if "%ACTION%"=="MOVE" (
			call :move_file "!ITEM_FULLNAME!" "!DESTINATION!" "%FORCE%" && (
				set /a items_copied+=1
				echo Moved file '!ITEM_FULLNAME!'
			) || (
				if !ERRORLEVEL! equ 58 (
					>&2 echo File '%%~I' already exists in destination
				) else (
					>&2 echo Error moving file '%%~I'
				)
			)
		) else if "%ACTION%"=="DELETE" (
			call :delete_file "!ITEM_FULLNAME!" && (
				set /a items_copied+=1
				echo Deleted file '%%~I'
			) || (
				>&2 echo Error deleting file '%%~I'
			)
		) else if "%ACTION%"=="LIST" (
			%FILESTREAM_FOUND% echo(%%~I
		)

		if "%ADAPT_CURRENT_DIRECTORY%"=="TRUE" popd
	)
)

if "%ACTION%"=="LIST" (
	for %%J in ("%TEMPFILE_FOUND%") do if not "%%~zJ"=="0" (
		echo ITEMS FOUND:
		type "%TEMPFILE_FOUND%"
		del "%TEMPFILE_FOUND%"
	)
	for %%J in ("%TEMPFILE_NONE%") do if not "%%~zJ"=="0" (
		echo(& echo ITEMS NOT FOUND:
		type "%TEMPFILE_NONE%"
		del "%TEMPFILE_NONE%"
	)
	for %%J in ("%TEMPFILE_FOUNDMULTI%") do if not "%%~zJ"=="0" (
		echo(& echo ITEMS APPEARING IN MORE THAN ONE LOCATION:
		type "%TEMPFILE_FOUNDMULTI%"
		del "%TEMPFILE_FOUNDMULTI%"
	)
)

echo(
if "%ACTION%"=="COPY" (
	echo Total of %ITEMS_COPIED% items were copied to directory '!DESTINATION!'
) else if "%ACTION%"=="MOVE" (
	echo Total of %ITEMS_COPIED% items were moved to directory '!DESTINATION!'
) else if "%ACTION%"=="DELETE" (
	echo Total of %ITEMS_COPIED% items were deleted
) else if "%ACTION%"=="LIST" (
	echo Total of %ITEMS_COPIED% items were found
)

popd

UsernameIssues said:

I envisioned a batch file that resides* in each of the sub-folders. That batch file would enumerate** the contents of its folder and check for newer*** files in the "master list folder". If a newer file is found, copy that newer file to the sub-folder where the batch file is being run from. [goofybt can confirm that the newer file should overwrite the older file.]

Oh, right, I see. Quite easily done then. Robocopy can do this.

From what I gather, UNI, the reason your AutoIt program cannot be repeated to provide subsequent maintenance to the destination folder is because it would copy in new files from the source as well as update the destination files but only updating the destination files is required? If so, I’m contemplating Robocopy’s /XL switch.

goofybt said:

Filtering is a little complex but doesn't take just a min to create with Excel.

As for my destination folder I created a generic folder, once the scrip runs then i can manual move the files to the destination named folder.

But it sounds like a few other things may be automated for your purposes, Goofybt, d’you think the whole job could be done in one swing?

master folder list

Source.

UsernameIssues said:

AutoIt knows the folder that the script is running from. If the maintenance script is placed in destination folder 1 and run from there, the script can enumerate the files in that folder, and copy newer/changed files from the source folder. The same script can be placed in destination folder 2 and run from there. Nothing would have to be edited in the script to make it work in each destination folder.

Exactly. Why carry a list around if the list can just be inferred from the destination’s contents? Hence my suggestion of using Robocopy’s /XL option which will only update the files in the destination and will not copy in new files from the source.

UsernameIssues said:

Robocopy's /XL switch seems to be like xcopy's /D switch.
Does Robocopy have a switch like xcopy's /U?

The /MAXAGE switch is Robocopy’s equivalent for Xcopy’s /D switch, and Robocopy’s /XL is exactly like Xcopy’s /U.

Descriptions from SS64 on Xcopy and Robocopy:.

Code:

/U    Copy only files that already exist in destination.

Code:

/XL : eXclude "Lonely" files and dirs (present in source but not destination)
      This will prevent any new files being added to the destination.

UsernameIssues said:

Also, I have no idea how many files we are talking about in these destination folders. Robocopy's file enumeration performance might be an issue. See the second post in this thread.

Robocopy. Performance. Issue? I don’t think so. Robocopy is a specialised backup tool, not a file lister. For doing what it does best—copying—Robocopy’s speed is not a facet to be questioned.

Sure, if you pit Robocopy and Xcopy against each other in a fresh folder copy battle, Xcopy will always top Robocopy. But if you ever intend to perform subsequent backup routines—Robocopy’s favourite thing to do—there’d be a very clear.

To emphasise that very, I’ve to together a test. AutoIt3’s DirCopy vs Robocopy vs Xcopy vs Posh’s Copy-Item together in a benchmarking to see who can copy 5k+ files quickest.

Code:

$ErrorActionPreference = 'SilentlyContinue'

md 'A','B','C','D' | out-null

(measure-command {.\au3_copy.exe S A}).TotalMilliseconds
(measure-command {robocopy.exe S B *}).TotalMilliseconds
(measure-command {xcopy.exe S C     }).TotalMilliseconds
(measure-command {copy-item S\* D   }).TotalMilliseconds

Round 1:

Code:

20249.1022
19675.9401
15140.8496
17868.3324

Round 2:

Code:

33723.6522
127.4226
26029.4904
22490.9411

A single second to process 5k files is pretty darn quick, wouldn’t you think?