Windows 7 Forums
Welcome to Windows 7 Forums. Our forum is dedicated to helping you find support and solutions for any problems regarding your Windows 7 PC be it Dell, HP, Acer, Asus or a custom build. We also provide an extensive Windows 7 tutorial section that covers a wide range of tips and tricks.



Windows 7: Batch file to run 10,000 instances of a program, in groups of 32

23 Jan 2013   #1

Windows 7 Ultimate x64
 
 
Batch file to run 10,000 instances of a program, in groups of 32

Dear all,

I have a computer with 32 (physical) cores and need to run a total of 10,000 statistical simulations. Each simulation is started with:

> program.exe "scriptName.txt"

where 'scriptName.txt' is 'script1.txt' , 'script2.txt', etc.
The program automatically quits after completion, but each program has a varying runtime.

Now my question: I have found information on how to start programs sequentially as well as simultaneously (using batch files), but does anyone know how to start 10,000 programs in groups of 32? The idea is to run 32 programs simultaneously, and whenever a program quits, a new one should be opened automatically in order for the 32 available cores to be efficiently used.

Thank you!

My System SpecsSystem Spec
.

23 Jan 2013   #2

Microsoft Community Contributor Award Recipient

Windows 7 Ult. x64 Windows 8.1 x64 Ubuntu 12.04 LTS Tri-Boot
 
 

Hi

What software are you using to run the realisations? Is it Windows based or Unix based?

I know how to accomplish this using a TCShell script but not .bat files unfortunately. If I provide the shell script, perhaps someone could convert to .bat format.

Generally simulations use RAM as opposed to CPU, since each realisation runs with the same simulation parameters and data, with only the random seed number varying - this depends largely on the software used. How much RAM do you have available?

Regards,
Golden
My System SpecsSystem Spec
23 Jan 2013   #3

Windows 7 Ultimate x64
 
 

Hi Golden,

Each of my runs takes several hours of computation time and requires <1 GB of RAM. My system has 128GB of RAM installed, so running 32 computations simultaneously is no problem.

The software is Open Source (it's called OpenBUGS) and can be installed in Linux or Windows, but I use the Windows version.

Do you have any suggestion on how to write the required batch file?
My System SpecsSystem Spec
.


23 Jan 2013   #4

Microsoft Community Contributor Award Recipient

Windows 7 Ult. x64 Windows 8.1 x64 Ubuntu 12.04 LTS Tri-Boot
 
 

Hi,

You could try this option, although it requires some work in setting up the simulation parameter files. Basically, you create 32 .bat files to run similatneously. Each .bat file runs 300 realisations (10,000 / 32 = 312.5), so we use 300. BAT file 1 to 31 loops through 300 realisations, BAT file 32 runs through the remaining 700 realisations.

This is what it looks like in TCSHELL format. You need to convert to the equivalent "DOS" format if using Windows BAT files.

BAT #1
Code:
 
set i = 1
while (${i} <= 300)
program.exe script${i}.txt
@ i = $i + 1
end
BAT#2
Code:
 
set i = 301
while (${i} <= 900)
program.exe script${i}.txt
@ i = $i + 1
end
BAT#3
Code:
 
set i = 901
while (${i} <= 1200)
program.exe script${i}.txt
@ i = $i + 1
end
.
.
.
.
.
n
.
.
.
.
.
.
BAT#32
Code:
 
set i = 9301
while (${i} <= 10000)
program.exe script${i}.txt
@ i = $i + 1
end
Good luck,
Golden
My System SpecsSystem Spec
24 Jan 2013   #5

Windows 7 Ultimate x64
 
 

Hi Golden,

Already thought about this, but it was precisely what I was trying to avoid.. It doesn't scale well.. Perhaps
someone knows a way to program this directly?

Best, Marcel
My System SpecsSystem Spec
24 Jan 2013   #6

Microsoft Community Contributor Award Recipient

Windows 7 Ult. x64 Windows 8.1 x64 Ubuntu 12.04 LTS Tri-Boot
 
 

Sorry Marcel, but you might be out of luck. I've tried looking for solutions to the same thing and gave up (180 realisations for 900 million nodes and 5 variables = 3 weeks computing time)

I've tried looking at many different solutions (for example Python scripting for SGeMS - http://sgems.sourceforge.net/) but its too buggy. Like you say, you need a programmatical solution, but that requires burrowing down into the code......very tricky and time consuming..............
My System SpecsSystem Spec
Reply

 Batch file to run 10,000 instances of a program, in groups of 32





Thread Tools




Our Sites

Site Links

About Us

Find Us

Windows 7 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 7" and related materials are trademarks of Microsoft Corp.

Designer Media Ltd

All times are GMT -5. The time now is 08:14 AM.
Twitter Facebook Google+



Windows 7 Forums

Seven Forums Android App Seven Forums IOS App
  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33