Batch file to run 10,000 instances of a program, in groups of 32


  1. Posts : 3
    Windows 7 Ultimate x64
       #1

    Batch file to run 10,000 instances of a program, in groups of 32


    Dear all,

    I have a computer with 32 (physical) cores and need to run a total of 10,000 statistical simulations. Each simulation is started with:

    > program.exe "scriptName.txt"

    where 'scriptName.txt' is 'script1.txt' , 'script2.txt', etc.
    The program automatically quits after completion, but each program has a varying runtime.

    Now my question: I have found information on how to start programs sequentially as well as simultaneously (using batch files), but does anyone know how to start 10,000 programs in groups of 32? The idea is to run 32 programs simultaneously, and whenever a program quits, a new one should be opened automatically in order for the 32 available cores to be efficiently used.

    Thank you!
      My Computer


  2. Posts : 19,384
    Windows 10 Pro x64 ; Xubuntu x64
       #2

    Hi

    What software are you using to run the realisations? Is it Windows based or Unix based?

    I know how to accomplish this using a TCShell script but not .bat files unfortunately. If I provide the shell script, perhaps someone could convert to .bat format.

    Generally simulations use RAM as opposed to CPU, since each realisation runs with the same simulation parameters and data, with only the random seed number varying - this depends largely on the software used. How much RAM do you have available?

    Regards,
    Golden
      My Computer


  3. Posts : 3
    Windows 7 Ultimate x64
    Thread Starter
       #3

    Hi Golden,

    Each of my runs takes several hours of computation time and requires <1 GB of RAM. My system has 128GB of RAM installed, so running 32 computations simultaneously is no problem.

    The software is Open Source (it's called OpenBUGS) and can be installed in Linux or Windows, but I use the Windows version.

    Do you have any suggestion on how to write the required batch file?
      My Computer


  4. Posts : 19,384
    Windows 10 Pro x64 ; Xubuntu x64
       #4

    Hi,

    You could try this option, although it requires some work in setting up the simulation parameter files. Basically, you create 32 .bat files to run similatneously. Each .bat file runs 300 realisations (10,000 / 32 = 312.5), so we use 300. BAT file 1 to 31 loops through 300 realisations, BAT file 32 runs through the remaining 700 realisations.

    This is what it looks like in TCSHELL format. You need to convert to the equivalent "DOS" format if using Windows BAT files.

    BAT #1
    Code:
     
    set i = 1
    while (${i} <= 300)
    program.exe script${i}.txt
    @ i = $i + 1
    end
    BAT#2
    Code:
     
    set i = 301
    while (${i} <= 900)
    program.exe script${i}.txt
    @ i = $i + 1
    end
    BAT#3
    Code:
     
    set i = 901
    while (${i} <= 1200)
    program.exe script${i}.txt
    @ i = $i + 1
    end
    .
    .
    .
    .
    .
    n
    .
    .
    .
    .
    .
    .
    BAT#32
    Code:
     
    set i = 9301
    while (${i} <= 10000)
    program.exe script${i}.txt
    @ i = $i + 1
    end
    Good luck,
    Golden
      My Computer


  5. Posts : 3
    Windows 7 Ultimate x64
    Thread Starter
       #5

    Hi Golden,

    Already thought about this, but it was precisely what I was trying to avoid.. It doesn't scale well.. Perhaps
    someone knows a way to program this directly?

    Best, Marcel
      My Computer


  6. Posts : 19,384
    Windows 10 Pro x64 ; Xubuntu x64
       #6

    Sorry Marcel, but you might be out of luck. I've tried looking for solutions to the same thing and gave up (180 realisations for 900 million nodes and 5 variables = 3 weeks computing time)

    I've tried looking at many different solutions (for example Python scripting for SGeMS - http://sgems.sourceforge.net/) but its too buggy. Like you say, you need a programmatical solution, but that requires burrowing down into the code......very tricky and time consuming..............
      My Computer


 

  Related Discussions
Our Sites
Site Links
About Us
Windows 7 Forums is an independent web site and has not been authorized, sponsored, or otherwise approved by Microsoft Corporation. "Windows 7" and related materials are trademarks of Microsoft Corp.

Designer Media Ltd
All times are GMT -5. The time now is 06:18.
Find Us