Parallel execution of a job-folder with per-folder numbers of processors

class pylada.process.pool.PoolProcess(jobfolder, outdir, processalloc, maxtrials=1, keepalive=False, **kwargs)

Bases: pylada.process.jobfolder.JobFolderProcess

Executes folder in child processes.

Much as its base class, JobFolderProcess, this process specialization is intended to run jobs in a jobfolder in parallel [*]. However, it allows to customize the number of processors dedicated to each job, rather than use the same number of processors for each job.

The customization is done via the function processalloc. It takes one argument, the executable jobfolder, and returns an integer signifying the requested number of processors.

def processalloc(folder):
  return (len(folder.structure) // 2) * 2

process = PoolProcess(jobfolder, outdir='here', processalloc=processalloc)
process.start(comm)

try: process.wait()
except Fail: pass

The interface is much the same as any other process. However, it takes as argument this processalloc function, on top of the jobfolder itself. In this case, each folder will be launched with approximately as many processors as there are atoms in the structure [†].

Once it is launched, the PoolProcess instance will attempt to run as many jobs as possible in parallel, until there it runs out of processors to allocate. Howe many processors, and which machines, is determined by the communicator passed to start(). Each time an executable folder is finished [‡], it tries again to pack jobs into the available processor pool.

Note

Upon failure, Fail is raised only once all the folders have been executed, not when the failure is detected.

[*]	Several job-folders are executed simultaneously, not withstanding the possibility that each of these is also executed in parallel via MPI.

[†]	Apparently, this is a pretty good rule-of-thumb for VASP calculations.

[‡]	More, specifically, each time poll() is called.

__init__(jobfolder, outdir, processalloc, maxtrials=1, keepalive=False, **kwargs)

Initializes a process.

Parameters:

• jobfolder (JobFolder) – Jobfolder for which executable folders should be launched. The name of the folders to launch are determined which __init__() is acalled. If jobfolder changes, then one should call update().
• outdir (str) – Path where the python child process should be executed.
• processalloc ((JobFolder)->int) – Function which determines how many processors each job requires. This is determined for each job when this instance is created. To change jobfolder, one should call update().
• keepalive (bool) – Whether to relinquish communicator once jobs are completed. If True, the communicator is not relinquished. The jobfolder can be updated and new jobs started. To finally relinquish the communicator, keepalive should be set to False. Both kill() and terminate() ignore this attribute and relinquish the communicator. However, since both side effects, this may not be the best way to do so.
• maxtrials (int) – Maximum number of times to try re-launching each process upon failure.
• kwargs – Keyword arguments to the functionals in the executable folders. These arguments will be applied indiscriminately to all folders.

processalloc = None

Determines number of processors to allocate to each job.

This is a function which takes a JobFolder instance and returns an integer.

start(comm)

Start executing job-folders.

update(jobfolder, deleteold=False)

Updates list of jobs.

Adds jobfolders which are not in self.jobfolder but in the input. Deletes those which in self.jobfolder but not in the input. Does nothing if job is currently running. Finished jobs are not updated. If deleteold is True, then removed finished jobs from job-folder.

Processes jobfolder from root, even if passed a child folder.