Parallel execution of a job-folder

class pylada.process.jobfolder.JobFolderProcess(jobfolder, outdir, maxtrials=1, nbpools=1, keepalive=False, **kwargs)

Bases: pylada.process.process.Process

Executes folder in child processes.

Expects a jobfolder on input. Executable job-folders are launched in parallel, with up to nbpools running instances. Each instance is allocated an equal number of processors.

If a folder does not execute correctly, then the failure code or exception is stored in errors until all folders have been executed. Only then is a Fail exception raised.

Note

The executable folders are determined when the process is first created. To modify jobfolder, one should call update().

See also

PoolProcess

__init__(jobfolder, outdir, maxtrials=1, nbpools=1, keepalive=False, **kwargs)

Initializes a process.

Parameters:

• jobfolder (JobFolder) – Jobfolder for which executable folders should be launched. The name of the folders to launch are determined which __init__() is acalled. If jobfolder changes, then one should call update().
• outdir (str) – Path where the python child process should be executed.
• nbpools (int) – Maximum number of executable folders to run in parallel. The processors will splitted into n of approximately equal length, where n is nbpool or the remaining number of executable jobs, which ever is smallest.
• keepalive (bool) – Whether to relinquish communicator once jobs are completed. If True, the communicator is not relinquished. The jobfolder can be updated and new jobs started. To finally relinquish the communicator, keepalive should be set to False. Both kill() and terminate() ignore this attribute and relinquish the communicator. However, since both side effects, this may not be the best way to do so.
• maxtrials (int) – Maximum number of times to try re-launching each process upon failure.
• kwargs – Keyword arguments to the functionals in the executable folders. These arguments will be applied indiscriminately to all folders.

_cleanup()

Cleans up after currently running processes.

done

True if job already finished.

errors = None

Map between name of failed jobs and exception.

jobfolder = None

Jobfolder for which executable folders should be launched.

The name of the folders to launch are determined which __init__() is acalled. If jobfolder changes, then one should call update().

keepalive = None

Whether to relinquish communicator once jobs are completed.

If True, the communicator is not relinquished. The jobfolder can be updated and new jobs started. To finally relinquish the communicator, keepalive should be set to False. Both kill() and terminate() ignore this attribute and relinquish the communicator. However, since both side effects, this may not be the best way to do so.

kill()

Kills all currently running processes.

Relinquishes communicator, even if keepalive is True.

nbjobsleft

Number of jobs left.

nbpools = None

Number of executable folders to launch in parallel.

The processors will splitted into n of approximately equal length, where n is nbpool or the remaining number of executable jobs, which ever is smallest.

nbrunning_processes

Number of running processes.

For simple processes, this will be one or zero. For multitasking processes this may be something more.

outdir = None

Execution directory of the folder.

params = None

Keyword arguments to the functionals in the executable folders.

These arguments will be applied indiscriminately to all folders.

poll()

Polls current job.

process = None

List of currently running processes.

Each item consists of an index into the job-folder, an instance derived from Process, e.g. CallProcess, and a communicator used by that process.

start(comm)

Start executing job-folders.

terminate()

Kills all currently running processes.

Relinquishes communicator, even if keepalive is True.

update(jobfolder, deleteold=False)

Updates list of jobs.

Adds jobfolders which are not in self.jobfolder but in the input. Updates jobs which are in self.jobfolder and input if not currently running. Does nothing if are job is currently running. If deleteold is True, then removed finished jobs from job-folder.

wait(sleep=1)

Waits for all job-folders to execute and finish.