Sequential execution of a generator of processes

class pylada.process.iterator.IteratorProcess(functional, outdir, maxtrials=1, **kwargs)

Bases: pylada.process.process.Process

Executes an iteration function in child process.

An iterator process is a meta-process which runs other processes sequentially. It is one which needs iterating over, as shown more explicitely below. Its interface is fairly similar to other processes.

program = ProgramProcess(generator, outdir=dir, **kwargs)
program.start(comm)
try: program.wait()
except Fail:
  # do something

The main difference is in generator: it is kind of functional which will make more than one call to an external program. For instance, it could be several calls to VASP, a first call to relax the strain, followed by a static calculation. We want a process interface which hides those details (i.e. the sequence of calls to one or more external programs) from the owner of the process: the owner only wants to know that the calculation is still on-going, not which particular step it is at.

To do this, we use a generator, i.e. an object which can be used in a loop. It should yield one of two kinds of objects: (i) an extractor. such as pylada.vasp.extract.Extract, or an instance derived from Process. In the former case, the IteratorProcess simply keeps looping. In the latter case, the yielded process object is started and control is relinquished to the owner of the IteratorProcess instance. When the looping is done, the final extraction object that was ever yielded is returned.

This approach can be put into the following (pseudo) code:

for process in generator(**kwargs):
  if hasattr(program, 'success'): 
    result = process
    # go to next iteration
    continue
  else:
    # start process obtained in this iteration
    process.start(self._comm)
    # relinquish control to owner of self
# return the final extraction object
return result

A generator which implements two calls to an external program EXTERNAL would look something like this:

def generator(...):
  # PERFORM FIRST CALCULATION
  # check for success, e.g. pre-existing calculation
  if Extract(outdir0).success: 
    yield Extract(outdir0)
  # otherwise yield a program process wich calls EXTERNAL
  else:
    yield ProgramProcess(EXTERNAL)
    # check for success of this calculation.
    # if an error occured, handle it or throw an error
    if not Extract(outdir0).success: raise Exception()
      

  # PERFORM SECOND CALCULATION
  # check for success, e.g. pre-existing calculation
  if Extract(outdir1).success: 
    yield Extract(outdir1)
  # otherwise yield a program process wich calls EXTERNAL
  else: yield ProgramProcess(EXTERNAL)

  # yield the final extraction object. 
  # this is usefull for things other than process management.
  yield ExtractFinal(outdir)

To implement it, we need an extraction class, such as pylada.crystal.extract.Extract, capable of checking in a directory whether a successfull calculation already exists. If it does, the generator should yield the extraction object. If it doesn’t, it should yield some Process instance capable of starting the external program of interest.

The processes yielded by the input generator could be anything. It could be, for instance, another instance of IteratorProcess, or simply a bunch of ProgramProcess, or something more comples, as long as it presents the interface defined by Process.

Note

It would be possible, of course, to simply create a python program which does all the calls to the external programs itself. Then we wouldn’t have to deal with generators and loops explicitely. However, such an approach would start one more child python program than strictly necessarily, and hence would be a bit heavier. Nevertheless, since that approach is somewhat simpler, it has been implemented in CallProcess.

See also

CallProcess, JobFolderProcess, PoolProcess.

__init__(functional, outdir, maxtrials=1, **kwargs)

Initializes a process.

Parameters:

• functional – A generator which yields processes and/or extraction objects.
• outdir (str) – Path where the processes should be executed.
• maxtrials (int) – Maximum number of times to try re-launching each process upon failure.
• kwargs – Keyword arguments to the generator should be given here, as keyword arguments to :py:class:IteratorProcess`.

_cleanup()

Cleans up behind process instance.

This may mean closing standard input/output file, removing temporary files.. By default, calls cleanup of process, and sets process to None.

process

Holds currently running process.

This would be the latest process yielded by the input generator functional.

functional = None

Iterable to execute.

outdir = None

Execution directory of the folder.

params = None

Extra parameters to pass on to iterator.

poll()

Polls current job.

Returns:True if the process is finished. Raises NotStarted:  If the process was never launched.

start(comm)

Starts current job.

Parameters:

comm (Communicator) – Holds information about how to launch an mpi-aware process.

Returns:

True if process is already finished.

Raises:

• MPISizeError – if no communicator is not None and comm[‘n’] == 0. Assumes that running on 0 processors is an error in resource allocation.
• AlreadyStarted – When called for a second time. Each process should be unique: we do not want to run the VASP program twice in the same location, especially not simultaneously.

wait()

Waits for process to end, then cleanup.