Execution of an external program

class pylada.process.program.ProgramProcess(program, outdir, cmdline=None, stdout=None, stderr=None, stdin=None, maxtrials=1, dompi=False, cmdlmodifier=None, onfinish=None, onfail=None, **kwargs)

Bases: pylada.process.process.Process

Executes an external program

This process creates and manages the execution of an external program, say VASP or CRYSTAL, via a subprocess.Popen instance. The external program can be launched with or without MPI, with or without standard output/input/error files. It is always launched within a specified directory.

A typical use case, taken from process/test/program.py, is the following:

program = ProgramProcess( executable, outdir=dir, 
                          cmdline=['--sleep', 0, '--order', 4], 
                          stdout=stdout, dompi=True )
program.start(comm)
try: program.wait()
except Fail:
  # do something

The above launches an external program taking a set of arguments. Its output is piped to a specific file for later grepping. It is launched using the super-computer’s MPI interface, with the number of processors specified by comm. It is launched in a directory dir. The snippet above has python wait for the external program to finish, while checking for exceptions if the program fails. The external program is started only once start() is called.

Note

A Fail exception is thrown when the program returns with a non-zero exit code. However, some propietary MPI crapware, such as Cray’s. will return 0 whenever MPI::Finalize() is called, even when the program itself returns non-zero. As a result, it is not possible to rely on a Fail exception being thrown correctly on all machines at all times.

__init__(program, outdir, cmdline=None, stdout=None, stderr=None, stdin=None, maxtrials=1, dompi=False, cmdlmodifier=None, onfinish=None, onfail=None, **kwargs)

Initializes a process.

Parameters:

• program (str) – Path to the executable of interest.
• outdir (str) – Path to the directory where the program should be executed.
• cmdline (list) – List of commandline arguments. The elements of the list should be translatable to strings in a meaningfull way, via str.
• stdout (str) – Path to a file where the standard output of the executable should go. Ignored if None, in which the standard output is likely piped to sys.stdout.
• stderr (str) – Path to a file where the standard error of the executable should go. Ignored if None, in which the standard error is likely piped to sys.stderr.
• stdin (str) – Path to a file where the standard input of the executable should go. Ignored if None, in which the standard input is likely piped to sys.stdin. Note that there is no way to communicate with the process managed by this instance via the standard input. Indeed, subprocess.Popen.communicate locks the process until it finishes, which defeat the whole purpose of the process module.
• maxtrials (int) – Maximum number of restarts. If the external program fails, it will automically be restarted up to this number of times.
• dompi (bool) – If True, then the external program is launched with MPI. The MPI infrastructure should be set up correctly, meaning pylada.launch_program(), pylada.machine_dependent_call_modifier(), pylada.modify_global_comm(), pylada.mpirun_exe, default_comm, figure_out_machines.
• cmdlmodifier (cmdlmodifier(formatter, comm)->comm) – This function is called prior to launching the program. It can be used to modify the formatting dictionary. It should return a communicator to use in the actual call. This communicator can be the same as passed to the function. However, if the communicator needs to be modified, then a new one should be created (and the original left untouched). Ownership should not be retaine beyond the call to this function.
• onfinish (onfinish(self, exitcode!=0)->None) – Called once the process is finished. The first argument is this instance, the second is True if an error occurred.
• kwargs – Other keyword arguments are ignored.

_cleanup()

Cleanup files and crap.

cmdline = None

Command line for the program.

cmdlmodifier = None

A function to modify command-line parameters.

This function is only invoked for mpirun programs. It can be used to, say, make sure a program is launched only with an even number of processes. It should add ‘placement’ to the dictionary.

dompi = None

Whether to run with mpi or not.

onfail = None

Called if program fails.

Some program, such as CRYSTAL, return error codes when unconverged. However, does not necessarily mean the program failed to run. This function is called when a failure occurs, to make sure it is real or not. It should raise Fail if an error has occurred and return normally otherwise.

onfinish = None

Callback when the processes finishes.

Called even on error. Should take two arguments:

• process: holds this instance
• error: True if an error occured.

It is called before the _cleanup() method. In other words, the process is passed as it is when the error is found.

outdir = None

Directory where to run job.

poll()

Polls current job.

Returns:True if external program is finished. Raises Fail:If external program returns a non-zero exit code.

program = None

External program to execute.

start(comm=None)

Starts current job.

Parameters:

comm (Communicator) – Holds information about how to launch an mpi-aware process.

Returns:

True if process is already finished.

Raises:

• MPISizeError – if no communicator is not None and comm[‘n’] == 0. Assumes that running on 0 processors is an error in resource allocation.
• AlreadyStarted – When called for a second time. Each process should be unique: we do not want to run the VASP program twice in the same location, especially not simultaneously.

stderr = None

Name of standard error file, if any.

stdin = None

Name of standard error file, if any.

stdout = None

Name of standard output file, if any.

wait()

Waits for process to end, then cleanup.