Pylada accepts a range of configuration files:
- Files located in the ‘’config’’ sub-directory where Pylada is installed
- Files located in one of the directories specified by PYLADA_CONFIG_DIR
- In the user configuration file ‘’~/.pylada’‘
Each file is executed and whatever is declared within is placed directly at the root of the pylada package. The files are read in the order given above. Within a given directory, files are read alphabetically. Later files will override previous files, e.g. ‘’~/.pylada’’ will override any configuration done previously.
Within an IPython session, or during an actual calculation, the configuration variables all reside at the root of the pylada package.
IPython is Pylada’s main interface. In practice Pylada defines an extension to IPython. The extension can be automatically loaded when starting IPython by adding
c.InteractiveShellApp.extensions = [ "pylada.ipython" ]
to the ipython configuration file (generally, “$HOME/.config/ipython/profile_default/ipython_config.py”).
Another option is to load the extension dynamically for each ipython session:
>>> load_ext lada.ipython
It is unlikely that users will need to do anything here. However, it is possible to customize how external programs are launched by Pylada by redifining pylada.launch_program().
Pylada can run external MPI software, such as VASP. Such software must generally be started through a call to a specific MPI program. It is done in practice via the configuration variable mpirun_exe. It can be set as:
# openmpi and friends
mpirun_exe = "mpirun -n {n} {placement} {program}"
# Crays
mpirun_exe = "aprun -n {n} {placement} {program}"
mpirun_exe is a format string. It can take any number of arguments. However, two are required: “n”, which is the number of processes, and “program”, which is the commandline for the program to launch. The latter will be manufactured by Pylada internally. It is a placeholder at this point. The other reseverved keyword is “ppn”, the number of processes per node. It should only be used for that purpose. “placement” is useful when running MPI codes side-by-side. Please see below for extra setup steps required in that case.
The keywords in pylada.mpirun_exe should be defined in pylada.default_comm. This is a dictionary which holds default values for the different keywords. The dictionary may hold more keywords than are present in pylada.mpirun_exe. The converse is not true. It could be for instance:
default_comm = {'n': 2, 'ppn': 4, 'placement': ''}
It is often possible to run calculations side-by-side. One can request 64 processors from a supercomputer and run two VASP calculations simultaneously in the same PBS job. There are a fair number of steps to get this part of Pylada running:
Set up Pylada to run a single MPI programming as described above
Set the environment variable do_multiple_mpi_programs to True.
Set up figure_out_machines. This is a string which contains a small python script. Pylada runs this python script at the start of a PBS/Slurm job to figure out the hostnames of each machine allocated to the job. For each core, the script should print out a line starting with “PYLADA MACHINE HOSTNAME”. It will be launched as an MPI program on all available cores. By default, it is the following simple program:
from socket import gethostname from boost.mpi import gather, world hostname = gethostname() results = gather(world, hostname, 0) if world.rank == 0: for hostname in results: print "PYLADA MACHINE HOSTNAME:", hostname world.barrier()Note
This is one of two places where boost.mpi is used. By replacing this function with, say, call mpi4py methods, one could remove the boost.mpi in Pylada for most use cases.
It is important that this function prints out to the standard output one line per core (not one line per machine).
The script is launched by pylada.process.mpi.create_global_comm().
The names of the machines determined in the previous step are stored in default_comm‘s pylada.process.mpi.Communicator.machines attribute. This is simply a dictionary mapping the hostnames determined previously to the number of cores. It is possible, however, to modify default_comm after the figure_out_machines script is launched and the results parsed. This is done via the method modify_global_comm(). This method takes a pylada.process.mpi.Communicator instance on input and modifes it in-place. By default, this method does nothing.
On a cray, one could set it up as follows:
def modify_global_comm(comm): """ Modifies global communicator to work on cray. Replaces hostnames with the host number. """ for key, value in comm.machines.items(): del comm.machines[key] comm.machines[str(int(key[3:]))] = valueThis would replace the hostnames with something aprun can use for MPI placement. modify_global_comm() is runned once at the beginning of a Pylada PBS/Slurm script.
To test that the hostnames where determined correctly, one should copy the file “process/tests/globalcomm.py” somewhere, edit it, and launch it. The names of the machines should be printed out correctly, with the right number of cores:
> cd testhere > cp /path/to/pylada/source/process/tests/globalcomm.py > vi globalcomm.py # This is a PBS script. # Modify it so it can be launched. > qsub globalcomm.py # Then, when it finishes: > cat global_comm_out EXPECTED N=64 PPN=32 FOUND n 64 ppn 32 placement "" MACHINES PYLADA MACHINE HOSTNAME hector.006 32 PYLADA MACHINE HOSTNAME hector.006 32 ...The above is an example output. One should try and launch this routine on more than one node, with varying number of processes per node, and so forth.
At this point, Pylada knows the name of each machine participating in a PBS/Slurm job. It still needs to be told how to run an MPI job on a subset of these machines. This will depend on the actual MPI implementation installed on the machine. Please first read the manual for your machine’s MPI implementation.
Pylada takes care of MPI placements by formatting the mpirun_exe string adequately. For this reason, it is expected that mpirun_exe contains a “{placement}” tag which will be replaced with the correct value at runtime.
At runtime, before placing the call to an external MPI program, the method pylada.machine_dependent_call_modifier() is called. It takes three arguments: a dictionary with which to format the mpirun_exe string, a dictionary or pylada.process.mpi.Communicator instance containing information relating to MPI, a dictionary containing the environment variables in which to run the MPI program. The first and second dictionary will be merged and used to format the mpirun_exe string. By default, this method creates a nodefile with only those machines involved in the current job. It then sets “placement” to “-machinefile filename” where filename is the nodefile.
On Crays, one could use the following:
def machine_dependent_call_modifier( formatter=None, comm=None, env=None ): """ Placement modifications for aprun MPI processes. aprun expects the machines (not cores) to be given on the commandline as a list of "-Ln" with n the machine number. """ from pylada import default_comm if formatter is None: return if len(getattr(comm, 'machines', [])) == 0: placement = "" elif sum(comm.machines.itervalues()) == sum(default_comm.machines.itervalues()): placement = "" else: l = [m for m, v in comm.machines.iteritems() if v > 0] placement = "-L{0}".format(','.join(l)) formatter['placement'] = placementNote that the above requires the pylada.modify_global_comm() from point 4.
Warning
All external program calls are routed through this function, whether or not it is an MPI program. Hence it is necessary to check that the program is to be launched an MPI or not. In the case of serial programs, “comm” may be None.
The whole deal can be tested using “process/tests/placement.py” This is a PBS job which performs MPI placement on a fake job. It should be copied somewhere, edited, and launched.
At least two arguments should be set prior to running this script. Check the bottom of the script. “ppn” specifies the number of processors per nodes. The job should be launched with 2*”ppn” cores. “path” should point to the source directory of Pylada. This is so that a small program can be found (pifunc) and used for testing. The program can be compiled by Pylada by setting “compile_test True” in cmake.
“placement.py” will launch several simultaneous instances of the “pifunc” program: one long on three quarters of allocated cores, and two smaller calculations on one eigth of the cores each.
One should check the ouput to make sure that the programs are running side-by-side (not all piled up on the same node), that they are runnning simultaneously, and that they run successfully (e.g. mpirun does launch them).
A ressource manager, such as pbs or slurm, takes care of allocating supercomputing ressources for submitted jobs. Pylada interfaces with these via a few global data variables:
pylada.qsub defines the executable to submit jobs.
# openpbs
qsub_exe = "qsub"
# slurn
qsub_exe = "sbatch"
The scripts themselves are defined via the format string pylada.pbs_string:
>>> pbs_string = "#! /bin/bash/\n"\
... "#SBATCH --account={account}\n"\
... "#SBATCH --time={walltime}\n"\
... "#SBATCH -N={nnodes}\n"\
... "#SBATCH -e={err}\n"\
... "#SBATCH -o={out}\n"\
... "#SBATCH -J={name}\n"\
... "#SBATCH -D={directory}\n\n"\
... "python {scriptcommand}\n"
Again, there are few reserved keywords which Pylada will use to fill in the string.
- account: defines the relevant account to which the calculation is submitted.
- queue: defines the relevant queue to which the calculation is submitted.
- walltime: maximum time for which the calculation will run
- n: Number of processes to run the job on.
- ppn: Number of processes per node.
- nnodes: number of nodes the calculation will run. This is generally computed from “ppn” and “n”.
- err: Standard error file. Generated by Pylada.
- out: Standard output file. Generated by Pylada.
- name: Name of the job. Generated by Pylada.
- directory: Directory where the job is launched. Generated by Pylada.
- scriptcommand: Script to launch. Generated by Pylada.
Most of the keywords are automatically generated by Pylada. Is is for the user to provide a script where the requisite number of keywords make sense for any particular ressource manager.
Default keyword values should be stored in the dictionary pylada.default_pbs.
The different queues (accounts) accessible to the users can be listed in pylada.queues (pylada.accounts). These will be made available to the users via %launch. If “queue” is not relevant to a particular supercomputer, pylada.queues can be set to the empty tuple.
The debug/interactive queue can be made more easily accessible via pylada.debug_queue:
pylada.debug_queue = 'queue', 'debug'
The first item of the tuple is the keyword that should be set to access the relevant resource. The second is the relevant value. These will differ from supercomputer to supercomputer. In practice, the first is generally “queue” or “account”, and the second is something like “debug”.
It is also possible to define pbs_string as a callable which takes keyword arguments and return a string. For instance, PBSpro does not accept names longuer that fifteen characters. That’s just to high-tech for an expensive propietary software:
default_pbs = { 'walltime': "00:55:00", 'nnodes': 1, 'ppn': 32,
'account': 'eO5', 'header': "", 'footer': "" }
def pbs_string(**kwargs):
if 'name' in kwargs:
kwargs['name'] = kwargs['name'][:min(len(kwargs['name']), 15)]
return "#! /bin/bash --login\n" \
"#PBS -e \"{err}\"\n" \
"#PBS -o \"{out}\"\n" \
"#PBS -N {name}\n" \
"#PBS -l mppwidth={n}\n" \
"#PBS -l mppnppn={ppn}\n" \
"#PBS -l walltime={walltime}\n" \
"#PBS -A {account}\n" \
"#PBS -V \n\n" \
"export PYLADA_TMPDIR=/work/e05/e05/`whoami`/pylada_tmp\n" \
"if [ ! -e $PYLADA_TMPDIR ] ; then\n" \
" mkdir -p $PYLADA_TMPDIR\n" \
"fi\n" \
"cd {directory}\n" \
"{header}\n" \
"python {scriptcommand}\n" \
"{footer}\n".format(**kwargs)
There are only two variable specific to vasp calculations:
is_vasp_4 defines whether the installed vasp program is version 4.6 or 5.0 and higher. In practice, this determines which POSCAR format to use, and whether or not some input options are available.
vasp_program defines the vasp executable. In general, it will be a string with path to the executable. It can also be a callable which takes the functional as input:
def vasp_program(self): """ Figures out the vasp executable. It is expected that two vasp executable exist, a *normal* vasp, and a one compiled for non-collinear calculations. """ lsorbit = getattr(self, 'lsorbit', False) == True return "vasp-4.6-nc" if lsorbit else "vasp-4.6"
Note
Pylada should be Set up properly to run mpi calculations.
Warning
Please follow the links for their description. Questions regarding how to compile should be addressed to the relevant authorities.