Ctrl+K

pympipool.scheduler.create_executor

pympipool.scheduler.create_executor#

pympipool.scheduler.create_executor(max_workers: int = 1, max_cores: int = 1, cores_per_worker: int = 1, threads_per_core: int = 1, gpus_per_worker: int = 0, oversubscribe: bool = False, cwd: str | None = None, executor=None, hostname_localhost: bool = False, backend: str = 'auto', block_allocation: bool = False, init_function: callable | None = None, command_line_argument_lst: list[str] = [])[source]#

Instead of returning a pympipool.Executor object this function returns either a pympipool.mpi.PyMPIExecutor, pympipool.slurm.PySlurmExecutor or pympipool.flux.PyFluxExecutor depending on which backend is available. The pympipool.flux.PyFluxExecutor is the preferred choice while the pympipool.mpi.PyMPIExecutor is primarily used for development and testing. The pympipool.flux.PyFluxExecutor requires flux-core from the flux-framework to be installed and in addition flux-sched to enable GPU scheduling. Finally, the pympipool.slurm.PySlurmExecutor requires the SLURM workload manager to be installed on the system.

Parameters:

  • max_workers (int) – for backwards compatibility with the standard library, max_workers also defines the number of cores which can be used in parallel - just like the max_cores parameter. Using max_cores is recommended, as computers have a limited number of compute cores.

  • max_cores (int) – defines the number cores which can be used in parallel

  • cores_per_worker (int) – number of MPI cores to be used for each function call

  • threads_per_core (int) – number of OpenMP threads to be used for each function call

  • gpus_per_worker (int) – number of GPUs per worker - defaults to 0

  • oversubscribe (bool) – adds the –oversubscribe command line flag (OpenMPI and SLURM only) - default False

  • cwd (str/None) – current working directory where the parallel python task is executed

  • hostname_localhost (boolean) – use localhost instead of the hostname to establish the zmq connection. In the context of an HPC cluster this essential to be able to communicate to an Executor running on a different compute node within the same allocation. And in principle any computer should be able to resolve that their own hostname points to the same address as localhost. Still MacOS >= 12 seems to disable this look up for security reasons. So on MacOS it is required to set this option to true

  • backend (str) – Switch between the different backends “flux”, “mpi” or “slurm”. Alternatively, when “auto” is selected (the default) the available backend is determined automatically.

  • block_allocation (boolean) – To accelerate the submission of a series of python functions with the same resource requirements, pympipool supports block allocation. In this case all resources have to be defined on the executor, rather than during the submission of the individual function.

  • init_function (None) – optional function to preset arguments for functions which are submitted later

  • command_line_argument_lst (list) – Additional command line arguments for the srun call (SLURM only)

Contents