SlurmProcess

A wrapper around SLURM with the same interface as asyncio.subprocess

import asyncmd

Define a simple sbatch script

The script just prints whatever input it gets via stdin.

NOTE: You might need to change the partition and memory to adopt it to the cluster you are using.

slurm_script_content = """#!/bin/bash -l
# Standard output and error will be determined from `jobname` variable passed to SlurmProcess
# Initial working directory will be determined by `workdir` variable passed to SlurmProcess
# Queue (Partition):
#SBATCH --partition=s.bio  ## ADOPT TO YOUR CLUSTER!
#SBATCH --mem=4750
#
# Number of nodes and MPI tasks per node:
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
# Wall clock limit:
#SBATCH --time=24:00:00

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export MPI_NUM_RANKS=$SLURM_NTASKS_PER_NODE
export OMP_PLACES=cores

# sleep for 10 seconds such that we can actually pipe some input to stdin before the job finishes
sleep 10

# print whatever we got via stdin
printf "$(</dev/stdin)"

# and a newline to seperate
printf "\n"
printf "Done!"
"""

# write the script out (the SlurmProcess expects a file as input script)
slurm_script_name = "test_submission.slurm"
with open(slurm_script_name, "w") as sfile:
    sfile.write(slurm_script_content)

Submit a job via slurm and communicate with it via stdin

# this file will be used to send data to the process,
# it is optional but if you want to send data into the process via communicate it must be given at submission time
# the reason being that slurm connects the stdin of the batch job to this file, i.e. it must exist when the job starts
stdin_file = "test_submission_stdin_file"
# create a slurm process
proc = asyncmd.slurm.SlurmProcess(jobname="test", sbatch_script=slurm_script_name, workdir=".", stdfiles_removal="success")
# and directly submit it
await proc.submit(stdin=stdin_file)

# we could also use the command below to create and submit the SlurmProcess in one go
# this is very similar to `asyncio.create_subprocess_exec` and the recommended way if you want to submit directly
#proc = await asyncmd.slurm.create_slurmprocess_submit(jobname="test", sbatch_script=slurm_script_name, workdir=".", stdfiles_removal="success", stdin=stdin_file)

stdout, stderr = await proc.communicate(b"Writing to proc stdin\nThis could e.g. used to pipe the index-groups into a trajconv execution\n...Or anything else that you want to give as input to a running SlurmJob")

Lets have a look at stdout and stderr

print(stdout.decode())

Writing to proc stdin
This could e.g. used to pipe the index-groups into a trajconv execution
...Or anything else that you want to give as input to a running SlurmJob
Done!

print(stderr.decode())

# and the exit code of the slurm job (should be 0 if everything above worked)
await proc.wait()

0