Match MPI, OpenMP, and GPU execution to the application

Audience: Cluster users · Reviewed: 2026-10-04

Version and teaching scope

Use documentation matching your installed release and local policy. Check sinfo --version; upstream documentation identified itself as 26.05 at review. Paths, partitions, accounts, modules, memory, MPI, GPUs and services are site-specific. Example programs beginning ./ must already exist. These are teaching examples, not a validated deployment recipe or evidence that a cluster was modified.

The scheduler reserves resources; the executable must know how to use them. These templates require the actual program and a compatible software environment. They deliberately omit a partition/account because those are local choices.

OpenMP: one process, eight threads on one node

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
#SBATCH --time=00:30:00
set -euo pipefail
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"
export OMP_PLACES=cores
export OMP_PROC_BIND=close
srun --cpu-bind=cores ./threaded_solver input.dat

The application must be built for OpenMP. Its own thread controls and threaded BLAS libraries may need separate settings. Binding depends on CPU allocation and runtime; test close versus spread rather than assuming one is fastest. Eight requested CPUs are not automatically eight physical cores on every site.

MPI: eight ranks distributed across two nodes

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --mem=8G
#SBATCH --time=00:30:00
set -euo pipefail
export OMP_NUM_THREADS=1
# Load the site's matching compiler/MPI/application environment first.
mpirun ./mpi_solver input.dat

This launcher example targets Open MPI integrated with Slurm. Current Open MPI guidance recommends mpirun inside the allocation. An alternative is srun --mpi=pmix ./mpi_solver input.dat only when the site's Slurm PMIx plugin and application MPI stack are compatible. Check srun --mpi=list; a listed plugin alone does not prove full application compatibility. Do not nest srun mpirun by default or mix MPI implementations at build and run time. Open MPI 5 and later do not support PMI-2 direct launch.

GPU: one GPU and four host CPUs

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:1
#SBATCH --mem=16G
#SBATCH --time=00:20:00
set -euo pipefail
# NVIDIA-specific diagnostic, only on an appropriate GPU node:
srun nvidia-smi
srun ./gpu_solver input.dat

GPU GRES types, partitions, binding, and drivers are site-specific. --mem is host RAM, not GPU VRAM. Respect Slurm's device visibility; do not overwrite it to reach unallocated GPUs. GPU visibility or an nvidia-smi listing does not prove the application performed useful GPU work. Validate numerical results and measure representative runs.

Package-specific launch requirements

For quantum chemistry software, follow the installed version's launch requirements: some programs launch their own workers, some depend on a particular MPI stack, and some support only single-node threading. These generic templates are not official launch scripts for a commercial package.

Official sources

Previous: Submit, inspect, and stop a job · Learning path · Next: Reproducible software and scratch workflows