Diagnose, measure, and automate scientific workflows

Audience: Cluster users · Reviewed: 2026-10-04

Version and teaching scope

Use documentation matching your installed release and local policy. Check sinfo --version; upstream documentation identified itself as 26.05 at review. Paths, partitions, accounts, modules, memory, MPI, GPUs and services are site-specific. Example programs beginning ./ must already exist. These are teaching examples, not a validated deployment recipe or evidence that a cluster was modified.

Diagnose before resubmitting

Diagnose the smallest observable failure before resubmitting a large calculation:

PENDING: inspect the reason. Priority and Resources are often ordinary waiting; a QOS/account limit or dependency requires a different response. A displayed reason may be only one currently blocking condition.
OUT_OF_MEMORY: inspect step-level memory measurements and the application's memory model. Reduce ranks, improve decomposition, or request justified headroom; do not assume MaxRSS is total memory summed over every rank.
TIMEOUT: distinguish slow progress from a hang; estimate an appropriate limit and use application checkpoints.
FAILED/NODE_FAIL: inspect stderr, exit code, scientific output, and node-related messages. A Slurm COMPLETED state does not validate scientific convergence or the physical model.

scontrol show job 12345
sacct -j 12345 --format=JobID,State,ExitCode,Elapsed,AllocCPUS,MaxRSS
# For a known running step; replace both job and step IDs:
sstat -j 12345.0 --format=JobID,AveCPU,MaxRSS

Accounting can be delayed or incomplete, and metric availability depends on configured collectors. Keep job and step rows distinct. Send support the ID, timestamp, minimal reproducer, loaded modules, and exact error, with secrets removed.

Independent inputs: a job array

For independent molecular inputs, an array is usually clearer than hundreds of manual submissions. Prepare inputs.txt with one safe relative filename per line and a pre-existing results directory. This example has 20 inputs and limits simultaneous elements to four:

#!/bin/bash
#SBATCH --array=0-19%4
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=00:20:00
#SBATCH --output=array-%A_%a.out
set -euo pipefail
input=$(sed -n "$((SLURM_ARRAY_TASK_ID + 1))p" inputs.txt)
[[ -n "$input" && -f "$input" ]] || { echo 'Invalid input' >&2; exit 2; }
srun ./serial_solver "$input" > "results/result-${SLURM_ARRAY_TASK_ID}.txt"

Array elements must not overwrite shared scratch or output. Each element receives its own resource request, and task completion order is not guaranteed. Adjust array bounds to the manifest; do not derive the list from a directory that changes mid-run.

Depend on a validated calculation stage

To analyze only after a successful calculation stage, run this submission sequence in Bash on the submit host:

set -euo pipefail
jid=$(sbatch --parsable array.sbatch)
jid=${jid%%;*}  # Remove an optional cluster suffix.
sbatch --dependency="afterok:${jid}" analyze.sbatch

analyze.sbatch must already exist. afterok checks scheduler success, not scientific validity. The calculation should return a nonzero exit status when validation fails. Inspect failed dependencies instead of repeatedly resubmitting downstream work.

Measure scientific throughput

For performance, keep molecule, method, basis, convergence threshold, and software stack fixed. Measure elapsed time and allocated CPU/GPU cost at several sizes. Doubling resources for a 10% time saving may reduce total project throughput. Separate compute, communication, memory, and I/O bottlenecks; changing hardware cannot fix an incorrect scientific model.

Official sources

Previous: Monitor and maintain without surprising users · Learning path