Monitor and maintain without surprising users
Audience: Authorized administrators · Reviewed: 2026-10-04
Version and teaching scope
Use documentation matching your installed release and local policy. Check sinfo --version; upstream documentation identified itself as 26.05 at review. Paths, partitions, accounts, modules, memory, MPI, GPUs and services are site-specific. Example programs beginning ./ must already exist. These are teaching examples, not a validated deployment recipe or evidence that a cluster was modified. Administrative operations require authorized operators and a reviewed change plan; ordinary users must not execute them or paste them into production.
Monitor the whole service
Monitor several layers: controller responsiveness, queue trends, node health, accounting/database health, storage capacity and latency, network errors, and application failure patterns. “All nodes are busy” is not sufficient evidence of a healthy or efficient cluster. Alert on sustained deviations and actionable failures, not every transient pending job.
Read-only starting points:
Use service logs and hardware telemetry appropriate to the installation. Collect timestamps, affected job IDs, and node names before making changes. Avoid exposing other users' paths, inputs, tokens, or project details in public incident reports.
Administrator-only planned maintenance
# Authorized operator, approved maintenance window; example node name only.
scontrol update NodeName=compute01 State=DRAIN Reason="planned maintenance"
# Verify running work finishes and the node is safe to service.
squeue -w compute01
# Perform the separately approved repair and validation.
# Only after the fault is resolved and readiness is verified:
scontrol update NodeName=compute01 State=RESUME
Draining prevents new allocations while existing work can finish. It is not the same as killing running jobs, and an empty queue alone does not prove the node is safe to power off. Check system activity and site procedures. Never resume a faulty node merely to make a dashboard green; validate health, mounts, identity, and a small test job first.
Recovery before upgrade
Back up non-secret configuration with version history and secrets through a protected process. Include controller state and consistent database backups in recovery planning; test restoration in isolation. High availability is not a backup. Before upgrades, check the exact supported version path and daemon order in the upgrade guide. Database schema changes can make a package-only downgrade unsafe. Retain a tested rollback plan and communicate user-visible changes.
Official sources
Previous: Reproducible software and scratch workflows · Learning path · Next: Diagnose, measure, and automate scientific workflows