Ofinno research on silent cluster stalls has been accepted at IEEE Cluster 2026 and will be published in the IEEE Xplore proceedings.
We’re proud to share that research from Ofinno’s Advanced Media Lab has been accepted at IEEE Cluster 2026 (September 22–25, 2026, Alexandria, VA), one of the leading international conferences on cluster computing. The work, “Jobguard: Bridging the Scheduler-to-Node Observability Gap in Slurm HPC Clusters” by Sai Kiran Boppana, HPC Software Engineer at Ofinno, was accepted as a peer-reviewed poster and will be published in the IEEE Xplore conference proceedings.
The problem of jobs that lie
High-performance computing clusters run on trust. A workload scheduler like Slurm hands out compute nodes to jobs, then relies on a chain of status signals to know each job is alive. If a process crashes or a node dies, the scheduler notices and reacts.
But there is a blind spot the scheduler cannot see into. Some jobs are technically alive yet have silently stopped making progress. A shared filesystem hangs. A daemon restarts and loses track of its processes. An application deadlocks. In every one of these cases, the scheduler keeps reporting the job as RUNNING while it burns allocated compute hours producing nothing.
These silent cluster stalls are expensive precisely because nothing looks wrong. In one real incident on Ofinno’s production cluster, a filesystem server hang wedged 352 jobs across three users for five days, with the scheduler reporting healthy status the entire time.
Enter jobguard
jobguard is a lightweight stall-detection framework Sai designed and deployed on Ofinno’s HPC clusters. Its core idea is simple. Instead of asking the scheduler whether a job is progressing, jobguard asks the operating system.
It works in two layers.
On every compute node, a small collector wakes up once an hour, finds the processes belonging to each job, and reads ground-truth signals straight from the Linux kernel, including process state, accumulated CPU time, and what each process is waiting on. The whole pass takes about three seconds and uses under 0.001% of a node’s compute capacity, with no resident daemon, no changes to Slurm, and no changes to user applications.
On a central node, an auditor reconciles what the kernel actually observed against what the scheduler claims. When the two disagree, meaning the scheduler says RUNNING while the kernel says nothing has moved, jobguard classifies the stall into one of three types.
Processes stuck in uninterruptible sleep because storage or the network has wedged.
“Ghost” jobs: the scheduler still tracks a job whose processes no longer exist.
Processes that exist but have made zero CPU progress across multiple audit cycles.
- Infrastructure stalls (Type I) are processes stuck in uninterruptible sleep because storage or the network has wedged.
- Middleware stalls (Type II) are “ghost” jobs, where the scheduler still tracks a job whose processes no longer exist.
- Application stalls (Type III) are processes that exist but have made zero CPU progress across multiple audit cycles.
A set of cross-cycle rules (multi-cycle confirmation thresholds, minimum-runtime filters, classification stickiness) keeps the system from crying wolf over jobs that are merely starting up or waiting on bursty I/O. Alerts are grouped per incident and dispatched at most once per user, and only when a stall persists beyond 24 hours.
Does it work? The numbers say yes.
The research was validated with a rigorous 78-trial fault-injection campaign spanning 16 fault classes on a dedicated testbed, including a deliberately designed “blind spot” class to map the boundary of what the classifier can and cannot see. The results were clear.
- 21 of 21 valid injected faults detected (100% detection)
- 0 false positives across 31 tricky control workloads designed to look like stalls (bursty I/O, slow startups, heartbeat patterns)
- 21 of 21 blind-spot workloads correctly bounded, confirming the classifier behaves exactly as designed, even at its limits
Production deployment on Ofinno’s 72-node cluster backed up the lab results with real saves. It caught the five-day filesystem wedge described above hours before any user noticed. It flagged roughly 530 ghost tasks from a controller restart within one cycle, correctly separating truly stuck work from work that recovered on its own. And it watched a 256-job contention event resolve itself, correctly sending zero alert emails.
Why it matters beyond Ofinno
jobguard’s design assumes almost nothing about the cluster it runs on. It needs only a Linux node and a scheduler that can map jobs to processes. That portability, plus the near-zero overhead, is what makes the approach interesting to the broader HPC community, and what earned it a place at IEEE Cluster through peer review.
For Ofinno, the work reflects something we care about deeply. Our research infrastructure is a first-class engineering problem. The same clusters this work protects power the wireless and media technology research at the heart of what we do. Reclaiming silently wasted compute hours means faster iteration for every researcher in the lab.
See it at IEEE Cluster 2026
Sai will present the jobguard poster at IEEE Cluster 2026 in Alexandria, Virginia, September 22–25, 2026. The extended abstract will appear in the IEEE Xplore conference proceedings following the conference. If you’re attending, stop by. He’d be glad to talk silent stalls, fault taxonomies, and what your scheduler isn’t telling you.