BrokenML: can agents fix broken ML jobs?

Write-up: https://zhebrak.io/posts/brokenml/

Repo: https://github.com/zhebrak/brokenml

I built a benchmark to determine the access level agents need to successfully fix broken ML workloads (PyTorch over Gloo on a two-node Slurm cluster or a llama.cpp server running in Docker Compose). A separate verifier re-runs the job on a fresh cluster to score each solution. The faults were designed to reflect what you see in practice, including shape mismatches across ranks or hanging collectives, a straggler node, OOM errors, broken model checkpoints, scheduling issues, poor serving throughput, and silent loss divergence.

Nine models ran every task at three access levels: config and code only, plus live read-only logs, or root on the cluster. The gap between open and closed models is mostly in what an agent can do from the config files and code alone, without full cluster access. Main caveats are CPU over Gloo (not GPUs) and only one run per task per model.