Toward Human-in-the-Loop Oversight for Long-Horizon Agent Projects

cogito233/swarm-oversight-harness

Contents: What the human sees · Why not watch the agents · The middle layer · Making acceptance mean something · Related work and limits · Try it

Multi-agent harnesses are maturing. Runtimes now keep many agents busy in parallel [1], and agent teams share persistent memory and build on each other’s work [5, 7]. As these systems improve, the scarce resource shifts from compute to the attention of the person who oversees them. I ran into this about four months ago in my own research projects and have been working around it since. I assumed this was common knowledge, but after this week’s announcements I am less sure, so I am writing down what I do.

Running ten agents at once is easy. Overseeing them on one project for three months is not. When I did this on research projects, the bottleneck after a few weeks was knowing what was true in my own project, more than compute or model quality.

The claim of this post is that the cost of overseeing a project depends on how many breaks in continuity the human has to reconcile, not on how many agents are running. The design goal follows from it: keep the project’s history continuous, and keep it in the place that is cheapest for a human to read. The analogy that helped me is the move from procedural to object-oriented programming. An agent session is a procedure call. It has a stack (its context window), does some work, returns a value, and its local state is gone. Most agent setups are procedural in this sense, because the state of the work lives in whichever agent is running. The middle layer makes the task the object instead: it has an identity, state that persists between calls, invariants, and an interface. Agents become methods that act on it, one after another or in parallel, and none of them owns it.

The rest of the post describes that layer in practice: what I look at, what I decide, and what the layer does for me. I keep three things in my head: the charter, the budget and the list of threads. I act at two points: approving what a task is for, and accepting what it produced. Everything else lives in task files in the repo. A planner agent maintains those files, and worker agents attach to a task, do their part, and exit. I am releasing the method as Swarm Oversight Harness.

1. What the human sees

My model of a project fits on one screen and has three parts:

1. Charter: what we are trying to find out, and the hypothesis whose outcome would change our plans.
2. Budget: how much compute and money each thread may spend before it has to come back and ask.
3. Threads: what each line of work is doing, what stage it has verifiably reached, and what it is waiting for.

An illustrative example (not a real project):

CHARTER  Can a 7B model learn to check its own patches with repo tests?
         H1: training on fail→pass test pairs adds ≥10 points of
             self-check accuracy on held-out repos.
         Kill if the 200-task pilot gains < 3 points.
BUDGET   2,000 GPU-h + $1,500 API in total. Over a thread's cap → ask.

STATE    critical path: T2 pilot result
  T1 build    fail→pass pipeline    verified, CI #41    40/100 GPU-h   → accept gate
  T2 run      SFT pilot, 200 tasks  launched, no agent  310/400 GPU-h  → done hook
  T3 review   red-team T1 leakage   running             $90/$200
  T4 read     related-work map      draft               $15/$50

I make two kinds of decision from this screen:

• Approve a task that the planner has drafted: what it is for, what counts as done, and how much it may spend.
• Accept a result: once accepted, the result is no longer a draft, and other work can build on it.

Everything below this screen is written for agents. I open a task’s files only when I need them for one of these two decisions. The budget is part of what a task is authorized to do, so an agent that would go over its thread’s cap stops and asks instead of retrying.

2. Why not watch the agents directly

Attention does not scale. DeepSeek’s Super-agent design makes the same point: today’s agents cap single-task performance at model capability and multi-task performance at human attention [1]. In my runs the ceiling was three to five sessions.

Projects lose continuity. How hard a project is to understand depends less on its size than on its continuity: one name for each thing, one version of each fact, and a record that reads in order. Every break makes the reader notice a discrepancy, decide which side is right, and remember the decision. Months of procedural agent work, where each session carries the state and then disappears, produce such breaks without ever raising an error. These are real incidents from my projects, lightly anonymized:

Each break was small, but together they meant I had to re-read the project every morning. Watching more agents would only have added more breaks, because each new session is one more place where state can live and then disappear.

3. The middle layer

Agents are procedures; tasks are objects

A single agent is easy to follow because it has one linear history that you can read in order. That history is lost when the call returns, that is, when the context window ends. A hand-built pipeline is worse, because its history is spread across stages and no single file says what happened in order.

The middle layer keeps a linear history but stores it in the task instead of the agent. The programming analogy maps directly:

The analogy has one important limit. An ordinary object overwrites its fields, but a task never rewrites its LOG, so it is closer to an object whose state is rebuilt from an event log. That is what keeps the history readable in order. Concretely, a task is a small, durable object in the repo:

• TASK.md: goal, files it may and must not touch, the authorization and budget it runs under, acceptance criteria with how each is checked, stop conditions;
• a LOG that is only ever appended to, with a snapshot of the current state at the top;
• RESULT files with evidence;
• one row in STATE, which is the screen in section 1.

One habit follows: an agent that launches a long experiment exits instead of polling. Before it exits, it appends a handoff to the task’s LOG: what was launched, where the outputs will appear, and how to check them. A completion hook or marker file signals the end of the run, and a new agent picks up the task. Polling would spend context on waiting, and the only record of what is pending would sit in a context that will be compacted.

The planner

A planner agent maintains the layer. Each round it re-reads the repo and fixes STATE wherever it disagrees with reality. It then names one critical path, splits work into tasks that touch different files, dispatches workers, has their results verified, and consolidates everything into one STATE update. I never merge N results by hand; if I find myself doing it, the layer is broken. One rule decides whether the planner keeps going or comes back to me: retry HOW problems, stop on WHAT problems. A failing test is a HOW problem and is retried within budget. A plan that contradicts reality, or a budget about to be exceeded, is a WHAT problem and is escalated to me.

Handing a task to an agent through files

The default way to delegate is to write the task into a subagent’s prompt and read back its summary. Over weeks, this breaks down. The task exists only in the parent agent’s context. The result is only the worker’s account. The parent has to merge N summaries. Failed attempts leave no record. In the middle layer the prompt is only a pointer, “do threads/T2/tasks/TASK-03.md”, and the result comes back as a file.

Which memory to trust and which to clean up

Moving memory into files only helps if the files stay trustworthy; otherwise the breaks in section 2 move into the repo. Not everything in the repo is equally valid, so each piece of memory is assigned to one of four kinds:

The planner does the cleanup at the start of each round. It compares STATE with git, tests and outputs. Every current doc records when it was last checked and which files it covers. If any of those files has changed since, the doc is marked stale until someone re-checks it. Superseded plans are not deleted; they move to history with a pointer to what replaced them. Anything that exists only in an agent’s context is treated as gone, because the agent will exit.

Three rules keep memory readable. Each thing has one name, based on its content (“the 4k-context ablation” rather than “arm C”). Each headline number is stored once and linked from elsewhere. Nothing is deleted to make the record look cleaner.

4. Making acceptance mean something

Acceptance is the one place my judgment enters the record, so it has to mean something. In research the output is a belief, and unlike a bad merge, a wrong belief raises no error. Agents that grade their own progress systematically accept non-improvements. Park and Choi show that a separate judge is not enough on its own: a strong judge that read the diff and the worker’s report still accepted cycles, and 44% of the cycles it accepted were real regressions [2]. What helped was externally grounded verification. In the middle layer this becomes three rules:

• The verifier re-runs checks. For each acceptance criterion it runs the check named in the task and compares the result with what was claimed.
• Six facts are reported separately: edited, committed, launched, ran OK, verified, accepted. Anything without evidence is recorded as unknown.
• Dead ends are recorded. A hypothesis that was killed goes into the decision log with its reason, so later agents do not try it again.

5. Related work and limits

Anthropic’s long-running harness already treats the environment as memory: a progress file and git history let one agent resume a project across context windows [3]. Agensh makes tasks claimable objects in a shared workspace and scales a decentralized team to 1,024 agents [5]. Evolving Agent Teams keeps a persistent learnings file so a supervisor can restructure teams that stall [4]. GitSwarm lets homogeneous agents compound each other’s intermediate work through shared persistent memory [7]. None of them separates who verifies a result from who decides that it counts. That separation is the human-facing part this post adds.

• Not a runtime. There is no scheduler and no daemon. For an always-on agent that takes requests from a team’s chat and GitHub, see DeepSeek’s Super-agent prototype [1].
• No enforcement. The rules hold only because agents read and follow them. Enforcement at the execution layer is a separate job, handled by tools such as agentsh [6].
• Not for chores. If you already know the answer and there is one result to merge, ask the agent directly.
• One person’s practice. This is based on about four months of my own research projects. It is not a controlled study, the incidents are mine, and the example in section 1 is illustrative. It may not transfer to teams, to work where actions cannot be undone, or to projects where the hard part is waiting on other people.

Try it

Paste this into Claude Code or Codex:

Install Swarm Oversight Harness from https://github.com/cogito233/swarm-oversight-harness — follow its INSTALL.md.

Then say set up swarm oversight in this repo.

Feedback and issues are welcome on GitHub.

Disclaimer

This post was drafted, checked against the cited sources, and typeset by an AI coding agent (Claude Code), working with the author in the way described above. The author decided what to write and accepted or rejected each result. Any errors that remain are the author’s.

Citation

Cited as:

Lyu, Zhiheng. (Oct 2026). “Toward Human-in-the-Loop Oversight for Long-Horizon Agent Projects”. cogito233.github.io. https://cogito233.github.io/blog/human-in-the-loop-oversight/.

Or

@article{lyu2026oversight,
  title   = "Toward Human-in-the-Loop Oversight for Long-Horizon Agent Projects",
  author  = "Lyu, Zhiheng",
  journal = "cogito233.github.io",
  year    = "2026",
  month   = "Oct",
  url     = "https://cogito233.github.io/blog/human-in-the-loop-oversight/"
}

References