The Problem With Letting AI Grade Its Own Homework

I've been spending a ridiculous amount of time working with different AI coding tools lately. Claude, Codex, Cline, Copilot, and whatever else happens to be particularly good at writing code this week. They're all useful, they all have their strengths, and every single one of them has an impressive ability to confidently tell you something works when it absolutely does not.

Anyone who has spent enough time coding with AI knows the routine. You give it a reasonably complicated task, it disappears into the code for a while, and eventually announces that everything is finished. The implementation is complete, the code looks great, and all that's left is for you to enjoy your wonderful new feature. Then you run the application and discover that apparently "finished" has a slightly different definition in the world of artificial intelligence.

I started wondering why I was relying on one AI to write the code, review the code, test the code, and ultimately decide whether its own work was any good. We generally don't let human developers approve their own pull requests without another set of eyes, so why was I trusting a machine that occasionally forgets what it was doing halfway through a conversation?

That little irritation eventually became Conspirator.

Getting the Machines to Work Together

The idea behind Conspirator is relatively simple. Instead of making one AI responsible for everything, let multiple coding agents work together with clearly defined responsibilities. One handles the implementation while another independently reviews the results. Ideally, they catch each other's mistakes before those mistakes become my problem.

Conspirator currently works with Claude Code, OpenAI Codex CLI, Cline, and GitHub Copilot CLI. By default, Claude takes the role of orchestrator and independent auditor while one of the other agents handles the actual implementation. I can tell Claude something as simple as "Pair on this using Codex," and Conspirator handles the process of assigning the work, collecting the results, running verification, and preparing the implementation for review.

The important distinction is that these aren't a bunch of AI models having a philosophical discussion about how wonderful their proposed solution might be. The implementing agent is expected to write actual code, commit its changes, and provide evidence of what it accomplished. Claude then inspects the implementation, checks the requirements, reviews the changes, and performs its own verification before approving anything.

If the code doesn't meet the requirements, it goes back for another round. The implementing AI doesn't get to declare victory just because it produced a particularly enthusiastic summary of its accomplishments. I've already got enough opportunities to be disappointed by software without allowing the software to award itself a participation trophy.

Less Talking, More Coding

Of course, getting multiple AI models involved introduces another problem. Tokens cost money, context windows aren't infinite, and AI agents have a remarkable tendency to explain everything they've ever learned whenever you ask them a simple question.

Having two models repeatedly exchange thousands of lines of source code, documentation, and conversational nonsense isn't particularly efficient. At that point, you're paying extra for the privilege of watching two computers write each other increasingly expensive emails.

To deal with that, I built Conspirator around a compact communication format called CAVEMAN/1. Rather than sending enormous explanations between agents, it passes along the information they actually need: goals, constraints, file references, acceptance criteria, changes, test results, and failures.

The agents can read the code directly from the repository, so there's no reason to keep copying the same files back and forth. Detailed output stays in local logs, while the communication between agents concentrates on what changed and what needs attention. When something fails, the next round receives the relevant failure information instead of another complete retelling of the project's history.

It's a fairly straightforward concept, although convincing AI not to talk excessively is apparently one of the more ambitious engineering challenges of our time.

Keeping Everyone Out of Each Other's Way

Another thing I wanted to avoid was having multiple agents trampling over the same files while working on a task. That's a great way to turn an ordinary bug fix into an archaeological expedition through conflicting changes.

Conspirator uses isolated Git worktrees to give the implementing agent its own workspace. The agent can make changes, run tests, and commit its implementation without interfering with the primary working tree. Once the work is complete, the verification and independent audit process determines whether those changes are ready to be merged.

The workflow also keeps a Git-based history of the task, including its progress, results, failures, and audit decisions. There isn't a separate database or service required to coordinate everything. Conspirator's runtime is written in Python using the standard library, with Git handling the persistent task history.

I also wanted failure handling to be predictable. If an implementation fails verification, Conspirator can send it back for another attempt, but there are limits on how many times that happens. Eventually, somebody needs to admit that the problem hasn't been solved rather than burning through another small country's electricity supply trying the same broken approach.

Approved work can be merged with a backup reference, and there's a rollback mechanism if something needs to be undone. None of this guarantees that bugs will magically disappear, but it gives the whole process a little more discipline than simply trusting whichever model sounds the most confident.

Why I Built It This Way

I don't particularly care which AI company wins the ongoing competition to produce the greatest coding model. Claude might be better at understanding one problem, Codex might handle another more effectively, and Cline or Copilot might be the better choice for something else entirely. That changes constantly, and I'd rather have a workflow that can take advantage of those differences than build everything around whichever model happens to be fashionable this month.

What I do care about is getting reliable, working code without having to personally babysit every decision along the way. I want to describe what needs to happen, let the tools do their jobs, and have something other than the original author determine whether the result is acceptable.

There are limits, obviously. Conspirator isn't intended to turn every tiny code change into a multinational development committee. If I'm changing a label or fixing a typo, I don't need Claude and Codex convening an emergency review board. The real value comes from features, bug fixes, refactoring, and other substantial changes where having an independent reviewer can justify the additional effort.

It's also worth remembering that two AI models can still agree on something completely wrong. Independent review improves the process, but it doesn't replace good requirements, meaningful tests, or the occasional involvement of a human being with a functioning brain.

The Bigger Picture

Conspirator is another piece of my growing FKNRTD collection of tools intended to make AI-assisted development a little less painful. Butcher works on cutting unnecessary token usage, Inquisitor makes sure we understand what we're supposed to be building, and Odin investigates and documents the results. Conspirator fits into that collection by making the coding agents collaborate without creating an unmanageable circus.

Ultimately, I want to spend less time managing AI tools and more time building the things I actually want to build. That means letting different models contribute where they're useful, making them accountable for their work, and keeping enough control over the process that I can figure out what happened when something inevitably catches fire.

If I'm going to have several artificial intelligences helping me write software, I might as well make them check each other's work. At the very least, when something goes wrong, I'll have more than one machine to blame.

Conspirator is available on GitHub:
https://github.com/nullcromancer/FKNRTD.SKL/tree/main/conspirator

As always, be good, or don't get caught.