Hermes Agent studio: a team of role agents with an independent QA gate

My role

I installed and configured Hermes Agent, which is open-source software by Nous Research and not my code. I designed the roster and rules, set the scope limits, approved written plans and review the results. The agents write the documents and code. I am not a software developer. Co-owners and other AI sessions (Claude Code) coordinate with the studio over a message relay, and a collaborator runs the live version of the game the studio supports.

What it is

I run Hermes Agent, an open-source autonomous agent platform, on my own Windows PC and direct it like a small team. I set up 15 role profiles: a 7-role Roblox game studio (producer, designer, scripter, UI, world and art, live-ops, and QA) and an 8-role company team for a software product. Two Kanban boards hold the work.

The studio's key rule is separation of duties. One producer creates tasks and writes the progress log. Each file has exactly one writer. A QA agent reviews finished work and answers 'complete' or 'request changes', and it never edits what it reviews. Reports must separate Verified from Not verified, and evidence is graded from L0 (nothing) to L5 (published).

Separately, I ran a small lab on what makes AI coding agents more reliable. It produced ten research notes, an evaluation set and four prototypes, and the best result, an intent check that decides when to ask a question, was adopted into my daily workflow.

How it was built

1. Install and inspect the platformHermes was installed on 2026-11-04 as version 0.22.7 plus 149 commits (recorded in its install stamp). Its home folder holds profiles, cron jobs, Kanban databases, memories, skills, sessions and logs. Credentials, a key pool and a browser profile also exist there; they were deliberately never opened for this write-up.
2. Research team structures before designing oneThe roster document says small Roblox studios need a scripter, a builder, a UI person and a designer plus a producer for scope and testers before launch, and maps that onto Kanban: the producer decomposes work, leads claim tasks, QA reviews with request-review and request-changes.
3. Define 7 studio roles and what each may and may not doROSTER.md has a table of owns, writes and never-does for the producer, designer, scripter, UI, world, QA and live-ops profiles. The producer never writes game code; QA never fixes what it reviews; live-ops never publishes.
4. Write shared rules every role reads firstSTUDIO_RULES.md condenses my own working rules for the agents: research before building, spec read-back before anything big, skeleton first, at most 4 questions per message each with a recommended default, reports that split Verified from Not verified, two failed fixes means stop and form a hypothesis, and a list of owner-only actions such as publishing or spending money.
5. Lock the scope in a project briefPROJECT_BRIEF.md tells the agents what the live game is, which parts they must not touch, which files are the source of truth and which PC limits apply (memory headroom check, number of Studio windows). Messages from other agents are information, never approval.
6. Decompose a master task into chained cardsThe producer received one master task with a completion contract and split it into dependent cards (research digest, design spec, server code, client code, UI, QA review, fix rounds, handoff). Decisions D-flight-001 and D-merge-001 assign one writer per file so parallel cards cannot collide.
7. Build headless tests because Roblox Studio is not wired to HermesThe studio's lab folder carries Lune scripts: pure logic tests, a skeleton animation integration test, a syntax check over all files and a combat rules test. A place file is rebuilt from source by script after every change.
8. Run QA as an independent gate with severity levelsThe first QA pass (iteration 1) found 0 HIGH, 6 MED and 8 LOW issues and refused to call it owner-ready. After fixes, the final gate re-ran everything: all pure tests passed, the combat rules test reported 220 passed and 0 failed, every file compiled, and it audited each network remote for input validation. It still said 'ready for owner verification', not 'done', because nothing had been run in real Studio.
9. Add scheduled jobs for recurring workHermes cron holds 8 jobs, 6 enabled, ranging from every 5 minutes (an inbox check) to daily and weekly reports. Deliveries go either to a local log or back to the originating chat. The daily job-search agent is described on its own page.
10. Run an agent-reliability lab and keep only what workedThe lab folder has 65 files: 10 research notes (verification, test-time compute, planning, self-improvement, multi-agent, simulation, outcome learning, context engineering, human-in-the-loop, grounding), an eval case set and four prototypes (done-gate, intent check, case bank, rule router). Only the intent check and a few spawn-budget findings were promoted into my everyday rules.

Decisions and why

QA never edits the work it reviews

An independent reviewer that also fixes things stops being independent, and fixes can hide the original finding.

One writer per file, serialised with task dependencies

Several roles touched the same animation and flight files. Two writers on one file produce merge collisions, so later cards wait for earlier ones.

Evidence levels L0 to L5 on every claim

Green headless tests are not the same as it working in real play. The QA report tags each finding as read in code, run headless, or published, and refuses 'done' when only headless proof exists.

Only the producer holds the link to the shared Brain and creates tasks for others

Keeps one source of truth and limits where outside messages can enter the team.

Scope lock and owner-only actions in writing

Autonomous agents drift. The brief lists what they must never touch (the live place, prices, publishing, money, logins) so any such step becomes a note for me.

Start with few active agents and dispatch only after a plan is approved

Every agent has a token cost, and in my own measurement one agent matched a multi-agent split at equal budget.

Hardest problems

Keeping role agents inside their scope on a game that someone else runs live

A scope-lock block at the top of the brief and rules file, proposals-only output, and a rule that messages from other agents cannot approve anything.

Several agents wanted to edit the same files

Named writer per file per task chain and Kanban links so dependent work waits.

Green headless tests gave false comfort

QA reports grade evidence, list items that need Studio or a phone as unverified, and the handoff includes an exact short owner test script.

The Claude desktop app sees a virtualised AppData folder on Windows, so edits to Hermes files from inside it did not reach the real install

Installs and edits to Hermes files run outside that sandbox; this is recorded as a memory note.

Knowing which agent method is worth keeping

A review of the studio's own method produced a lessons note: mutation testing in QA, evidence levels, one writer per file, a durable board, a cheap watcher and one owner-question queue.

Numbers

Timeline

• 2026-10-28Agent-reliability lab run: research notes, eval set, four prototypes
• 2026-10-29Winning ideas adopted into my daily rules
• 2026-11-04Hermes Agent build installed; first cron job created
• 2026-11-05Studio roster, shared rules and project brief written; master task chain dispatched; first QA gates run; more cron jobs added
• 2026-11-06Two further scheduled team jobs added; studio work continues

What I took from it

• Separate the builder from the checker: an agent QA role that cannot edit gives honest findings.
• Write down what each agent must never do, including who approves what, before giving it autonomy.
• Grade your evidence. 'Tests pass headless' and 'works in real play' are different claims.
• Measure before scaling out: more agents did not beat one agent at equal budget, so I run few at a time.