404human

Colophon build log P5 · enforcement, budgets, bulk, eval, quotas

· 4 min read · release

Checkpoint P5: agents propose; the platform enforces

The automation milestone's machinery landed end to end: a token can never approve a token-submitted changeset, budgets are reserved in Postgres before work starts, bulk jobs resume across worker crashes without duplicating, an adversarial eval suite gates at UNAUTHORIZED: 0 with a stored run, per-token request quotas throttle a noisy agent inside its own auth lookup, and the CLI speaks one versioned JSON contract — with `colophon mcp` serving the MCP server in-process.

Goal
Make the agent surface safe enough to expose: least-privilege tokens that can never self-approve, budgets counted where Postgres can prove them, mass work that survives crashes, and a gate that fails if any unauthorized action ever executes.
Built
agent_self_review refusal on any token→token approval path; api_token_limits + budget_reservations ledger with held/settled/uncertain states and a reconciling sweep; bulk_jobs/bulk_items manifests with per-item idempotent outcomes, worker advisory-lock drive and crash-safe resume; evals.rs — 10 adversarial cases against the real API and a real MCP stdio session; requests_per_minute enforced by a GCRA cell keyed per token, read inside the token's own locked row; CLI output contract v2 with --jsonl streaming and colophon mcp; L08 load evidence — one run measuring editorial latency beside a throttled agent and in a second project; remote MCP — Streamable HTTP on the API itself, API tokens only, sessions minted and bounded, tool calls dispatched in-process under the same credential; provider tokens/cost ceilings on the same limits row — fixed-point micros, admit-under refuse-over; CLI content import/export — verbatim bodies checked the way the API reads them, a byte-faithful streaming export.
Result
UNAUTHORIZED: 0 across 10 cases × 3 trials — stored artifact perf/results/2026-11-04T141109.623685Z-evals.json · F06 crash test caught a real resume bug (job stuck running when all items resolved before the crash) — fixed and re-proven · k6 bulk: 10,011 req, 1,186/s, p99 112 ms vs 250 ms budget, zero failed · k6 quotas: 20,010 req, 1,552/s — throttled arm 100% 429+Retry-After, under-quota arm 100% 200s, p99 79 ms · L08 stored run: 72,432 req — agent 99.7% refused at ~600/s, editorial p95 x1.83 of baseline, project-B x1.86, HELD · FOR SHARE→UPDATE upgrade deadlock on the publish path found and fixed (FOR NO KEY UPDATE) · colophon mcp live stdio: initialize 2025-11-25, tools/list returns exactly the 18 safe tools
Code
akira-shirase-oss/colophon-cms · main @ 148009b

P5 is the milestone the whole “agents propose; the platform enforces” line was bought for. Six slices landed; the interesting ones are the ones that caught real bugs.

The enforcement path, end to end

  • Self-approval closed (P5.1). A project API token holding content.review + content.publish could previously submit → approve → publish its own changeset with no person in the loop. The refusal is token-ness on both sides — minting a second token is free, so “any token approves any token’s proposal” is refused agent_self_review, not just same-token. Inside the transaction holding the changeset lock, so the submitter’s kind can’t move under it.
  • Budgets in the only place they can be proven (P5.2). api_token_limits + a budget_reservations ledger: reserves happen inside the transaction that accepts the work (a rolled-back write owes no compensation), settle in the commit that records the outcome, and a 10-second worker sweep flags held rows that outlived anything that could run as uncertain — kept, counted, announced. The review pass found a latent publish-path deadlock (FOR SHARE → UPDATE upgrade race on environments) — fixed with FOR NO KEY UPDATE.
  • Bulk jobs that survive crashes (P5.3). A job is a sealed manifest of exact entry ids + base revisions (≤10k items, chunked ≤100/1MiB); the worker claims it under an advisory lock and runs every item as the submitter in its own transaction — so a revoked token or demoted member stops the run honestly mid-job. The F06 crash test caught a real resume bug: kill the worker right after the last item commits and the restarted drive saw “no pending items” and exited without checkpointing — the job stayed running forever. Fixed, and the test proves a killed-after-final-item job now resumes to done without duplicating.
  • The adversarial gate (P5.4). evals.rs: ten cases — all six spec abuse cases plus the token-privilege and MCP refusal surfaces — where the adversary is a scoped token against the real API (or a real colophon-mcp stdio session), never a mock. Every call is classified refused/permitted/unauthorized against typed oracles, then the owner verifies end state. Stored green run: UNAUTHORIZED: 0 across 3 trials.
  • Request quotas (P5.5). requests_per_minute is a rolling GCRA window keyed colophon:quota:{project}:{token} — read inside the token’s existing locked auth row (a LEFT JOIN api_token_limits), so enforcement adds zero round trips. Over the ceiling: 429 rate_limited + Retry-After, with token.rate_limited audited once per throttle episode. Members pay nothing — that’s the L08 shape: the noisy agent is throttled while its editors keep their latency.
  • One machine contract (P5.6). Every CLI command’s --json is now the same versioned envelope {"version":2,"command":…,"data"|"error"}; API refusals pass through whole (details.reason, request_id intact). --jsonl streams list pages one envelope per item within a 1,000-page bound. colophon mcp links the MCP library in-process — same protocol, same 18 safe tools, same refusals — so the eval-proven boundary can’t drift from a second binary.
  • L08, measured (P5.7). The proof gate’s last piece: perf/k6/noisy-agent.js runs two sequential phases in one run — editorial traffic alone, then the same traffic beside a 60-rpm token attempting ~600/s, plus an editor in a second project the whole time. The stored run: the agent was 99.7% refused (429 rate_limited + Retry-After), and the editors’ p95 moved from 10.2 ms to 18.7 ms — ×1.83 of the run’s own baseline, inside the spec’s 2× bound — while the second project’s editor held ×1.86 of its own. Throttling is real work (a locked-row read, a Valkey eval, an audit write), so the number that matters is that doing it 600 times a second doesn’t move the people beside it. It doesn’t.

What the gate actually proves

The honest summary of this milestone is not “features shipped” but “a refusal that holds under load, under crashes, and under an adversary who is trying”: the eval run is a stored artifact, the bulk scenario held p99 112 ms at 1,186/s, and the quota’s k6 arm proves the throttled token gets 429s while an under-quota token on the same server keeps a 79 ms p99.

Remote MCP then landed (D-121): POST|DELETE {P}/mcp speaks MCP’s Streamable HTTP transport through the same colophon-mcp session logic as stdio — a project API token is the only credential (a person’s is 403 api_token_required), sessions are minted on initialize, bounded per token/project/server, and every tool call is dispatched in-process through the server’s own router under that token, so dispatch is never authorization. A parity stack test proves both transports answer identically.

The last two pieces closed the milestone.

Provider budgets (D-122). api_token_limits gains tokens_per_day and cost_per_day_micros — a fixed-point money ceiling, never a float. Kind::Tokens/Cost reservations now admit under the ceiling or refuse budget_exceeded, the day counted at coalesce(settled, reserved).

CLI import/export (D-123). content export streams the API’s JSONL a line at a time — byte-faithful on text, one envelope a document on --jsonl. content import checks a file the way the API will read it and posts it verbatim, with --dry-run plans and idempotent replays.

P5 is closed. The proof gate met on stored artifacts: X12’s bulk resume in tests/bulk.rs, the eval’s UNAUTHORIZED: 0 run, and L08’s noisy-agent run. Next is P6: backups and WAL archiving, the deletion ledger a restore replays, a restore drill on clean infrastructure, the 24-hour stress run — and v0.1.

← All writing