RichardHam.co.uk Blog

Tag: automation

  • When Maintenance Starts to Look Like the Product

    One of the clearest signs that a system is growing up is that the most important work stops looking dramatic.

    There is less fascination with launch theatre and more attention on maintenance, review discipline, dependency hygiene, documentation, recovery paths, and the quiet operational habits that make future change cheaper than past change.

    From the outside, that can look unexciting. From the inside, it is often the moment the product becomes believable.

    Maintenance is where trust becomes visible

    Early-stage work is usually easy to narrate. New feature. New workflow. New integration. New capability.

    Maintenance work is harder to sell because it rarely produces a neat headline. But it is where a team proves whether it is building something durable or simply accumulating demonstrations.

    If the documentation sharpens, the dependency stream stays current, the review process gets clearer, and the operational logs start forming a usable trail, those are not background chores. They are evidence that the system can survive repetition.

    That matters because most real-world products do not fail during the polished demo. They fail during handover, under load, during maintenance, or when somebody new has to operate them without the full story.

    Governance is not separate from delivery

    A lot of teams treat governance as a separate lane from product work, as though it begins after the useful engineering is finished.

    I think that is backwards.

    Governance is simply the part of delivery that reduces ambiguity for the next decision. It is what turns one-off effort into something that can be inspected, repeated, and improved.

    That can show up in small ways:

    • proposal scoring that makes prioritisation legible
    • review guidance that reduces inconsistent judgement
    • issue tracking that records blockers instead of burying them in chat
    • branch hygiene that makes the delivery path safer
    • run logs that explain what changed and why

    None of that steals time from the product. In mature systems, it becomes part of the product because it changes the cost and risk of every future change.

    The product is bigger than the feature set

    This is the shift many teams eventually have to make.

    The product is not only the visible interface or the raw capability. It is also the collection of operating properties that determine whether the capability can be trusted. Can another operator pick it up? Can a failure be explained? Can a dependency be updated without drama? Can the next release happen without rediscovering everything from scratch?

    Once you ask those questions seriously, maintenance stops looking secondary.

    A healthy maintenance pattern normally improves at least one of these:

    • repeatability
    • auditability
    • reversibility
    • ownership clarity
    • change safety

    If the work improves none of those things, it may be motion without much payoff. But when it does improve them, it is absolutely product work.

    The hidden cost of pretending maintenance is optional

    Teams that down-rank maintenance tend to pay for it later in awkward ways.

    A roadmap becomes harder to trust because the underlying stack drifts. Delivery gets slower because every change has to rediscover old context. Incidents become more expensive because the recovery path is still tribal knowledge. Content and documentation diverge because nobody kept a canonical source of truth.

    None of that feels catastrophic at first. It just makes every subsequent piece of work more fragile.

    That is why mature engineering groups often sound calmer, not louder. They know that the best way to speed up later is to reduce the amount of avoidable uncertainty now.

    What good maintenance work usually has in common

    When maintenance is genuinely improving the system rather than just consuming time, I usually see a few shared characteristics.

    It leaves a clearer trail

    The next operator can understand what happened without interviewing the previous one.

    It reduces future decision cost

    A known pattern, documented rule, or reusable checklist means the same problem will be cheaper next time.

    It makes failures less mysterious

    Even when something still goes wrong, the team has better evidence and a cleaner path to recovery.

    It protects momentum instead of slowing it

    Strong maintenance work makes future delivery easier because the underlying operating model is less chaotic.

    Why this matters so much in automation and AI

    Automation magnifies both good and bad maintenance habits.

    If the workflow is opaque, poor maintenance leaves you with a black box that degrades quietly. If the workflow is well-governed, maintenance turns it into something operators can trust, audit, and extend without guessing.

    That is one reason I think governance, maintenance, and documentation matter more as systems become more autonomous. The machine may be doing more of the execution, but the human still has to understand the operating model well enough to own the outcome.

    That is impossible if maintenance has been treated as optional admin.

    The real signal of maturity

    The real signal of maturity is not that a team has stopped building. It is that the team has started building in a way that leaves the environment safer for the next change.

    That often looks like better maintenance because that is what it is.

    Not glamorous. Not particularly marketable on its own. But essential.

    And once a product reaches that stage, the maintenance work does not sit behind the product. It becomes one of the reasons the product is worth trusting in the first place.

    If you are trying to make systems easier to run, safer to change, and less dependent on tribal memory, the AI & Automation Architecture work is built around exactly that operating model. Or get in touch if you want help turning maintenance, governance, and delivery discipline into an actual advantage rather than a recurring source of drag.

  • The Week Maintenance Became the Product

    Introduction

    When I reviewed this week’s GitHub activity, the obvious story was not a flashy launch or a dramatic refactor. It was something more interesting: the maintenance work became the signal.

    Across 20 public repositories, there were 126 events in the last seven days. That includes dependency churn, proposal scoring, documentation hardening, automation that keeps generating useful operational signals, and a few infrastructure notes that show where the rough edges still are. In other words, the system is doing what mature systems do: it is spending less time proving that it works and more time proving that it can be trusted.

    That matters. A lot of teams talk about shipping. Fewer teams talk about the work that makes shipping repeatable. This week was a good reminder that the second part is where the real leverage lives.

    What happened

    Project Atlas moved from ideas into structure

    The project-atlas-foundation repo was the busiest in the set, with 40 events. The pattern was clear: proposals were being scored, shaping docs were being created, and the launch checklist was being tightened up.

    A few examples stood out:

    • proposal scoring for items #3–#7
    • issue creation for shaping the next phase of work
    • launch planning documents and handover notes
    • an infrastructure blocker being logged rather than ignored

    That last point matters. Good teams do not hide blockers behind optimism. They surface them early, name them clearly, and move on with the fix.

    What I like about this kind of activity is that it shows a project moving from momentum to discipline. Ideas are important, but structured ideas are what survive contact with reality.

    HamMediaLabs built the scaffolding around the work

    The HamMediaLabs repo contributed 24 events, and the theme was governance. Onboarding material, a development guide, a risk register, a PR review dashboard, a dependency health report, and a branch hygiene policy all landed in quick succession.

    That is not glamorous work. It is, however, the work that keeps the rest of the team from drifting into inconsistency.

    I have seen enough small teams to know this pattern well: once the repository starts to matter, the undocumented habits start to cost real time. Branch hygiene prevents stale work from hanging around. A PR dashboard shows where the bottlenecks are. A risk register makes it harder to ignore known issues until they become incidents.

    This is governance that lives with the code, which is the only place it reliably gets used.

    Control Tower kept producing daily operational signals

    The control-tower repo was smaller in volume, but it was one of the most revealing. The automation continued to produce daily “Decision Desk” issues, including entries for June 24 and June 25.

    That might sound routine, and that is exactly why it matters.

    A healthy automation pipeline should stop feeling novel. It should become part of the operating rhythm. When the bot keeps producing the same class of signal every day, it means the process is stable enough to be useful and visible enough to trust.

    The lesson here is not that automation is exciting. It is that automation is only useful when it becomes boring in the right way.

    ai-cost-tracker showed the cost of staying current

    The ai-cost-tracker repo generated a series of Dependabot updates across scipy, openai, pytest, coverage, numpy, and a pip group update. That is the kind of activity people often skim past, but I think it tells an important story.

    Dependency maintenance is not just housekeeping. It is a proxy for the health of the project.

    If updates are ignored for too long, the stack gets harder to trust. If they are handled routinely, the project stays closer to current, and current is cheaper than catching up later. In a world where AI tooling and Python libraries evolve quickly, that matters even more. Every stale dependency is a future problem with interest attached.

    hermes-agent continued to harden the core toolchain

    The hermes-agent repo added a SecureScore view through PR #1, while other commits focused on gateway hardening, Windows restart reliability, and test improvements.

    That combination is exactly what I want to see from a core platform repository. A view is added because the team needs better visibility. The gateway is hardened because resilience matters. Tests are improved because confidence is not something you can fake for very long.

    This is the difference between building a tool and operating a system.

    Key takeaways

    1. Maintenance is becoming product work

    The old mental model says feature work is valuable and maintenance is overhead. This week argues for a better model: maintenance is part of the product.

    If you are scoring proposals, documenting risks, tightening branch rules, and keeping dependencies current, you are not stepping away from the product. You are building the conditions that let the product keep existing.

    2. Automation is most valuable when it is visible

    The daily Decision Desk issues in control-tower are a good example. Automation should not disappear into a black box. It should leave a trace that operators can inspect.

    That trace becomes a decision record, a trend line, and a health indicator all at once. If your automation cannot explain itself in the repository, it is probably too fragile to trust elsewhere.

    3. Governance only works when it is close to the code

    Onboarding guides, risk registers, review dashboards, and branch hygiene policies are all useful because they are embedded in the same workflow as the work they govern.

    That is the difference between documentation and practice. One gets read when a problem appears. The other shapes the problem before it appears.

    4. Dependency updates are an operational metric

    The ai-cost-tracker updates are not just noise from a bot. They are evidence that the project is being actively maintained.

    If dependency updates are arriving regularly, that means somebody is paying attention. If they are not, the project may still look healthy right up until the day it suddenly is not.

    5. Mature teams spend more time making work repeatable

    The most important shift I saw this week was not in any single repository. It was in the shape of the work overall.

    The repositories are spending time on handover notes, launch checklists, review dashboards, daily operational signals, and stability fixes. That is what maturity looks like in practice. Not less work, just better-structured work.

    Closing thought

    If you only scan the headlines of a busy week, you can miss the real story. This one was not about one big release. It was about the quiet engineering that makes releases sustainable.

    That is usually where the long-term value sits: in the boring, repeatable work that turns a collection of repos into a system.

    If you are building something similar and want help turning operational complexity into something more manageable, that is exactly the kind of work I cover in my services and contact pages.

  • The Week Maintenance Became the Product

    Introduction

    When I reviewed this week’s GitHub activity, the obvious story was not a flashy launch or a dramatic refactor. It was something more interesting: the maintenance work became the signal.

    Across 20 public repositories, there were 126 events in the last seven days. That includes dependency churn, proposal scoring, documentation hardening, automation that keeps generating useful operational signals, and a few infrastructure notes that show where the rough edges still are. In other words, the system is doing what mature systems do: it is spending less time proving that it works and more time proving that it can be trusted.

    That matters. A lot of teams talk about shipping. Fewer teams talk about the work that makes shipping repeatable. This week was a good reminder that the second part is where the real leverage lives.

    What happened

    Project Atlas moved from ideas into structure

    The project-atlas-foundation repo was the busiest in the set, with 40 events. The pattern was clear: proposals were being scored, shaping docs were being created, and the launch checklist was being tightened up.

    A few examples stood out:

    • proposal scoring for items #3–#7
    • issue creation for shaping the next phase of work
    • launch planning documents and handover notes
    • an infrastructure blocker being logged rather than ignored

    That last point matters. Good teams do not hide blockers behind optimism. They surface them early, name them clearly, and move on with the fix.

    What I like about this kind of activity is that it shows a project moving from momentum to discipline. Ideas are important, but structured ideas are what survive contact with reality.

    HamMediaLabs built the scaffolding around the work

    The HamMediaLabs repo contributed 24 events, and the theme was governance. Onboarding material, a development guide, a risk register, a PR review dashboard, a dependency health report, and a branch hygiene policy all landed in quick succession.

    That is not glamorous work. It is, however, the work that keeps the rest of the team from drifting into inconsistency.

    I have seen enough small teams to know this pattern well: once the repository starts to matter, the undocumented habits start to cost real time. Branch hygiene prevents stale work from hanging around. A PR dashboard shows where the bottlenecks are. A risk register makes it harder to ignore known issues until they become incidents.

    This is governance that lives with the code, which is the only place it reliably gets used.

    Control Tower kept producing daily operational signals

    The control-tower repo was smaller in volume, but it was one of the most revealing. The automation continued to produce daily “Decision Desk” issues, including entries for June 24 and June 25.

    That might sound routine, and that is exactly why it matters.

    A healthy automation pipeline should stop feeling novel. It should become part of the operating rhythm. When the bot keeps producing the same class of signal every day, it means the process is stable enough to be useful and visible enough to trust.

    The lesson here is not that automation is exciting. It is that automation is only useful when it becomes boring in the right way.

    ai-cost-tracker showed the cost of staying current

    The ai-cost-tracker repo generated a series of Dependabot updates across scipy, openai, pytest, coverage, numpy, and a pip group update. That is the kind of activity people often skim past, but I think it tells an important story.

    Dependency maintenance is not just housekeeping. It is a proxy for the health of the project.

    If updates are ignored for too long, the stack gets harder to trust. If they are handled routinely, the project stays closer to current, and current is cheaper than catching up later. In a world where AI tooling and Python libraries evolve quickly, that matters even more. Every stale dependency is a future problem with interest attached.

    hermes-agent continued to harden the core toolchain

    The hermes-agent repo added a SecureScore view through PR #1, while other commits focused on gateway hardening, Windows restart reliability, and test improvements.

    That combination is exactly what I want to see from a core platform repository. A view is added because the team needs better visibility. The gateway is hardened because resilience matters. Tests are improved because confidence is not something you can fake for very long.

    This is the difference between building a tool and operating a system.

    Key takeaways

    1. Maintenance is becoming product work

    The old mental model says feature work is valuable and maintenance is overhead. This week argues for a better model: maintenance is part of the product.

    If you are scoring proposals, documenting risks, tightening branch rules, and keeping dependencies current, you are not stepping away from the product. You are building the conditions that let the product keep existing.

    2. Automation is most valuable when it is visible

    The daily Decision Desk issues in control-tower are a good example. Automation should not disappear into a black box. It should leave a trace that operators can inspect.

    That trace becomes a decision record, a trend line, and a health indicator all at once. If your automation cannot explain itself in the repository, it is probably too fragile to trust elsewhere.

    3. Governance only works when it is close to the code

    Onboarding guides, risk registers, review dashboards, and branch hygiene policies are all useful because they are embedded in the same workflow as the work they govern.

    That is the difference between documentation and practice. One gets read when a problem appears. The other shapes the problem before it appears.

    4. Dependency updates are an operational metric

    The ai-cost-tracker updates are not just noise from a bot. They are evidence that the project is being actively maintained.

    If dependency updates are arriving regularly, that means somebody is paying attention. If they are not, the project may still look healthy right up until the day it suddenly is not.

    5. Mature teams spend more time making work repeatable

    The most important shift I saw this week was not in any single repository. It was in the shape of the work overall.

    The repositories are spending time on handover notes, launch checklists, review dashboards, daily operational signals, and stability fixes. That is what maturity looks like in practice. Not less work, just better-structured work.

    Closing thought

    If you only scan the headlines of a busy week, you can miss the real story. This one was not about one big release. It was about the quiet engineering that makes releases sustainable.

    That is usually where the long-term value sits: in the boring, repeatable work that turns a collection of repos into a system.

    If you are building something similar and want help turning operational complexity into something more manageable, that is exactly the kind of work I cover in my services and contact pages.

  • GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    g

    The week of June 15-22 was not defined by a single dramatic event. Instead, it was the kind of week that separates platforms that merely work from platforms that hold up under sustained pressure. Across 50 repositories, over 100 events — commits, pull requests, issues — pushed forward several long-running threads: memory architecture, health monitoring, security governance, and the unglamorous but essential work of keeping production infrastructure honest.

    What Happened

    hermes-mgmt — Hardening the Core

    The hermes-mgmt repository remained the busiest node in the network, with 20-plus commits, 20 PRs, and 13 issues. The dominant theme was memory system reliability. A significant fix (PR covering issues #573, #574, #575) hardened dual_memory.py across three dimensions: Ollama-first Mem0 configuration, deterministic Letta archival behaviour, and Qdrant vector dimension alignment. These are the kinds of fixes that don’t make headlines but prevent the subtle data corruption that erodes trust in AI systems over time.

    A related fix (issue #411) corrected a memory drift check that was comparing the wrong Letta data — core-memory passages instead of ARCHIVAL passages. This is a telling detail: as memory architectures grow more layered (core, archival, vector), the surface area for misaligned reads increases. Catching this before it caused silent data degradation matters.

    Health probes got a major upgrade with PR #568, which introduced functional health checks for services that report as “green” but are actually broken — Letta, n8n, Langfuse, Qdrant, Gateway, and Ollama. This is a pattern anyone running distributed systems will recognise: the dashboard shows all green, but something is quietly failing. Functional probes go beyond “is the process running?” to “does the service actually respond correctly?” — a distinction that matters enormously in production.

    Secrets management continued to mature. PR #552 delivered a comprehensive secrets management architecture built around Bitwarden and HashiCorp Vault, while PR #527 removed committed default secrets from the Langfuse compose configuration. These are the foundational security practices that need to be in place before scale makes them painful to retrofit.

    On the cron and scheduling side, the system now runs 38 cron jobs with 36 healthy — a 95% health rate that reflects the cumulative effect of weeks of hardening work. PR #504 corrected cron exit-code semantics so that findings and alerts are no longer misinterpreted as failures, which was causing unnecessary noise in the monitoring pipeline.

    Several open issues point to the next layer of work: Telegram flood-control protection (#583), gateway ungraceful shutdown (#582), Ollama-agent sequential dispatch performance (#580), and a Langfuse Cost Report cron that’s been disabled due to stale API keys (#576). None of these are emergencies, but they represent the known gaps that get addressed in order of priority.

    hamnet — Infrastructure Truth

    The hamnet repository, which manages infrastructure automation, saw 9 commits and 10 PRs with a focus on hosting hardening and monitoring.

    SSH hardening was codified for the VPS fleet (PR #131), including a dynamic-IP allowlist failsafe — a practical safeguard for environments where IP addresses shift. Vhost routing was hardened after www.theitrevolution.co.uk was found serving the wrong site, a class of misconfiguration that can be difficult to spot without deliberate verification.

    On the monitoring side, an Alertmanager webhook receiver was added with X-Webhook-Token authentication (PR #87), and a new Mac AI Server dashboard was built for the Ollama era (PR #134). The dashboard work reflects a broader shift: as the AI infrastructure stack evolves (Ollama replacing previous model servers), the monitoring layer needs to evolve in parallel or it becomes a liability rather than an asset.

    An open issue (#95) flags an SSL certificate SAN mismatch for www.richardham.co.uk — the kind of thing that works until it doesn’t, usually at the worst possible moment.

    hermes-agent — Gateway Stability

    The hermes-agent repository received 10 commits focused on gateway reliability. The command-line matcher was hardened, Windows restart no longer causes a silent outage, and the gateway now refreshes its cached agent max_iterations from the current config rather than stale values. A fix for dict choice unwrapping in the clarify function rounds out a set of small but meaningful stability improvements.

    These are the fixes that users never notice — because they prevent the failures that would have been noticed. Silent outages and stale config caches are particularly insidious because they can persist for extended periods before manifesting as user-visible problems.

    hermes-securescore — Evidence Collection

    The SecureScore project advanced with 9 commits and 2 PRs, adding both a Docker collector and a Hermes config collector for security evidence. A high-risk action approval playbook was also added, formalising the governance process for sensitive operations. The project recorded its live activation this week — a milestone that moves it from development into operational use.

    hermes-voice-satellite — Laying Groundwork

    The voice satellite project saw 9 commits establishing the MVP build plan, an operations runbook, a Hermes voice satellite API contract, and a Termux bootstrap script for Android (S24). A native Android app placeholder was also added. This is early-stage infrastructure work — the kind of foundation that needs to exist before feature development can accelerate.

    project-atlas-foundation — Launch Readiness

    Project Atlas received 10 commits focused on governance and launch preparation: a PR template, MIT licence, security policy, CODEOWNERS, a safety checklist, research workflow, promotion process, CI configuration, and hardened lifecycle guides. Scoring of proposals #3-#7 was completed alongside shaping docs and a launch checklist update. This is the organisational scaffolding that turns a codebase into a project other people can contribute to.

    AgentRadar — New Arrival

    AgentRadar received its initial commit this week. Details are sparse, but a new repository appearing in the portfolio is always worth noting — it represents a new thread that will either find its place or be retired honestly.

    richardham-web-and-Brand — Content Pipeline

    The web-and-brand repository saw 10 commits and 5 PRs, primarily focused on blog publishing and content pipeline maintenance. Four new blog posts in the agentic AI series were added with proper date spacing, and a build fix declared window.__calComLoaded to resolve a strict type-check failure. A WordPress publishing blocker was documented (VPS SSH key issue), which is the kind of honest infrastructure transparency that keeps content pipelines reliable.

    Key Takeaways

    Memory architecture is the new frontier. The volume of work on dual-memory hardening, Letta archival alignment, and Qdrant dimension matching signals that the memory subsystem has become a first-class concern. As AI agents handle longer and more complex tasks, the reliability of their memory layer directly determines the reliability of everything built on top of it. The drift-check fix — comparing the right data — is a small change with outsized implications.

    Functional health probes close a critical gap. The distinction between “process is running” and “service is working” is one of the most common blind spots in monitoring. Adding functional probes for six core services moves the observability stack from surface-level to meaningful. This is the kind of investment that pays for itself the first time it catches a green-but-broken service before a user does.

    Security governance is becoming systematic, not reactive. Between the secrets management architecture, the SecureScore evidence collectors, the high-risk action approval playbook, and the removal of committed defaults, the pattern is clear: security is being built into the development process rather than bolted on after incidents. This is the maturation path every platform needs to follow.

    Infrastructure truth matters. The hamnet work — SSH hardening, vhost routing fixes, SSL certificate monitoring, dashboard reconciliation — is the unglamorous foundation that everything else depends on. When www.theitrevolution.co.uk serves the wrong site, no amount of AI sophistication compensates. Keeping the infrastructure layer honest is a continuous discipline, not a one-time project.

    The content pipeline is converging with the platform. The parallel work on blog publishing, brand positioning, and the agentic AI content series is not separate from the technical work — it is how the technical work becomes visible and valuable. A hardened platform with strong governance needs an equally strong narrative around it.

    Looking Ahead

    The open issues across the portfolio paint a clear picture of next week’s priorities: resolve the Telegram flood-control design, address the gateway ungraceful shutdown path, fix the Langfuse Cost Report cron’s stale API keys, and close the SSL certificate SAN mismatch. On the infrastructure side, the Mac AI Server Ollama dashboard and metrics pipeline need to land, and the remaining open PRs in the web-and-brand repository need to progress through review.

    The velocity of the last week is notable not for its drama but for its consistency. Every repository moved forward. No single event dominated. That is what a healthy development portfolio looks like in practice — not a single sprint, but sustained, disciplined progress across every layer of the stack.

  • GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    The week of June 15-22 was not defined by a single dramatic event. Instead, it was the kind of week that separates platforms that merely work from platforms that hold up under sustained pressure. Across 50 repositories, over 100 events — commits, pull requests, issues — pushed forward several long-running threads: memory architecture, health monitoring, security governance, and the unglamorous but essential work of keeping production infrastructure honest.

    What Happened

    hermes-mgmt — Hardening the Core

    The hermes-mgmt repository remained the busiest node in the network, with 20-plus commits, 20 PRs, and 13 issues. The dominant theme was memory system reliability. A significant fix (PR covering issues #573, #574, #575) hardened dual_memory.py across three dimensions: Ollama-first Mem0 configuration, deterministic Letta archival behaviour, and Qdrant vector dimension alignment. These are the kinds of fixes that don’t make headlines but prevent the subtle data corruption that erodes trust in AI systems over time.

    A related fix (issue #411) corrected a memory drift check that was comparing the wrong Letta data — core-memory passages instead of ARCHIVAL passages. This is a telling detail: as memory architectures grow more layered (core, archival, vector), the surface area for misaligned reads increases. Catching this before it caused silent data degradation matters.

    Health probes got a major upgrade with PR #568, which introduced functional health checks for services that report as “green” but are actually broken — Letta, n8n, Langfuse, Qdrant, Gateway, and Ollama. This is a pattern anyone running distributed systems will recognise: the dashboard shows all green, but something is quietly failing. Functional probes go beyond “is the process running?” to “does the service actually respond correctly?” — a distinction that matters enormously in production.

    Secrets management continued to mature. PR #552 delivered a comprehensive secrets management architecture built around Bitwarden and HashiCorp Vault, while PR #527 removed committed default secrets from the Langfuse compose configuration. These are the foundational security practices that need to be in place before scale makes them painful to retrofit.

    On the cron and scheduling side, the system now runs 38 cron jobs with 36 healthy — a 95% health rate that reflects the cumulative effect of weeks of hardening work. PR #504 corrected cron exit-code semantics so that findings and alerts are no longer misinterpreted as failures, which was causing unnecessary noise in the monitoring pipeline.

    Several open issues point to the next layer of work: Telegram flood-control protection (#583), gateway ungraceful shutdown (#582), Ollama-agent sequential dispatch performance (#580), and a Langfuse Cost Report cron that’s been disabled due to stale API keys (#576). None of these are emergencies, but they represent the known gaps that get addressed in order of priority.

    hamnet — Infrastructure Truth

    The hamnet repository, which manages infrastructure automation, saw 9 commits and 10 PRs with a focus on hosting hardening and monitoring.

    SSH hardening was codified for the VPS fleet (PR #131), including a dynamic-IP allowlist failsafe — a practical safeguard for environments where IP addresses shift. Vhost routing was hardened after www.theitrevolution.co.uk was found serving the wrong site, a class of misconfiguration that can be difficult to spot without deliberate verification.

    On the monitoring side, an Alertmanager webhook receiver was added with X-Webhook-Token authentication (PR #87), and a new Mac AI Server dashboard was built for the Ollama era (PR #134). The dashboard work reflects a broader shift: as the AI infrastructure stack evolves (Ollama replacing previous model servers), the monitoring layer needs to evolve in parallel or it becomes a liability rather than an asset.

    An open issue (#95) flags an SSL certificate SAN mismatch for www.richardham.co.uk — the kind of thing that works until it doesn’t, usually at the worst possible moment.

    hermes-agent — Gateway Stability

    The hermes-agent repository received 10 commits focused on gateway reliability. The command-line matcher was hardened, Windows restart no longer causes a silent outage, and the gateway now refreshes its cached agent max_iterations from the current config rather than stale values. A fix for dict choice unwrapping in the clarify function rounds out a set of small but meaningful stability improvements.

    These are the fixes that users never notice — because they prevent the failures that would have been noticed. Silent outages and stale config caches are particularly insidious because they can persist for extended periods before manifesting as user-visible problems.

    hermes-securescore — Evidence Collection

    The SecureScore project advanced with 9 commits and 2 PRs, adding both a Docker collector and a Hermes config collector for security evidence. A high-risk action approval playbook was also added, formalising the governance process for sensitive operations. The project recorded its live activation this week — a milestone that moves it from development into operational use.

    hermes-voice-satellite — Laying Groundwork

    The voice satellite project saw 9 commits establishing the MVP build plan, an operations runbook, a Hermes voice satellite API contract, and a Termux bootstrap script for Android (S24). A native Android app placeholder was also added. This is early-stage infrastructure work — the kind of foundation that needs to exist before feature development can accelerate.

    project-atlas-foundation — Launch Readiness

    Project Atlas received 10 commits focused on governance and launch preparation: a PR template, MIT licence, security policy, CODEOWNERS, a safety checklist, research workflow, promotion process, CI configuration, and hardened lifecycle guides. Scoring of proposals #3-#7 was completed alongside shaping docs and a launch checklist update. This is the organisational scaffolding that turns a codebase into a project other people can contribute to.

    AgentRadar — New Arrival

    AgentRadar received its initial commit this week. Details are sparse, but a new repository appearing in the portfolio is always worth noting — it represents a new thread that will either find its place or be retired honestly.

    richardham-web-and-Brand — Content Pipeline

    The web-and-brand repository saw 10 commits and 5 PRs, primarily focused on blog publishing and content pipeline maintenance. Four new blog posts in the agentic AI series were added with proper date spacing, and a build fix declared window.__calComLoaded to resolve a strict type-check failure. A WordPress publishing blocker was documented (VPS SSH key issue), which is the kind of honest infrastructure transparency that keeps content pipelines reliable.

    Key Takeaways

    Memory architecture is the new frontier. The volume of work on dual-memory hardening, Letta archival alignment, and Qdrant dimension matching signals that the memory subsystem has become a first-class concern. As AI agents handle longer and more complex tasks, the reliability of their memory layer directly determines the reliability of everything built on top of it. The drift-check fix — comparing the right data — is a small change with outsized implications.

    Functional health probes close a critical gap. The distinction between “process is running” and “service is working” is one of the most common blind spots in monitoring. Adding functional probes for six core services moves the observability stack from surface-level to meaningful. This is the kind of investment that pays for itself the first time it catches a green-but-broken service before a user does.

    Security governance is becoming systematic, not reactive. Between the secrets management architecture, the SecureScore evidence collectors, the high-risk action approval playbook, and the removal of committed defaults, the pattern is clear: security is being built into the development process rather than bolted on after incidents. This is the maturation path every platform needs to follow.

    Infrastructure truth matters. The hamnet work — SSH hardening, vhost routing fixes, SSL certificate monitoring, dashboard reconciliation — is the unglamorous foundation that everything else depends on. When www.theitrevolution.co.uk serves the wrong site, no amount of AI sophistication compensates. Keeping the infrastructure layer honest is a continuous discipline, not a one-time project.

    The content pipeline is converging with the platform. The parallel work on blog publishing, brand positioning, and the agentic AI content series is not separate from the technical work — it is how the technical work becomes visible and valuable. A hardened platform with strong governance needs an equally strong narrative around it.

    Looking Ahead

    The open issues across the portfolio paint a clear picture of next week’s priorities: resolve the Telegram flood-control design, address the gateway ungraceful shutdown path, fix the Langfuse Cost Report cron’s stale API keys, and close the SSL certificate SAN mismatch. On the infrastructure side, the Mac AI Server Ollama dashboard and metrics pipeline need to land, and the remaining open PRs in the web-and-brand repository need to progress through review.

    The velocity of the last week is notable not for its drama but for its consistency. Every repository moved forward. No single event dominated. That is what a healthy development portfolio looks like in practice — not a single sprint, but sustained, disciplined progress across every layer of the stack.

  • GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    The week of June 15-22 was not defined by a single dramatic event. Instead, it was the kind of week that separates platforms that merely work from platforms that hold up under sustained pressure. Across 50 repositories, over 100 events — commits, pull requests, issues — pushed forward several long-running threads: memory architecture, health monitoring, security governance, and the unglamorous but essential work of keeping production infrastructure honest.

    What Happened

    hermes-mgmt — Hardening the Core

    The hermes-mgmt repository remained the busiest node in the network, with 20-plus commits, 20 PRs, and 13 issues. The dominant theme was memory system reliability. A significant fix (PR covering issues #573, #574, #575) hardened dual_memory.py across three dimensions: Ollama-first Mem0 configuration, deterministic Letta archival behaviour, and Qdrant vector dimension alignment. These are the kinds of fixes that don’t make headlines but prevent the subtle data corruption that erodes trust in AI systems over time.

    A related fix (issue #411) corrected a memory drift check that was comparing the wrong Letta data — core-memory passages instead of ARCHIVAL passages. This is a telling detail: as memory architectures grow more layered (core, archival, vector), the surface area for misaligned reads increases. Catching this before it caused silent data degradation matters.

    Health probes got a major upgrade with PR #568, which introduced functional health checks for services that report as “green” but are actually broken — Letta, n8n, Langfuse, Qdrant, Gateway, and Ollama. This is a pattern anyone running distributed systems will recognise: the dashboard shows all green, but something is quietly failing. Functional probes go beyond “is the process running?” to “does the service actually respond correctly?” — a distinction that matters enormously in production.

    Secrets management continued to mature. PR #552 delivered a comprehensive secrets management architecture built around Bitwarden and HashiCorp Vault, while PR #527 removed committed default secrets from the Langfuse compose configuration. These are the foundational security practices that need to be in place before scale makes them painful to retrofit.

    On the cron and scheduling side, the system now runs 38 cron jobs with 36 healthy — a 95% health rate that reflects the cumulative effect of weeks of hardening work. PR #504 corrected cron exit-code semantics so that findings and alerts are no longer misinterpreted as failures, which was causing unnecessary noise in the monitoring pipeline.

    Several open issues point to the next layer of work: Telegram flood-control protection (#583), gateway ungraceful shutdown (#582), Ollama-agent sequential dispatch performance (#580), and a Langfuse Cost Report cron that’s been disabled due to stale API keys (#576). None of these are emergencies, but they represent the known gaps that get addressed in order of priority.

    hamnet — Infrastructure Truth

    The hamnet repository, which manages infrastructure automation, saw 9 commits and 10 PRs with a focus on hosting hardening and monitoring.

    SSH hardening was codified for the VPS fleet (PR #131), including a dynamic-IP allowlist failsafe — a practical safeguard for environments where IP addresses shift. Vhost routing was hardened after www.theitrevolution.co.uk was found serving the wrong site, a class of misconfiguration that can be difficult to spot without deliberate verification.

    On the monitoring side, an Alertmanager webhook receiver was added with X-Webhook-Token authentication (PR #87), and a new Mac AI Server dashboard was built for the Ollama era (PR #134). The dashboard work reflects a broader shift: as the AI infrastructure stack evolves (Ollama replacing previous model servers), the monitoring layer needs to evolve in parallel or it becomes a liability rather than an asset.

    An open issue (#95) flags an SSL certificate SAN mismatch for www.richardham.co.uk — the kind of thing that works until it doesn’t, usually at the worst possible moment.

    hermes-agent — Gateway Stability

    The hermes-agent repository received 10 commits focused on gateway reliability. The command-line matcher was hardened, Windows restart no longer causes a silent outage, and the gateway now refreshes its cached agent max_iterations from the current config rather than stale values. A fix for dict choice unwrapping in the clarify function rounds out a set of small but meaningful stability improvements.

    These are the fixes that users never notice — because they prevent the failures that would have been noticed. Silent outages and stale config caches are particularly insidious because they can persist for extended periods before manifesting as user-visible problems.

    hermes-securescore — Evidence Collection

    The SecureScore project advanced with 9 commits and 2 PRs, adding both a Docker collector and a Hermes config collector for security evidence. A high-risk action approval playbook was also added, formalising the governance process for sensitive operations. The project recorded its live activation this week — a milestone that moves it from development into operational use.

    hermes-voice-satellite — Laying Groundwork

    The voice satellite project saw 9 commits establishing the MVP build plan, an operations runbook, a Hermes voice satellite API contract, and a Termux bootstrap script for Android (S24). A native Android app placeholder was also added. This is early-stage infrastructure work — the kind of foundation that needs to exist before feature development can accelerate.

    project-atlas-foundation — Launch Readiness

    Project Atlas received 10 commits focused on governance and launch preparation: a PR template, MIT licence, security policy, CODEOWNERS, a safety checklist, research workflow, promotion process, CI configuration, and hardened lifecycle guides. Scoring of proposals #3-#7 was completed alongside shaping docs and a launch checklist update. This is the organisational scaffolding that turns a codebase into a project other people can contribute to.

    AgentRadar — New Arrival

    AgentRadar received its initial commit this week. Details are sparse, but a new repository appearing in the portfolio is always worth noting — it represents a new thread that will either find its place or be retired honestly.

    richardham-web-and-Brand — Content Pipeline

    The web-and-brand repository saw 10 commits and 5 PRs, primarily focused on blog publishing and content pipeline maintenance. Four new blog posts in the agentic AI series were added with proper date spacing, and a build fix declared window.__calComLoaded to resolve a strict type-check failure. A WordPress publishing blocker was documented (VPS SSH key issue), which is the kind of honest infrastructure transparency that keeps content pipelines reliable.

    Key Takeaways

    Memory architecture is the new frontier. The volume of work on dual-memory hardening, Letta archival alignment, and Qdrant dimension matching signals that the memory subsystem has become a first-class concern. As AI agents handle longer and more complex tasks, the reliability of their memory layer directly determines the reliability of everything built on top of it. The drift-check fix — comparing the right data — is a small change with outsized implications.

    Functional health probes close a critical gap. The distinction between “process is running” and “service is working” is one of the most common blind spots in monitoring. Adding functional probes for six core services moves the observability stack from surface-level to meaningful. This is the kind of investment that pays for itself the first time it catches a green-but-broken service before a user does.

    Security governance is becoming systematic, not reactive. Between the secrets management architecture, the SecureScore evidence collectors, the high-risk action approval playbook, and the removal of committed defaults, the pattern is clear: security is being built into the development process rather than bolted on after incidents. This is the maturation path every platform needs to follow.

    Infrastructure truth matters. The hamnet work — SSH hardening, vhost routing fixes, SSL certificate monitoring, dashboard reconciliation — is the unglamorous foundation that everything else depends on. When www.theitrevolution.co.uk serves the wrong site, no amount of AI sophistication compensates. Keeping the infrastructure layer honest is a continuous discipline, not a one-time project.

    The content pipeline is converging with the platform. The parallel work on blog publishing, brand positioning, and the agentic AI content series is not separate from the technical work — it is how the technical work becomes visible and valuable. A hardened platform with strong governance needs an equally strong narrative around it.

    Looking Ahead

    The open issues across the portfolio paint a clear picture of next week’s priorities: resolve the Telegram flood-control design, address the gateway ungraceful shutdown path, fix the Langfuse Cost Report cron’s stale API keys, and close the SSL certificate SAN mismatch. On the infrastructure side, the Mac AI Server Ollama dashboard and metrics pipeline need to land, and the remaining open PRs in the web-and-brand repository need to progress through review.

    The velocity of the last week is notable not for its drama but for its consistency. Every repository moved forward. No single event dominated. That is what a healthy development portfolio looks like in practice — not a single sprint, but sustained, disciplined progress across every layer of the stack.

  • What Actually Makes an AI Stack Coherent

    t

    A lot of people describe an AI stack by listing components.

    Model router. Agent framework. automation layer. memory system. observability tool. evaluation harness. MCP server. Dashboard. Scheduler. Local inference. Cloud fallback.

    That list can sound impressive, but it does not tell you whether the system is coherent. It only tells you what has been installed.

    In practice, coherence comes from something more demanding: each component needs a clear job, the boundaries between them need to make sense, and the whole stack needs to be operable by someone who did not build it from memory.

    That is the difference between an AI stack and an AI estate.

    The real problem is integration, not acquisition

    Most modern AI components are fairly easy to stand up in isolation. The hard part begins when they have to behave as one system.

    That is where questions start mattering:

    • which layer actually owns routing?
    • where does workflow state live?
    • what should be remembered and for how long?
    • which layer enforces approval or review?
    • where do traces go when something fails?
    • how does an operator explain the end-to-end behaviour afterward?

    If the answer to those questions is fuzzy, the stack may still look sophisticated from a distance. But it will behave like a collection of parts rather than an operating model.

    The components are less important than their roles

    A coherent stack usually has a few recurring functions, even if the exact tools differ.

    1. Orchestration

    Something needs to decide how work is delegated, sequenced, and surfaced back to the operator. That can be an agent gateway, a workflow engine, or a mix of the two. What matters is that the control plane is understandable.

    2. Automation

    Event-driven and scheduled work needs a predictable home. If webhooks, cron tasks, notifications, and system glue are scattered across ad hoc scripts, the stack becomes harder to reason about very quickly.

    3. Memory

    Useful AI systems usually need both immediate context and some form of longer-lived recall. The difficult part is not just adding memory. It is deciding what deserves to persist, what should stay local, and what should never be carried forward automatically.

    4. Model routing

    Without an explicit routing layer, cost and trust boundaries tend to drift. Cheap models get used where better judgement was required, or expensive models get wasted on routine work. A coherent stack makes that decision visible and deliberate.

    5. Observability and evaluation

    If you cannot inspect what happened, replay the path, and compare changes over time, the stack becomes increasingly hard to trust. This is especially true once several tools and providers sit in the same chain.

    What coherence looks like operationally

    The practical signs are usually boring in the best possible way.

    A coherent system lets an operator answer simple questions quickly:

    • what happened?
    • why did it happen that way?
    • which component made the decision?
    • what data or memory influenced it?
    • what changed compared with the previous run?
    • how do we recover if the preferred path is unavailable?

    If those answers require guesswork, the stack is still immature no matter how many components it contains.

    The biggest design mistake

    The most common mistake I see is confusing capability expansion with architectural progress.

    Adding a new framework, model, or tool can increase capability. But it also adds operational cost. More boundaries. More failure modes. More decisions about ownership. More ambiguity if the roles are not explicit.

    That is why I think the most important design question is not “what else can we add?” It is “what problem does this layer uniquely solve, and what would break if it were removed?”

    If the answer is vague, the component is probably decorative.

    Why this matters for serious AI work

    Once AI systems move beyond experimentation, they start inheriting the obligations of any production environment:

    • predictable behaviour
    • clear trust boundaries
    • controlled cost
    • useful monitoring
    • recovery paths that do not depend on one person’s memory
    • documentation that survives handover

    That is why coherence matters. It is what turns a stack from a demo environment into something a business can depend on.

    A simpler test

    If I had to reduce the whole topic to one test, it would be this:

    Can you explain your AI stack in terms of operating responsibilities rather than product names?

    If you can, there is a good chance the architecture is maturing.

    If you cannot, the problem is probably not that you need one more tool. It is that the current layers have not been given clear enough jobs yet.

    That is where most of the real architecture work lives.

    And it is usually far more valuable than adding another box to the diagram.

    If you are trying to make an AI stack coherent enough to run like real infrastructure rather than a pile of experiments, the AI & Automation Architecture work is designed around exactly that problem. Or get in touch if you want a practical review of the roles, boundaries, and operating model in your current setup.

  • What Multi-Agent Operations Actually Look Like in Practice

    What Multi-Agent Operations Actually Look Like in Practice

    Most organisations experimenting with AI agents are still operating them like a single chat window. Someone opens a prompt, asks the agent to do something, waits for the output, and moves on. That works for demos. It does not work when you are running agents against production systems or trying to get consistent results across a team.

    The gap is not technical. It is operational. The organisations getting genuine value from AI agents are not the ones with the most advanced models. They are the ones that figured out how to coordinate agents the way you would coordinate a team: clear roles, defined handoffs, checkpoint reviews, and someone accountable for the outcome.

    The Governance Gap Nobody Talks About

    The current wave of AI agent tooling is impressive. You can spin up an agent that writes code, another that reviews it, another that runs tests, and a fourth that deploys. The demos are compelling. The problem is that most organisations have not thought about what happens when these agents operate together at scale.

    Who coordinates them? What happens when two agents make conflicting changes? Where is the state stored, and who can inspect it? If an agent fails halfway through a task, what recovers? If an agent produces an incorrect output that another agent consumes, how do you trace the error back?

    These are the same questions you would ask about any multi-person production system. The difference is that agents do not have common sense, do not ask clarifying questions by default, and do not stop when something looks wrong unless you have built in the checks.

    The governance gap is this: most teams have moved from “can we run an agent?” to “we are running agents” without establishing the coordination layer in between.

    The Pattern That Actually Works

    After running autonomous coding agents in production for several months, the pattern that has proven reliable is a hybrid orchestration model. It has four parts.

    A coordinator role. One agent, or one human, owns the overall task. This role does not do the detailed work. It defines the objective, breaks it into independent subtasks, assigns each to a worker, and reviews the results. In practice, this is the role I occupy when running Hermes Agent, Claude Code, or Codex on a project. I set the direction, handle security decisions and state management, and delegate the pure coding work.

    Parallel worker agents. When subtasks are independent, they run simultaneously. Three agents working on three separate services at the same time complete in minutes what a single agent would handle sequentially in an hour. The key requirement is that the subtasks must be genuinely independent. If agent B depends on agent A’s output, running them in parallel creates conflicts, not speed.

    State machines for complex flows. When a task has sequential dependencies, a simple state machine prevents chaos. Each agent picks up the task at a defined state, does its work, writes output to a known location, and transitions the task forward. If an agent fails, the state does not advance. The next agent picks up the failed state and either retries or escalates.

    Checkpoint reviews. At defined points in the flow, a human reviews the output before the next stage begins. This is not a bottleneck. It is a safety mechanism. The review confirms that the output is sane, the state is correct, and the next stage has what it needs. In practice, these reviews take seconds when things are going well and save hours when they are not.

    A Concrete Example: Diagnosing Three Services at Once

    Suppose three independent services are exhibiting issues simultaneously. A traditional approach investigates them sequentially: diagnose service A, fix it, move to service B, fix it, move to service C.

    With a multi-agent setup, the coordinator defines the diagnostic task for each service and spins up three parallel subagents. Each agent gets the same instructions: examine the logs, identify the root cause, propose a fix, and write its findings to a shared state file. The agents do not communicate with each other. They do not need to. They are working on independent systems.

    When all three agents have completed their tasks, the coordinator reviews the findings, checks for conflicts (two agents proposing changes to a shared dependency, for example), and either approves the fixes or escalates for human review.

    A diagnostic process that would take a single engineer most of a day takes under thirty minutes. The quality is not lower — each agent focuses on a single problem without context-switching. The risk is not higher — the checkpoint review catches anything anomalous before it reaches production.

    This is not theoretical. It is a routine operational pattern that runs on free-tier models for the worker agents. The expensive model is the coordinator, and even that role can be handled by a human with a clear framework.

    The Cost Conversation

    There is a persistent misconception that running AI agents at scale requires expensive API subscriptions. In practice, the opposite is true. Worker agents doing diagnostics, code generation, and testing do not need frontier models. They need competent instruction-following, and that is available on free tiers or at very low cost.

    The coordinator role is where model quality matters. This is the agent making decisions about task decomposition, conflict resolution, and escalation. It needs to reason well. But there is only one coordinator, and it does relatively little token-heavy work compared to the workers.

    The cost structure in a well-designed multi-agent system is front-loaded into the coordination layer and minimal in the execution layer. You are paying for one good decision-maker and many cheap workers. The economics favour this model, which is one reason it works for cost-conscious organisations, not just well-funded ones.

    Failure recovery follows the same logic. When an agent fails on a free tier, the cost of retry is zero. When an agent fails on an expensive tier, every retry is a budget event. Putting cheap agents on high-volume work and the expensive agent on high-judgement work is not just an architectural decision. It is a cost optimisation.

    What Organisations Should Do Next

    If you are running or planning to run AI agents in production, the operational model matters more than model selection. Here is where to start.

    Define the coordinator role first. Decide whether a human or an agent owns task decomposition and review. Document what this role is responsible for and what decisions require escalation. This is your governance layer.

    Identify independent subtasks. Look at your current agent workflows and find the tasks that can run in parallel. Sequential workflows where tasks are independent are leaving time on the table.

    Build state into your workflows. Every agent should write its output to a known location in a known format. Every downstream agent should read from that location. If you cannot inspect workflow state at any point without replaying the entire execution, your state management is insufficient.

    Set checkpoint reviews at decision points. Not at every step — that defeats the purpose. At points where an incorrect output would propagate downstream and cause real damage. A review that takes five seconds and prevents a two-hour debugging session is time well spent.

    Use the right model for the right role. Do not pay frontier-model prices for tasks that a free-tier model handles competently. Reserve your budget for the coordination and review layers where reasoning quality directly affects outcomes.

    If your organisation is moving from AI experimentation to production agent operations, the coordination layer is where the value is — and where the risk lives. The AI & Automation Architecture service covers the design of multi-agent systems with proper governance, state management, and cost controls. Or get in touch for a conversation about what your agent operations should look like before they scale.

  • How Multi-Agent Operations Work in Practice

    How Multi-Agent Operations Work in Practice

    Most organisations still treat AI agents like a single chat window with extra buttons. That is fine for a demo. It is not fine when the work touches production systems.

    The difference is operational, not magical. The teams getting value from agents are the ones that add roles, checkpoints, and ownership.

    What works

    A reliable setup usually has four pieces:

    • a coordinator that defines the task and checks the output
    • worker agents that do independent chunks in parallel
    • state that lives somewhere everyone can inspect
    • review points before anything risky moves forward

    That is not glamorous, but it works.

    A simple example

    If three services are acting up at once, a good coordinator breaks the problem apart and sends each service to a separate worker. The workers do not need to talk to each other because the tasks are independent. The coordinator then compares the results and decides whether to approve the fix or escalate.

    That pattern saves time without turning the system into a black box.

    The cost question

    People often assume agent work must be expensive. In practice, the opposite is usually true. The cheap model can do the repetitive work. The better reasoning model is reserved for coordination and review.

    That split matters. It keeps the system affordable and keeps judgment where it belongs.

    What to do first

    1. Decide who owns the outcome.
    2. Identify which tasks can run in parallel.
    3. Make state visible.
    4. Add checkpoints where mistakes would hurt.

    That is enough to get started. The rest is tuning.

  • What Multi-Agent Operations Actually Look Like in Practice

    What Multi-Agent Operations Actually Look Like in Practice

    Most organisations experimenting with AI agents are still operating them like a single chat window. Someone opens a prompt, asks the agent to do something, waits for the output, and moves on. That works for demos. It does not work when you are running agents against production systems or trying to get consistent results across a team.

    The gap is not technical. It is operational. The organisations getting genuine value from AI agents are not the ones with the most advanced models. They are the ones that figured out how to coordinate agents the way you would coordinate a team: clear roles, defined handoffs, checkpoint reviews, and someone accountable for the outcome.

    The Governance Gap Nobody Talks About

    The current wave of AI agent tooling is impressive. You can spin up an agent that writes code, another that reviews it, another that runs tests, and a fourth that deploys. The demos are compelling. The problem is that most organisations have not thought about what happens when these agents operate together at scale.

    Who coordinates them? What happens when two agents make conflicting changes? Where is the state stored, and who can inspect it? If an agent fails halfway through a task, what recovers? If an agent produces an incorrect output that another agent consumes, how do you trace the error back?

    These are the same questions you would ask about any multi-person production system. The difference is that agents do not have common sense, do not ask clarifying questions by default, and do not stop when something looks wrong unless you have built in the checks.

    The governance gap is this: most teams have moved from “can we run an agent?” to “we are running agents” without establishing the coordination layer in between.

    The Pattern That Actually Works

    After running autonomous coding agents in production for several months, the pattern that has proven reliable is a hybrid orchestration model. It has four parts.

    A coordinator role. One agent, or one human, owns the overall task. This role does not do the detailed work. It defines the objective, breaks it into independent subtasks, assigns each to a worker, and reviews the results. In practice, this is the role I occupy when running Hermes Agent, Claude Code, or Codex on a project. I set the direction, handle security decisions and state management, and delegate the pure coding work.

    Parallel worker agents. When subtasks are independent, they run simultaneously. Three agents working on three separate services at the same time complete in minutes what a single agent would handle sequentially in an hour. The key requirement is that the subtasks must be genuinely independent. If agent B depends on agent A’s output, running them in parallel creates conflicts, not speed.

    State machines for complex flows. When a task has sequential dependencies, a simple state machine prevents chaos. Each agent picks up the task at a defined state, does its work, writes output to a known location, and transitions the task forward. If an agent fails, the state does not advance. The next agent picks up the failed state and either retries or escalates.

    Checkpoint reviews. At defined points in the flow, a human reviews the output before the next stage begins. This is not a bottleneck. It is a safety mechanism. The review confirms that the output is sane, the state is correct, and the next stage has what it needs. In practice, these reviews take seconds when things are going well and save hours when they are not.

    A Concrete Example: Diagnosing Three Services at Once

    Suppose three independent services are exhibiting issues simultaneously. A traditional approach investigates them sequentially: diagnose service A, fix it, move to service B, fix it, move to service C.

    With a multi-agent setup, the coordinator defines the diagnostic task for each service and spins up three parallel subagents. Each agent gets the same instructions: examine the logs, identify the root cause, propose a fix, and write its findings to a shared state file. The agents do not communicate with each other. They do not need to. They are working on independent systems.

    When all three agents have completed their tasks, the coordinator reviews the findings, checks for conflicts (two agents proposing changes to a shared dependency, for example), and either approves the fixes or escalates for human review.

    A diagnostic process that would take a single engineer most of a day takes under thirty minutes. The quality is not lower — each agent focuses on a single problem without context-switching. The risk is not higher — the checkpoint review catches anything anomalous before it reaches production.

    This is not theoretical. It is a routine operational pattern that runs on free-tier models for the worker agents. The expensive model is the coordinator, and even that role can be handled by a human with a clear framework.

    The Cost Conversation

    There is a persistent misconception that running AI agents at scale requires expensive API subscriptions. In practice, the opposite is true. Worker agents doing diagnostics, code generation, and testing do not need frontier models. They need competent instruction-following, and that is available on free tiers or at very low cost.

    The coordinator role is where model quality matters. This is the agent making decisions about task decomposition, conflict resolution, and escalation. It needs to reason well. But there is only one coordinator, and it does relatively little token-heavy work compared to the workers.

    The cost structure in a well-designed multi-agent system is front-loaded into the coordination layer and minimal in the execution layer. You are paying for one good decision-maker and many cheap workers. The economics favour this model, which is one reason it works for cost-conscious organisations, not just well-funded ones.

    Failure recovery follows the same logic. When an agent fails on a free tier, the cost of retry is zero. When an agent fails on an expensive tier, every retry is a budget event. Putting cheap agents on high-volume work and the expensive agent on high-judgement work is not just an architectural decision. It is a cost optimisation.

    What Organisations Should Do Next

    If you are running or planning to run AI agents in production, the operational model matters more than model selection. Here is where to start.

    Define the coordinator role first. Decide whether a human or an agent owns task decomposition and review. Document what this role is responsible for and what decisions require escalation. This is your governance layer.

    Identify independent subtasks. Look at your current agent workflows and find the tasks that can run in parallel. Sequential workflows where tasks are independent are leaving time on the table.

    Build state into your workflows. Every agent should write its output to a known location in a known format. Every downstream agent should read from that location. If you cannot inspect workflow state at any point without replaying the entire execution, your state management is insufficient.

    Set checkpoint reviews at decision points. Not at every step — that defeats the purpose. At points where an incorrect output would propagate downstream and cause real damage. A review that takes five seconds and prevents a two-hour debugging session is time well spent.

    Use the right model for the right role. Do not pay frontier-model prices for tasks that a free-tier model handles competently. Reserve your budget for the coordination and review layers where reasoning quality directly affects outcomes.

    If your organisation is moving from AI experimentation to production agent operations, the coordination layer is where the value is — and where the risk lives. The AI & Automation Architecture service covers the design of multi-agent systems with proper governance, state management, and cost controls. Or get in touch for a conversation about what your agent operations should look like before they scale.

Powered by