RichardHam.co.uk Blog

Tag: AI

  • GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    The week of June 15-22 was not defined by a single dramatic event. Instead, it was the kind of week that separates platforms that merely work from platforms that hold up under sustained pressure. Across 50 repositories, over 100 events — commits, pull requests, issues — pushed forward several long-running threads: memory architecture, health monitoring, security governance, and the unglamorous but essential work of keeping production infrastructure honest.

    What Happened

    hermes-mgmt — Hardening the Core

    The hermes-mgmt repository remained the busiest node in the network, with 20-plus commits, 20 PRs, and 13 issues. The dominant theme was memory system reliability. A significant fix (PR covering issues #573, #574, #575) hardened dual_memory.py across three dimensions: Ollama-first Mem0 configuration, deterministic Letta archival behaviour, and Qdrant vector dimension alignment. These are the kinds of fixes that don’t make headlines but prevent the subtle data corruption that erodes trust in AI systems over time.

    A related fix (issue #411) corrected a memory drift check that was comparing the wrong Letta data — core-memory passages instead of ARCHIVAL passages. This is a telling detail: as memory architectures grow more layered (core, archival, vector), the surface area for misaligned reads increases. Catching this before it caused silent data degradation matters.

    Health probes got a major upgrade with PR #568, which introduced functional health checks for services that report as “green” but are actually broken — Letta, n8n, Langfuse, Qdrant, Gateway, and Ollama. This is a pattern anyone running distributed systems will recognise: the dashboard shows all green, but something is quietly failing. Functional probes go beyond “is the process running?” to “does the service actually respond correctly?” — a distinction that matters enormously in production.

    Secrets management continued to mature. PR #552 delivered a comprehensive secrets management architecture built around Bitwarden and HashiCorp Vault, while PR #527 removed committed default secrets from the Langfuse compose configuration. These are the foundational security practices that need to be in place before scale makes them painful to retrofit.

    On the cron and scheduling side, the system now runs 38 cron jobs with 36 healthy — a 95% health rate that reflects the cumulative effect of weeks of hardening work. PR #504 corrected cron exit-code semantics so that findings and alerts are no longer misinterpreted as failures, which was causing unnecessary noise in the monitoring pipeline.

    Several open issues point to the next layer of work: Telegram flood-control protection (#583), gateway ungraceful shutdown (#582), Ollama-agent sequential dispatch performance (#580), and a Langfuse Cost Report cron that’s been disabled due to stale API keys (#576). None of these are emergencies, but they represent the known gaps that get addressed in order of priority.

    hamnet — Infrastructure Truth

    The hamnet repository, which manages infrastructure automation, saw 9 commits and 10 PRs with a focus on hosting hardening and monitoring.

    SSH hardening was codified for the VPS fleet (PR #131), including a dynamic-IP allowlist failsafe — a practical safeguard for environments where IP addresses shift. Vhost routing was hardened after www.theitrevolution.co.uk was found serving the wrong site, a class of misconfiguration that can be difficult to spot without deliberate verification.

    On the monitoring side, an Alertmanager webhook receiver was added with X-Webhook-Token authentication (PR #87), and a new Mac AI Server dashboard was built for the Ollama era (PR #134). The dashboard work reflects a broader shift: as the AI infrastructure stack evolves (Ollama replacing previous model servers), the monitoring layer needs to evolve in parallel or it becomes a liability rather than an asset.

    An open issue (#95) flags an SSL certificate SAN mismatch for www.richardham.co.uk — the kind of thing that works until it doesn’t, usually at the worst possible moment.

    hermes-agent — Gateway Stability

    The hermes-agent repository received 10 commits focused on gateway reliability. The command-line matcher was hardened, Windows restart no longer causes a silent outage, and the gateway now refreshes its cached agent max_iterations from the current config rather than stale values. A fix for dict choice unwrapping in the clarify function rounds out a set of small but meaningful stability improvements.

    These are the fixes that users never notice — because they prevent the failures that would have been noticed. Silent outages and stale config caches are particularly insidious because they can persist for extended periods before manifesting as user-visible problems.

    hermes-securescore — Evidence Collection

    The SecureScore project advanced with 9 commits and 2 PRs, adding both a Docker collector and a Hermes config collector for security evidence. A high-risk action approval playbook was also added, formalising the governance process for sensitive operations. The project recorded its live activation this week — a milestone that moves it from development into operational use.

    hermes-voice-satellite — Laying Groundwork

    The voice satellite project saw 9 commits establishing the MVP build plan, an operations runbook, a Hermes voice satellite API contract, and a Termux bootstrap script for Android (S24). A native Android app placeholder was also added. This is early-stage infrastructure work — the kind of foundation that needs to exist before feature development can accelerate.

    project-atlas-foundation — Launch Readiness

    Project Atlas received 10 commits focused on governance and launch preparation: a PR template, MIT licence, security policy, CODEOWNERS, a safety checklist, research workflow, promotion process, CI configuration, and hardened lifecycle guides. Scoring of proposals #3-#7 was completed alongside shaping docs and a launch checklist update. This is the organisational scaffolding that turns a codebase into a project other people can contribute to.

    AgentRadar — New Arrival

    AgentRadar received its initial commit this week. Details are sparse, but a new repository appearing in the portfolio is always worth noting — it represents a new thread that will either find its place or be retired honestly.

    richardham-web-and-Brand — Content Pipeline

    The web-and-brand repository saw 10 commits and 5 PRs, primarily focused on blog publishing and content pipeline maintenance. Four new blog posts in the agentic AI series were added with proper date spacing, and a build fix declared window.__calComLoaded to resolve a strict type-check failure. A WordPress publishing blocker was documented (VPS SSH key issue), which is the kind of honest infrastructure transparency that keeps content pipelines reliable.

    Key Takeaways

    Memory architecture is the new frontier. The volume of work on dual-memory hardening, Letta archival alignment, and Qdrant dimension matching signals that the memory subsystem has become a first-class concern. As AI agents handle longer and more complex tasks, the reliability of their memory layer directly determines the reliability of everything built on top of it. The drift-check fix — comparing the right data — is a small change with outsized implications.

    Functional health probes close a critical gap. The distinction between “process is running” and “service is working” is one of the most common blind spots in monitoring. Adding functional probes for six core services moves the observability stack from surface-level to meaningful. This is the kind of investment that pays for itself the first time it catches a green-but-broken service before a user does.

    Security governance is becoming systematic, not reactive. Between the secrets management architecture, the SecureScore evidence collectors, the high-risk action approval playbook, and the removal of committed defaults, the pattern is clear: security is being built into the development process rather than bolted on after incidents. This is the maturation path every platform needs to follow.

    Infrastructure truth matters. The hamnet work — SSH hardening, vhost routing fixes, SSL certificate monitoring, dashboard reconciliation — is the unglamorous foundation that everything else depends on. When www.theitrevolution.co.uk serves the wrong site, no amount of AI sophistication compensates. Keeping the infrastructure layer honest is a continuous discipline, not a one-time project.

    The content pipeline is converging with the platform. The parallel work on blog publishing, brand positioning, and the agentic AI content series is not separate from the technical work — it is how the technical work becomes visible and valuable. A hardened platform with strong governance needs an equally strong narrative around it.

    Looking Ahead

    The open issues across the portfolio paint a clear picture of next week’s priorities: resolve the Telegram flood-control design, address the gateway ungraceful shutdown path, fix the Langfuse Cost Report cron’s stale API keys, and close the SSL certificate SAN mismatch. On the infrastructure side, the Mac AI Server Ollama dashboard and metrics pipeline need to land, and the remaining open PRs in the web-and-brand repository need to progress through review.

    The velocity of the last week is notable not for its drama but for its consistency. Every repository moved forward. No single event dominated. That is what a healthy development portfolio looks like in practice — not a single sprint, but sustained, disciplined progress across every layer of the stack.

  • GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    g

    The week of June 15-22 was not defined by a single dramatic event. Instead, it was the kind of week that separates platforms that merely work from platforms that hold up under sustained pressure. Across 50 repositories, over 100 events — commits, pull requests, issues — pushed forward several long-running threads: memory architecture, health monitoring, security governance, and the unglamorous but essential work of keeping production infrastructure honest.

    What Happened

    hermes-mgmt — Hardening the Core

    The hermes-mgmt repository remained the busiest node in the network, with 20-plus commits, 20 PRs, and 13 issues. The dominant theme was memory system reliability. A significant fix (PR covering issues #573, #574, #575) hardened dual_memory.py across three dimensions: Ollama-first Mem0 configuration, deterministic Letta archival behaviour, and Qdrant vector dimension alignment. These are the kinds of fixes that don’t make headlines but prevent the subtle data corruption that erodes trust in AI systems over time.

    A related fix (issue #411) corrected a memory drift check that was comparing the wrong Letta data — core-memory passages instead of ARCHIVAL passages. This is a telling detail: as memory architectures grow more layered (core, archival, vector), the surface area for misaligned reads increases. Catching this before it caused silent data degradation matters.

    Health probes got a major upgrade with PR #568, which introduced functional health checks for services that report as “green” but are actually broken — Letta, n8n, Langfuse, Qdrant, Gateway, and Ollama. This is a pattern anyone running distributed systems will recognise: the dashboard shows all green, but something is quietly failing. Functional probes go beyond “is the process running?” to “does the service actually respond correctly?” — a distinction that matters enormously in production.

    Secrets management continued to mature. PR #552 delivered a comprehensive secrets management architecture built around Bitwarden and HashiCorp Vault, while PR #527 removed committed default secrets from the Langfuse compose configuration. These are the foundational security practices that need to be in place before scale makes them painful to retrofit.

    On the cron and scheduling side, the system now runs 38 cron jobs with 36 healthy — a 95% health rate that reflects the cumulative effect of weeks of hardening work. PR #504 corrected cron exit-code semantics so that findings and alerts are no longer misinterpreted as failures, which was causing unnecessary noise in the monitoring pipeline.

    Several open issues point to the next layer of work: Telegram flood-control protection (#583), gateway ungraceful shutdown (#582), Ollama-agent sequential dispatch performance (#580), and a Langfuse Cost Report cron that’s been disabled due to stale API keys (#576). None of these are emergencies, but they represent the known gaps that get addressed in order of priority.

    hamnet — Infrastructure Truth

    The hamnet repository, which manages infrastructure automation, saw 9 commits and 10 PRs with a focus on hosting hardening and monitoring.

    SSH hardening was codified for the VPS fleet (PR #131), including a dynamic-IP allowlist failsafe — a practical safeguard for environments where IP addresses shift. Vhost routing was hardened after www.theitrevolution.co.uk was found serving the wrong site, a class of misconfiguration that can be difficult to spot without deliberate verification.

    On the monitoring side, an Alertmanager webhook receiver was added with X-Webhook-Token authentication (PR #87), and a new Mac AI Server dashboard was built for the Ollama era (PR #134). The dashboard work reflects a broader shift: as the AI infrastructure stack evolves (Ollama replacing previous model servers), the monitoring layer needs to evolve in parallel or it becomes a liability rather than an asset.

    An open issue (#95) flags an SSL certificate SAN mismatch for www.richardham.co.uk — the kind of thing that works until it doesn’t, usually at the worst possible moment.

    hermes-agent — Gateway Stability

    The hermes-agent repository received 10 commits focused on gateway reliability. The command-line matcher was hardened, Windows restart no longer causes a silent outage, and the gateway now refreshes its cached agent max_iterations from the current config rather than stale values. A fix for dict choice unwrapping in the clarify function rounds out a set of small but meaningful stability improvements.

    These are the fixes that users never notice — because they prevent the failures that would have been noticed. Silent outages and stale config caches are particularly insidious because they can persist for extended periods before manifesting as user-visible problems.

    hermes-securescore — Evidence Collection

    The SecureScore project advanced with 9 commits and 2 PRs, adding both a Docker collector and a Hermes config collector for security evidence. A high-risk action approval playbook was also added, formalising the governance process for sensitive operations. The project recorded its live activation this week — a milestone that moves it from development into operational use.

    hermes-voice-satellite — Laying Groundwork

    The voice satellite project saw 9 commits establishing the MVP build plan, an operations runbook, a Hermes voice satellite API contract, and a Termux bootstrap script for Android (S24). A native Android app placeholder was also added. This is early-stage infrastructure work — the kind of foundation that needs to exist before feature development can accelerate.

    project-atlas-foundation — Launch Readiness

    Project Atlas received 10 commits focused on governance and launch preparation: a PR template, MIT licence, security policy, CODEOWNERS, a safety checklist, research workflow, promotion process, CI configuration, and hardened lifecycle guides. Scoring of proposals #3-#7 was completed alongside shaping docs and a launch checklist update. This is the organisational scaffolding that turns a codebase into a project other people can contribute to.

    AgentRadar — New Arrival

    AgentRadar received its initial commit this week. Details are sparse, but a new repository appearing in the portfolio is always worth noting — it represents a new thread that will either find its place or be retired honestly.

    richardham-web-and-Brand — Content Pipeline

    The web-and-brand repository saw 10 commits and 5 PRs, primarily focused on blog publishing and content pipeline maintenance. Four new blog posts in the agentic AI series were added with proper date spacing, and a build fix declared window.__calComLoaded to resolve a strict type-check failure. A WordPress publishing blocker was documented (VPS SSH key issue), which is the kind of honest infrastructure transparency that keeps content pipelines reliable.

    Key Takeaways

    Memory architecture is the new frontier. The volume of work on dual-memory hardening, Letta archival alignment, and Qdrant dimension matching signals that the memory subsystem has become a first-class concern. As AI agents handle longer and more complex tasks, the reliability of their memory layer directly determines the reliability of everything built on top of it. The drift-check fix — comparing the right data — is a small change with outsized implications.

    Functional health probes close a critical gap. The distinction between “process is running” and “service is working” is one of the most common blind spots in monitoring. Adding functional probes for six core services moves the observability stack from surface-level to meaningful. This is the kind of investment that pays for itself the first time it catches a green-but-broken service before a user does.

    Security governance is becoming systematic, not reactive. Between the secrets management architecture, the SecureScore evidence collectors, the high-risk action approval playbook, and the removal of committed defaults, the pattern is clear: security is being built into the development process rather than bolted on after incidents. This is the maturation path every platform needs to follow.

    Infrastructure truth matters. The hamnet work — SSH hardening, vhost routing fixes, SSL certificate monitoring, dashboard reconciliation — is the unglamorous foundation that everything else depends on. When www.theitrevolution.co.uk serves the wrong site, no amount of AI sophistication compensates. Keeping the infrastructure layer honest is a continuous discipline, not a one-time project.

    The content pipeline is converging with the platform. The parallel work on blog publishing, brand positioning, and the agentic AI content series is not separate from the technical work — it is how the technical work becomes visible and valuable. A hardened platform with strong governance needs an equally strong narrative around it.

    Looking Ahead

    The open issues across the portfolio paint a clear picture of next week’s priorities: resolve the Telegram flood-control design, address the gateway ungraceful shutdown path, fix the Langfuse Cost Report cron’s stale API keys, and close the SSL certificate SAN mismatch. On the infrastructure side, the Mac AI Server Ollama dashboard and metrics pipeline need to land, and the remaining open PRs in the web-and-brand repository need to progress through review.

    The velocity of the last week is notable not for its drama but for its consistency. Every repository moved forward. No single event dominated. That is what a healthy development portfolio looks like in practice — not a single sprint, but sustained, disciplined progress across every layer of the stack.

  • What Multi-Agent Operations Actually Look Like in Practice

    What Multi-Agent Operations Actually Look Like in Practice

    Most organisations experimenting with AI agents are still operating them like a single chat window. Someone opens a prompt, asks the agent to do something, waits for the output, and moves on. That works for demos. It does not work when you are running agents against production systems or trying to get consistent results across a team.

    The gap is not technical. It is operational. The organisations getting genuine value from AI agents are not the ones with the most advanced models. They are the ones that figured out how to coordinate agents the way you would coordinate a team: clear roles, defined handoffs, checkpoint reviews, and someone accountable for the outcome.

    The Governance Gap Nobody Talks About

    The current wave of AI agent tooling is impressive. You can spin up an agent that writes code, another that reviews it, another that runs tests, and a fourth that deploys. The demos are compelling. The problem is that most organisations have not thought about what happens when these agents operate together at scale.

    Who coordinates them? What happens when two agents make conflicting changes? Where is the state stored, and who can inspect it? If an agent fails halfway through a task, what recovers? If an agent produces an incorrect output that another agent consumes, how do you trace the error back?

    These are the same questions you would ask about any multi-person production system. The difference is that agents do not have common sense, do not ask clarifying questions by default, and do not stop when something looks wrong unless you have built in the checks.

    The governance gap is this: most teams have moved from “can we run an agent?” to “we are running agents” without establishing the coordination layer in between.

    The Pattern That Actually Works

    After running autonomous coding agents in production for several months, the pattern that has proven reliable is a hybrid orchestration model. It has four parts.

    A coordinator role. One agent, or one human, owns the overall task. This role does not do the detailed work. It defines the objective, breaks it into independent subtasks, assigns each to a worker, and reviews the results. In practice, this is the role I occupy when running Hermes Agent, Claude Code, or Codex on a project. I set the direction, handle security decisions and state management, and delegate the pure coding work.

    Parallel worker agents. When subtasks are independent, they run simultaneously. Three agents working on three separate services at the same time complete in minutes what a single agent would handle sequentially in an hour. The key requirement is that the subtasks must be genuinely independent. If agent B depends on agent A’s output, running them in parallel creates conflicts, not speed.

    State machines for complex flows. When a task has sequential dependencies, a simple state machine prevents chaos. Each agent picks up the task at a defined state, does its work, writes output to a known location, and transitions the task forward. If an agent fails, the state does not advance. The next agent picks up the failed state and either retries or escalates.

    Checkpoint reviews. At defined points in the flow, a human reviews the output before the next stage begins. This is not a bottleneck. It is a safety mechanism. The review confirms that the output is sane, the state is correct, and the next stage has what it needs. In practice, these reviews take seconds when things are going well and save hours when they are not.

    A Concrete Example: Diagnosing Three Services at Once

    Suppose three independent services are exhibiting issues simultaneously. A traditional approach investigates them sequentially: diagnose service A, fix it, move to service B, fix it, move to service C.

    With a multi-agent setup, the coordinator defines the diagnostic task for each service and spins up three parallel subagents. Each agent gets the same instructions: examine the logs, identify the root cause, propose a fix, and write its findings to a shared state file. The agents do not communicate with each other. They do not need to. They are working on independent systems.

    When all three agents have completed their tasks, the coordinator reviews the findings, checks for conflicts (two agents proposing changes to a shared dependency, for example), and either approves the fixes or escalates for human review.

    A diagnostic process that would take a single engineer most of a day takes under thirty minutes. The quality is not lower — each agent focuses on a single problem without context-switching. The risk is not higher — the checkpoint review catches anything anomalous before it reaches production.

    This is not theoretical. It is a routine operational pattern that runs on free-tier models for the worker agents. The expensive model is the coordinator, and even that role can be handled by a human with a clear framework.

    The Cost Conversation

    There is a persistent misconception that running AI agents at scale requires expensive API subscriptions. In practice, the opposite is true. Worker agents doing diagnostics, code generation, and testing do not need frontier models. They need competent instruction-following, and that is available on free tiers or at very low cost.

    The coordinator role is where model quality matters. This is the agent making decisions about task decomposition, conflict resolution, and escalation. It needs to reason well. But there is only one coordinator, and it does relatively little token-heavy work compared to the workers.

    The cost structure in a well-designed multi-agent system is front-loaded into the coordination layer and minimal in the execution layer. You are paying for one good decision-maker and many cheap workers. The economics favour this model, which is one reason it works for cost-conscious organisations, not just well-funded ones.

    Failure recovery follows the same logic. When an agent fails on a free tier, the cost of retry is zero. When an agent fails on an expensive tier, every retry is a budget event. Putting cheap agents on high-volume work and the expensive agent on high-judgement work is not just an architectural decision. It is a cost optimisation.

    What Organisations Should Do Next

    If you are running or planning to run AI agents in production, the operational model matters more than model selection. Here is where to start.

    Define the coordinator role first. Decide whether a human or an agent owns task decomposition and review. Document what this role is responsible for and what decisions require escalation. This is your governance layer.

    Identify independent subtasks. Look at your current agent workflows and find the tasks that can run in parallel. Sequential workflows where tasks are independent are leaving time on the table.

    Build state into your workflows. Every agent should write its output to a known location in a known format. Every downstream agent should read from that location. If you cannot inspect workflow state at any point without replaying the entire execution, your state management is insufficient.

    Set checkpoint reviews at decision points. Not at every step — that defeats the purpose. At points where an incorrect output would propagate downstream and cause real damage. A review that takes five seconds and prevents a two-hour debugging session is time well spent.

    Use the right model for the right role. Do not pay frontier-model prices for tasks that a free-tier model handles competently. Reserve your budget for the coordination and review layers where reasoning quality directly affects outcomes.

    If your organisation is moving from AI experimentation to production agent operations, the coordination layer is where the value is — and where the risk lives. The AI & Automation Architecture service covers the design of multi-agent systems with proper governance, state management, and cost controls. Or get in touch for a conversation about what your agent operations should look like before they scale.

  • What Multi-Agent Operations Actually Look Like in Practice

    What Multi-Agent Operations Actually Look Like in Practice

    Most organisations experimenting with AI agents are still operating them like a single chat window. Someone opens a prompt, asks the agent to do something, waits for the output, and moves on. That works for demos. It does not work when you are running agents against production systems or trying to get consistent results across a team.

    The gap is not technical. It is operational. The organisations getting genuine value from AI agents are not the ones with the most advanced models. They are the ones that figured out how to coordinate agents the way you would coordinate a team: clear roles, defined handoffs, checkpoint reviews, and someone accountable for the outcome.

    The Governance Gap Nobody Talks About

    The current wave of AI agent tooling is impressive. You can spin up an agent that writes code, another that reviews it, another that runs tests, and a fourth that deploys. The demos are compelling. The problem is that most organisations have not thought about what happens when these agents operate together at scale.

    Who coordinates them? What happens when two agents make conflicting changes? Where is the state stored, and who can inspect it? If an agent fails halfway through a task, what recovers? If an agent produces an incorrect output that another agent consumes, how do you trace the error back?

    These are the same questions you would ask about any multi-person production system. The difference is that agents do not have common sense, do not ask clarifying questions by default, and do not stop when something looks wrong unless you have built in the checks.

    The governance gap is this: most teams have moved from “can we run an agent?” to “we are running agents” without establishing the coordination layer in between.

    The Pattern That Actually Works

    After running autonomous coding agents in production for several months, the pattern that has proven reliable is a hybrid orchestration model. It has four parts.

    A coordinator role. One agent, or one human, owns the overall task. This role does not do the detailed work. It defines the objective, breaks it into independent subtasks, assigns each to a worker, and reviews the results. In practice, this is the role I occupy when running Hermes Agent, Claude Code, or Codex on a project. I set the direction, handle security decisions and state management, and delegate the pure coding work.

    Parallel worker agents. When subtasks are independent, they run simultaneously. Three agents working on three separate services at the same time complete in minutes what a single agent would handle sequentially in an hour. The key requirement is that the subtasks must be genuinely independent. If agent B depends on agent A’s output, running them in parallel creates conflicts, not speed.

    State machines for complex flows. When a task has sequential dependencies, a simple state machine prevents chaos. Each agent picks up the task at a defined state, does its work, writes output to a known location, and transitions the task forward. If an agent fails, the state does not advance. The next agent picks up the failed state and either retries or escalates.

    Checkpoint reviews. At defined points in the flow, a human reviews the output before the next stage begins. This is not a bottleneck. It is a safety mechanism. The review confirms that the output is sane, the state is correct, and the next stage has what it needs. In practice, these reviews take seconds when things are going well and save hours when they are not.

    A Concrete Example: Diagnosing Three Services at Once

    Suppose three independent services are exhibiting issues simultaneously. A traditional approach investigates them sequentially: diagnose service A, fix it, move to service B, fix it, move to service C.

    With a multi-agent setup, the coordinator defines the diagnostic task for each service and spins up three parallel subagents. Each agent gets the same instructions: examine the logs, identify the root cause, propose a fix, and write its findings to a shared state file. The agents do not communicate with each other. They do not need to. They are working on independent systems.

    When all three agents have completed their tasks, the coordinator reviews the findings, checks for conflicts (two agents proposing changes to a shared dependency, for example), and either approves the fixes or escalates for human review.

    A diagnostic process that would take a single engineer most of a day takes under thirty minutes. The quality is not lower — each agent focuses on a single problem without context-switching. The risk is not higher — the checkpoint review catches anything anomalous before it reaches production.

    This is not theoretical. It is a routine operational pattern that runs on free-tier models for the worker agents. The expensive model is the coordinator, and even that role can be handled by a human with a clear framework.

    The Cost Conversation

    There is a persistent misconception that running AI agents at scale requires expensive API subscriptions. In practice, the opposite is true. Worker agents doing diagnostics, code generation, and testing do not need frontier models. They need competent instruction-following, and that is available on free tiers or at very low cost.

    The coordinator role is where model quality matters. This is the agent making decisions about task decomposition, conflict resolution, and escalation. It needs to reason well. But there is only one coordinator, and it does relatively little token-heavy work compared to the workers.

    The cost structure in a well-designed multi-agent system is front-loaded into the coordination layer and minimal in the execution layer. You are paying for one good decision-maker and many cheap workers. The economics favour this model, which is one reason it works for cost-conscious organisations, not just well-funded ones.

    Failure recovery follows the same logic. When an agent fails on a free tier, the cost of retry is zero. When an agent fails on an expensive tier, every retry is a budget event. Putting cheap agents on high-volume work and the expensive agent on high-judgement work is not just an architectural decision. It is a cost optimisation.

    What Organisations Should Do Next

    If you are running or planning to run AI agents in production, the operational model matters more than model selection. Here is where to start.

    Define the coordinator role first. Decide whether a human or an agent owns task decomposition and review. Document what this role is responsible for and what decisions require escalation. This is your governance layer.

    Identify independent subtasks. Look at your current agent workflows and find the tasks that can run in parallel. Sequential workflows where tasks are independent are leaving time on the table.

    Build state into your workflows. Every agent should write its output to a known location in a known format. Every downstream agent should read from that location. If you cannot inspect workflow state at any point without replaying the entire execution, your state management is insufficient.

    Set checkpoint reviews at decision points. Not at every step — that defeats the purpose. At points where an incorrect output would propagate downstream and cause real damage. A review that takes five seconds and prevents a two-hour debugging session is time well spent.

    Use the right model for the right role. Do not pay frontier-model prices for tasks that a free-tier model handles competently. Reserve your budget for the coordination and review layers where reasoning quality directly affects outcomes.

    If your organisation is moving from AI experimentation to production agent operations, the coordination layer is where the value is — and where the risk lives. The AI & Automation Architecture service covers the design of multi-agent systems with proper governance, state management, and cost controls. Or get in touch for a conversation about what your agent operations should look like before they scale.

  • How Multi-Agent Operations Work in Practice

    How Multi-Agent Operations Work in Practice

    Most organisations still treat AI agents like a single chat window with extra buttons. That is fine for a demo. It is not fine when the work touches production systems.

    The difference is operational, not magical. The teams getting value from agents are the ones that add roles, checkpoints, and ownership.

    What works

    A reliable setup usually has four pieces:

    • a coordinator that defines the task and checks the output
    • worker agents that do independent chunks in parallel
    • state that lives somewhere everyone can inspect
    • review points before anything risky moves forward

    That is not glamorous, but it works.

    A simple example

    If three services are acting up at once, a good coordinator breaks the problem apart and sends each service to a separate worker. The workers do not need to talk to each other because the tasks are independent. The coordinator then compares the results and decides whether to approve the fix or escalate.

    That pattern saves time without turning the system into a black box.

    The cost question

    People often assume agent work must be expensive. In practice, the opposite is usually true. The cheap model can do the repetitive work. The better reasoning model is reserved for coordination and review.

    That split matters. It keeps the system affordable and keeps judgment where it belongs.

    What to do first

    1. Decide who owns the outcome.
    2. Identify which tasks can run in parallel.
    3. Make state visible.
    4. Add checkpoints where mistakes would hurt.

    That is enough to get started. The rest is tuning.

  • What Multi-Agent Operations Actually Look Like in Practice

    What Multi-Agent Operations Actually Look Like in Practice

    Most organisations experimenting with AI agents are still operating them like a single chat window. Someone opens a prompt, asks the agent to do something, waits for the output, and moves on. That works for demos. It does not work when you are running agents against production systems or trying to get consistent results across a team.

    The gap is not technical. It is operational. The organisations getting genuine value from AI agents are not the ones with the most advanced models. They are the ones that figured out how to coordinate agents the way you would coordinate a team: clear roles, defined handoffs, checkpoint reviews, and someone accountable for the outcome.

    The Governance Gap Nobody Talks About

    The current wave of AI agent tooling is impressive. You can spin up an agent that writes code, another that reviews it, another that runs tests, and a fourth that deploys. The demos are compelling. The problem is that most organisations have not thought about what happens when these agents operate together at scale.

    Who coordinates them? What happens when two agents make conflicting changes? Where is the state stored, and who can inspect it? If an agent fails halfway through a task, what recovers? If an agent produces an incorrect output that another agent consumes, how do you trace the error back?

    These are the same questions you would ask about any multi-person production system. The difference is that agents do not have common sense, do not ask clarifying questions by default, and do not stop when something looks wrong unless you have built in the checks.

    The governance gap is this: most teams have moved from “can we run an agent?” to “we are running agents” without establishing the coordination layer in between.

    The Pattern That Actually Works

    After running autonomous coding agents in production for several months, the pattern that has proven reliable is a hybrid orchestration model. It has four parts.

    A coordinator role. One agent, or one human, owns the overall task. This role does not do the detailed work. It defines the objective, breaks it into independent subtasks, assigns each to a worker, and reviews the results. In practice, this is the role I occupy when running Hermes Agent, Claude Code, or Codex on a project. I set the direction, handle security decisions and state management, and delegate the pure coding work.

    Parallel worker agents. When subtasks are independent, they run simultaneously. Three agents working on three separate services at the same time complete in minutes what a single agent would handle sequentially in an hour. The key requirement is that the subtasks must be genuinely independent. If agent B depends on agent A’s output, running them in parallel creates conflicts, not speed.

    State machines for complex flows. When a task has sequential dependencies, a simple state machine prevents chaos. Each agent picks up the task at a defined state, does its work, writes output to a known location, and transitions the task forward. If an agent fails, the state does not advance. The next agent picks up the failed state and either retries or escalates.

    Checkpoint reviews. At defined points in the flow, a human reviews the output before the next stage begins. This is not a bottleneck. It is a safety mechanism. The review confirms that the output is sane, the state is correct, and the next stage has what it needs. In practice, these reviews take seconds when things are going well and save hours when they are not.

    A Concrete Example: Diagnosing Three Services at Once

    Suppose three independent services are exhibiting issues simultaneously. A traditional approach investigates them sequentially: diagnose service A, fix it, move to service B, fix it, move to service C.

    With a multi-agent setup, the coordinator defines the diagnostic task for each service and spins up three parallel subagents. Each agent gets the same instructions: examine the logs, identify the root cause, propose a fix, and write its findings to a shared state file. The agents do not communicate with each other. They do not need to. They are working on independent systems.

    When all three agents have completed their tasks, the coordinator reviews the findings, checks for conflicts (two agents proposing changes to a shared dependency, for example), and either approves the fixes or escalates for human review.

    A diagnostic process that would take a single engineer most of a day takes under thirty minutes. The quality is not lower — each agent focuses on a single problem without context-switching. The risk is not higher — the checkpoint review catches anything anomalous before it reaches production.

    This is not theoretical. It is a routine operational pattern that runs on free-tier models for the worker agents. The expensive model is the coordinator, and even that role can be handled by a human with a clear framework.

    The Cost Conversation

    There is a persistent misconception that running AI agents at scale requires expensive API subscriptions. In practice, the opposite is true. Worker agents doing diagnostics, code generation, and testing do not need frontier models. They need competent instruction-following, and that is available on free tiers or at very low cost.

    The coordinator role is where model quality matters. This is the agent making decisions about task decomposition, conflict resolution, and escalation. It needs to reason well. But there is only one coordinator, and it does relatively little token-heavy work compared to the workers.

    The cost structure in a well-designed multi-agent system is front-loaded into the coordination layer and minimal in the execution layer. You are paying for one good decision-maker and many cheap workers. The economics favour this model, which is one reason it works for cost-conscious organisations, not just well-funded ones.

    Failure recovery follows the same logic. When an agent fails on a free tier, the cost of retry is zero. When an agent fails on an expensive tier, every retry is a budget event. Putting cheap agents on high-volume work and the expensive agent on high-judgement work is not just an architectural decision. It is a cost optimisation.

    What Organisations Should Do Next

    If you are running or planning to run AI agents in production, the operational model matters more than model selection. Here is where to start.

    Define the coordinator role first. Decide whether a human or an agent owns task decomposition and review. Document what this role is responsible for and what decisions require escalation. This is your governance layer.

    Identify independent subtasks. Look at your current agent workflows and find the tasks that can run in parallel. Sequential workflows where tasks are independent are leaving time on the table.

    Build state into your workflows. Every agent should write its output to a known location in a known format. Every downstream agent should read from that location. If you cannot inspect workflow state at any point without replaying the entire execution, your state management is insufficient.

    Set checkpoint reviews at decision points. Not at every step — that defeats the purpose. At points where an incorrect output would propagate downstream and cause real damage. A review that takes five seconds and prevents a two-hour debugging session is time well spent.

    Use the right model for the right role. Do not pay frontier-model prices for tasks that a free-tier model handles competently. Reserve your budget for the coordination and review layers where reasoning quality directly affects outcomes.

    If your organisation is moving from AI experimentation to production agent operations, the coordination layer is where the value is — and where the risk lives. The AI & Automation Architecture service covers the design of multi-agent systems with proper governance, state management, and cost controls. Or get in touch for a conversation about what your agent operations should look like before they scale.

  • How Multi-Agent Operations Work in Practice

    e

    Most organisations still treat AI agents like a single chat window with extra buttons. That is fine for a demo. It is not fine when the work touches production systems.

    The difference is operational, not magical. The teams getting value from agents are the ones that add roles, checkpoints, and ownership.

    What works

    A reliable setup usually has four pieces:

    • a coordinator that defines the task and checks the output
    • worker agents that do independent chunks in parallel
    • state that lives somewhere everyone can inspect
    • review points before anything risky moves forward

    That is not glamorous, but it works.

    A simple example

    If three services are acting up at once, a good coordinator breaks the problem apart and sends each service to a separate worker. The workers do not need to talk to each other because the tasks are independent. The coordinator then compares the results and decides whether to approve the fix or escalate.

    That pattern saves time without turning the system into a black box.

    The cost question

    People often assume agent work must be expensive. In practice, the opposite is usually true. The cheap model can do the repetitive work. The better reasoning model is reserved for coordination and review.

    That split matters. It keeps the system affordable and keeps judgment where it belongs.

    What to do first

    1. Decide who owns the outcome.
    2. Identify which tasks can run in parallel.
    3. Make state visible.
    4. Add checkpoints where mistakes would hurt.

    That is enough to get started. The rest is tuning.

  • HeliOS-Studio: AI Startup Studio Ignites

    t

    After a few years of building AI tooling, I hit a point where the stack stopped feeling like scaffolding. It started to look like a product in its own right.

    That is a strange moment. You begin by solving a narrow operational problem, then realise the workflow you built to support the work is now valuable enough to stand on its own.

    The shape of it

    The setup was simple in principle:

    • one layer for orchestration
    • one layer for safe execution
    • one layer for inference
    • one layer for content and delivery

    The names changed over time. The pattern did not.

    What the studio produced

    The useful output was not a single breakthrough. It was a steady stream of small, shippable things: business plans, MVP outlines, content drafts, docs, and working repos.

    That changed how I thought about progress. Instead of asking, “Can the system automate this?” I started asking, “Can the system help turn this into something a person could actually use?”

    The takeaway

    Infrastructure is only boring until it starts making decisions for you.

    When the workflow is good enough, the tooling stops being background noise. It becomes part of the offer.

  • EU AI Act Compliance: Governance Frameworks in Practice

    EU AI Act: My Clients Were Ready. Most Weren’t.
    Published: November 10, 2025 (retrospective)

    EU AI Act enforcement began in earnest in late 2025. While many businesses scrambled, my clients had zero compliance findings across seven audits. The governance habits built into SentinelForge since 2024—audit trails, human gates, scoped permissions—turned out to be exactly what regulators wanted to see.

    Framework Coverage

    Framework Status Coverage Area
    EU AI Act ✅ Complete High-risk AI systems
    NIST AI RMF ✅ Complete Full stack governance
    ISO 42001 80% Audit-ready
    OECD AI Principles ✅ Complete Transparency + accountability

    What Auditors Actually Look For

    1. Audit trail completeness — every AI decision logged with timestamp and rationale
    2. Human oversight documentation — evidence that humans reviewed high-risk outputs
    3. Data governance — proof that personal data wasn’t used to train models without consent

    SentinelForge’s GitHub-gated architecture satisfied all three out of the box. The logs were already there.

    The Lesson

    Compliance isn’t a bolt-on. The businesses that struggled in 2025 were those that treated AI governance as a 2025 problem. We started in 2023.

    Need EU AI Act readiness for your AI systems? Book a governance audit.

    Next: HeliOS-Studio—AI startup studio ignites (Feb 2026).

  • AI Arms Race: Predictive Cyber Defence

    AI Arms Race: Predictive Cyber Defence Is Here
    Published: August 20, 2025 (retrospective)

    The AI cybersecurity market is projected to hit $60B by 2028—and for good reason. In August 2025, SentinelForge v2’s predictive threat hunting caught a client ransomware pivot 72 hours before it would have detonated. No SOC. No SIEM subscription. Just CrewAI agents, local LLMs, and disciplined governance.

    SentinelForge v2 Production Stack

    proxmox-ve
└── sentinelforge (docker)
    ├── crewai crews     (24/7 autonomous monitoring)
    ├── ollama           (local inference)
    ├── grafana          (observability)
    └── uptimekuma       (SLA: 99.9%)

    The Catch: Anatomy of a Prevention

    • Day 1: Anomalous LDAP query pattern flagged by Audit Crew
    • Day 2: Lateral movement indicators correlated across 3 systems
    • Day 3 (72h): Human review triggered; client isolated affected segment
    • Result: Zero encryption, zero ransom, zero downtime

    What This Means for SMEs

    Enterprise-grade predictive defence is now accessible without enterprise budgets. The stack cost: £0/month in cloud tokens, running on repurposed hardware.

    1. AI agents don’t get tired—24/7 monitoring without alert fatigue.
    2. Local inference keeps sensitive threat data off third-party servers.
    3. Governance logs every detection decision—invaluable for insurance and compliance.

    Want predictive AI defence for your business? Book a Secure AI QuickScan.

    Next: EU AI Act compliance—governance frameworks in practice (Nov 2025).

Powered by