RichardHam.co.uk Blog

Tag: AI

  • Weekly GitHub Activity — 22–29 June 2026

    Introduction

    Another busy week in the repositories. Between infrastructure automation, platform hardening, commercial product development, and a small mountain of documentation, the commit logs tell the story of a systemthat’s rapidly moving from “hand-crafted and held together with SSH sessions” toward something far more repeatable and governed. Let’s walk through what landed.

    What Happened

    hermes-mgmt: Hardening, Governance, and AI Spend Controls

    The hermes-mgmt repository saw 10 commits and 20 pull requests this week, almost all of them orbiting two themes: Tier 2 operational hardening and AI spend governance.

    On the hardening side, a suite of operator runbooks and governance records landed — documenting the execution of Tier 2 hardening scripts, session decisions from backlog triage, and a full operator TODO instruction document. PR #694 tackled memory infrastructure head-on, covering a Qdrant deployment, gateway shutdown procedures, and Telegram 429 rate-limit handling. PR #695 addressed bug remediation across Spotify integration, n8n authentication, vision/video backend issues, and a 429 retry storm that was causing more harm than good.

    The AI spend work is particularly interesting. PR #696 introduced budget policies, cost telemetry, context-optimisation strategies, and prompt-caching controls — essentially an AI spend optimisation control plane. As LLM-powered workloads grow, having a formal mechanism to track, cap, and optimise spend is no longer nice-to-have; it’s operational hygiene. The design docs for child components that accompanied this work sketch out a system where every agent invocation carries a cost context, and budget alarms fire before the invoice does.

    Security compliance also got attention. PR #691 documented progress against SecureScore controls SEC-002, SEC-006, RES-002, and MAT-001 — a reminder that even in a solo-operator environment, treating compliance as code keeps the audit trail honest.

    A quieter but important fix: PR #640 made the Mem0→Letta memory backfill idempotent, preventing duplicate records on re-runs. Small change, big impact on data integrity.

    hamnet: The Control Plane Takes Shape

    hamnet was the most active repository this week — 10 commits, 18 pull requests, and 40 open issues. The headline is the emergence of a single-pane control-plane dashboard (PR #183), backed by a growing collection of infrastructure collectors.

    The control-plane work is being built incrementally and deliberately:

    • PR #177 established the identity model, a static inventory MVP, and the first batch of workload collectors.
    • PR #179 added a second batch — AgentRadar, Ollama status, repository state, and NetBox integration.
    • PR #176 fixed Grafana drift authentication and a tvheadend timeout issue that was causing monitoring gaps.

    On the infrastructure-as-code front, PR #178 delivered a comprehensive IaC roadmap with schemas, runbooks, and a system-classification model. PR #194 proposed enforcing unattended-upgrades policies on external cloud hosts through Ansible — a defensive measure that ensures security patches land even when nobody’s watching.

    Several fixes made day-to-day operations smoother: a script to tag missing VMs with ansible:managed in NetBox (commit 33d2fd9), a UTF-8 locale fix for ansible-inventory (commit 198d9b8), and adding Gitea to the Docker container detection logic (commit a950c67) — a gap that meant the monitoring stack was blind to a self-hosted service.

    A Docker image drift management runbook (commit 81fa372) and operator runbooks for 11 infrastructure issues (commit 8f09076) round out the picture. hamnet is building the operational backbone: not just running services, but knowing what’s running, whether it should be, and what to do when it isn’t.

    The 40 open issues deserve a mention — they’re a candid inventory of what needs attention, from an Oracle scanner that’s out of sync and a PVE exporter reporting zero metrics to Loki missing log-shipping and stale NetBox entries. Filing them is half the battle.

    ms365-agentic-ai: Collectors, Cost Controls, and Production Readiness

    The ms365-agentic-ai repository is accelerating. Five commits this week laid significant groundwork:

    • A deployable Azure Functions host with a run-history sink and live-path tests (commit 5465364).
    • Live read-only fetch wiring for all collectors (commit 416c537).
    • A cost-management component combining an estimator with an Azure cost collector (commit 86ab202).
    • Production-readiness runbook with dry-run and preflight tooling (commit 5da09c4).
    • A simple start guide, docs index, and FAQ/troubleshooting section (commit 212729a).

    The 10 open issues (#6–#15) are essentially the product roadmap: approval-gated remediation models, audit logging, Teams health digests, Azure resource health and Defender XDR collectors, Graph permissions matrices, and licensing/cost matrices. What’s notable is the breadth — this isn’t just a tool, it’s an operational platform being designed with governance from the start.

    richardham-co-uk-ConsultancyOS: Commercial Foundations

    ConsultancyOS — the commercial operating-system template — got its infrastructure in order this week. PR #12 added CI, licensing, a contributing guide, and a docs handbook. A SessionStart hook for web sessions (commit 1a54d22) shows the product thinking extending into the user experience.

    The 9 open issues map out the productisation journey: operational dashboards, a GitHub delivery-repo template, LLM integration, n8n workflow scaffolding, accountant review processes, company identity, and integrations with Dolibarr CRM and Invoice Ninja. This is a project that’s explicitly moving from “personal tooling” toward “repeatable commercial offering.”

    on-maintenance-ai-roadmap: Client-Facing Delivery

    The on-maintenance-ai-roadmap repository focused on communication and deployment model this week. A client-facing project plan (PR #23), an Architecture Decision Record for production hosting in client-owned Azure (PR #21), and a deploy-from-GitHub infrastructure model (PR #24) all point to a delivery pattern where the client retains ownership and control of their environment while the project provides the automation and intelligence layer.

    ai-cloud-credits-grant: Cloud Funding Applications

    Six commits across the ai-cloud-credits-grant repository added usage plans for NVIDIA Inception, Cloudflare, Oracle OCI, AWS, and Google Cloud. These are structured applications for cloud-provider credit programmes — the kind of unglamorous but essential work that unlocks compute capacity for projects that need it. The 6 open issues track the remaining work: a master company profile, applications to the major providers, and an evidence pack.

    Key Takeaways

    1. The control plane is real. hamnet’s dashboard, collectors, and identity model aren’t prototypes — they’re an incremental build toward a system that provides visibility and governance across the entire infrastructure. When you can see everything from one pane, you can fix things before they break.

    2. AI spend is now a first-class concern. The hermes-mgmt AI spend control plane — with budget policies, cost telemetry, and prompt caching — reflects a mature approach to LLM operations. The era of “just call the API and hope the bill is reasonable” is over, at least in this stack.

    3. Documentation as a delivery artifact. Across every repository this week, documentation wasn’t an afterthought — it was a first-class deliverable. Operator runbooks, governance records, ADRs, client-facing plans, and onboarding guides shipped alongside the code. This is the behaviour that separates sustainable projects from weekend hacks.

    4. Compliance isn’t optional, even for small teams. The SecureScore work in hermes-mgmt and the approval-gated remediation model in ms365-agentic-ai show that compliance and auditability are being designed in, not bolted on. That’s cheaper and more effective at any scale.

    5. Open issues are a feature, not a bug. hamnet’s 40 open issues and ms365-agentic-ai’s 10 open issues aren’t backlogs — they’re transparent roadmaps. Filing what needs doing, and making it visible, is how solo operators avoid the “I’ll remember that later” trap.

    This post is part of a weekly series summarising GitHub activity across the project portfolio. Previous entries are available in the blog archive.

  • The Secrets Management Mistakes I See in AI Infrastructure

    The Secrets Management Mistakes I See in AI Infrastructure

    I have reviewed dozens of AI infrastructure stacks over the past year. The pattern is depressingly consistent: a team spins up Langfuse for LLM observability, adds n8n for workflow automation, plugs in Hermes or another AI agent — and within a month, there are database passwords committed to git and API keys baked directly into Docker Compose files.

    It is not that these teams do not care about security. It is that the tooling is easy to deploy and terrifyingly easy to deploy wrong. The secrets management layer is an afterthought, and by the time anyone notices, the rot is already baked into the repository history — often with several stale copies scattered across forks, CI logs, and deployment scripts.

    The tools are not the problem. The defaults are survivable if you change them immediately and manage them properly. The problem is that almost nobody does.

    Why this matters now

    AI infrastructure is different from a typical web app in one important respect: it holds the keys to your models, your data pipelines, and increasingly, your customer data. When a secrets leak happens in an AI stack, it is not just a credential rotation exercise. It can mean exposing vector stores full of proprietary documents, handing over API keys with uncapped billing, or losing control of an agent that has been given broad access to your internal systems.

    UK SMEs are adopting these tools faster than their security practices can keep up. Boards are asking for AI capability. Technical founders want to move fast. The result is that Langfuse and n8n instances go live with the same enthusiasm and the same rigour as a weekend side project.

    The NCSC has been clear that the shared responsibility model applies here. The platform provides the controls. You have to configure them. And right now, most teams deploying AI infrastructure are not.

    A war story from the field

    I recently reviewed a deployment where the Langfuse Docker Compose file had been committed to a version-controlled repository with production database credentials hardcoded directly into it. Not in an environment variable reference — the actual username and password, sitting in plaintext in a file that had been committed, reviewed, merged, and deployed.

    That was not the only issue. The same deployment had a Claude configuration that did not properly handle credential rotation after operator restarts. Every time the container restarted — patching, scaling, node migration — the service would silently fall back to insecure defaults. Nobody noticed for weeks because the service appeared to be running. It was running. It was just doing so with credentials that had long since been rotated and should no longer have been valid.

    I also found an n8n instance where the encryption key had not been persisted. The team had recreated their n8n container as part of a routine update, and the platform generated a new encryption key on startup. Every credential stored in n8n’s database became undecryptable. Every workflow that depended on stored API keys, database connections, or OAuth tokens broke simultaneously.

    The error message is admirably specific: “A different encryptionKey was used to encrypt the data.” But by the time you see it, all of your workflows are failing in production — and unless you have the original encryption key backed up somewhere outside the container, those credentials are gone.

    These are not edge cases. They are the most common findings in every AI infrastructure review I have done in the last twelve months.

    The five mistakes I see most often

    1. Hardcoded secrets in Docker Compose and environment files

    This is the big one. Teams copy a docker-compose.yml from a project README, fill in their passwords in plaintext, and commit it. Sometimes they remember to add .env to .gitignore but leave the Compose file itself exposed. Sometimes they move the secrets to an environment file but commit that too, because the .gitignore was only added after the first commit.

    2. Using default credentials past the first five minutes of setup

    Default credentials exist so you can get started quickly. They should exist in production for approximately zero seconds after the health check passes. I regularly find admin/admin or changeme on instances that have been running for months, sometimes years. If a tool ships with a default password, changing it should be the very first action in your runbook — not something you plan to do later.

    3. Losing the encryption key when containers are recreated

    This one catches people out constantly with n8n, but it applies to any platform that encrypts stored credentials. When you recreate a container without persisting the encryption keys, a new one is generated and the old data is orphaned. The platform cannot protect you from this. It is a configuration decision. You need to persist encryption keys outside the container lifecycle — in your secrets manager, in your CI/CD pipeline configuration, in a mounted volume that survives container recreation.

    4. Treating the AI stack as lower risk than the rest of the infrastructure

    There is a pernicious perception that the “AI tools” are supplementary, experimental, not worth the same rigour as the production database or the payment gateway. This is backwards. Your AI stack touches your most sensitive data — the documents you embed, the conversations you log, the internal APIs your agents call. It has the broadest external API surface in your architecture. It deserves more scrutiny, not less.

    5. Relying on git history alone to “remove” secrets

    Committing a secret, then removing it in a follow-up commit, does not remove it from git history. It just adds another commit on top. The secret is still there, reachable via git log -p, via GitHub’s commit history, via any clone or fork. Once a secret has been committed, the only safe response is rotation — treat it as compromised and issue new credentials.

    What to do next

    You do not need an enterprise secrets vault to get the basics right. You need a checklist and the discipline to follow it.

    1. Audit your repositories now. Search for common patterns: PASSWORD=, SECRET=, API_KEY=, base64-encoded blobs in environment files. Use tools like gitleaks or trufflehog to scan both current state and history. If anything turns up, rotate the credentials immediately — do not just remove them from the latest commit.

    2. Externalise every secret. In Docker Compose, reference ${VAR} values and use a .env file that is .gitignored at the repository root level. If you are on a managed platform, use its native secrets manager. No exceptions, no “just for now”, no “it is only a development environment.”

    3. Persist your n8n encryption key. Set the N8N_ENCRYPTION_KEY environment variable explicitly — do not let n8n auto-generate it. Store it in your secrets manager and make it part of your container orchestration configuration, not the container itself. Back it up. Test that you can restore it.

    4. Rotate after every environment change. When a container is recreated, when a team member leaves, when you are not even sure something was exposed — rotate. Treat the cost of rotation as negligible compared to the cost of a breach. Automate it wherever you can.

    5. Apply the same standard to AI tools as everything else. Your Langfuse instance, your n8n deployment, your AI agent platform — these hold credentials and data that would interest an attacker. Give them the same security treatment you would give a production database. No concessions for “it is internal” or “it is just a prototype.”

    Where Richard can help

    If you are deploying AI infrastructure and want an honest assessment of how your secrets are managed — before an incident forces the conversation — I offer infrastructure security reviews and fractional CISO engagements tailored to UK SMEs running containers, workflows, and AI tooling.

    The mistakes above are among the most common findings in the engagements I do. They are also almost always fixable in a day.

    Get in touch or review the available services to arrange a review.

  • GitHub Weekly — SecureScore Goes Live, Agentic Ops Mature, and the Atlas Foundation Takes Shape

    Introduction

    Seventy-four repos, one week, and a surprising amount of shipped work. This week’s activity spans three themes: production security tooling crossing from plan into live operation, agentic infrastructure maturing with better observability and failover, and a new initiative — the Atlas Foundation — taking shape around public-good AI proposals. Here’s what happened, what shipped, and what it signals.

    What Happened

    SecureScore: From Dry-Run to Live

    The Hermes SecureScore project crossed a meaningful threshold this week. After install dry-runs and collector CLI fixes over the previous days, the project recorded its live activation on June 19. The Docker collector for SecureScore evidence shipped, a config collector bug for string provider entries was fixed, and the evidence pipeline moved from testing to production. On the dashboard side, the main Hermes Agent repo merged PR #1 adding a SecureScore view, confirming this isn’t a standalone experiment but an integrated part of operations.

    Why it matters: security评分 that runs locally, on your own infra, without sending data externally — that’s a pattern more teams will need as AI agent deployments multiply and audit requirements tighten.

    Hermes Mgmt: Dashboard v0.17 and the Work Behind It

    The management dashboard saw a methodical series of doc-and-script updates. Highlights:

    • v0.17.0 migration guide published, including a process restart requirement — a small detail that saves a lot of “why isn’t this working?” confusion.
    • Langfuse cost analytics landed as a cross-provider spend report with a weekly digest cron. Teams running multi-model setups can now track spend across providers without spreadsheet gymnastics.
    • Agent Tool Audit Report 2026-06-24 opened as a PR, systematically cataloguing what the agent toolkit actually contains and where the gaps are.
    • Telegram flood protection (Layers 3+4+5) merged, covering restart hygiene, notification deduplication, and rate-limit handling.
    • Security vulnerability remediation — 16 CVEs patched across nltk and starlette dependencies.

    The pattern here isn’t glamorous features; it’s the unglamorous maintainability work that keeps agent infrastructure from rotting.

    Agentic Coordination: Failover, Cost Awareness, and Escalation

    Three PRs from the Hermes Mgmt repo tell a story about agentic ops moving past the demo phase:

    1. Local-primary triage and tiered escalation (PR #624) — an EPIC outlining how a local model (Hermes3:8b) handles first-tier requests and escalates to stronger models when responses are weak or empty. This is the failover tier the project has been building toward.
    2. Credit-awareness fix plan for OpenRouter — because nothing kills an agent workflow faster than hitting a model credit limit mid-task without warning.
    3. Agent Radar got its initial commit with a roadmap of MVP tasks: detector rules for common agent frameworks, repo status badges, and schema examples. The idea is to build a tool that scans repos and identifies agent frameworks automatically.

    Infrastructure: Hardening and Monitoring

    On the infrastructure side:

    • Hamnet shipped SSH hardening for the VPS with a dynamic-IP allowlist failsafe — the kind of defensive depth that matters when IPs change and locks you out. Vhost routing was also hardened after an incident where a domain served the wrong site.
    • Ollama dashboard and metrics pipeline (Hamnet PR #134) — because if you’re running models locally, you need to know the server is alive without checking manually.
    • A WordPress blog import runbook was added, documenting the pipeline that keeps the blog publishing workflow reproducible.

    Atlas Foundation: Public-Good Agent Proposals

    The project-atlas-foundation repo saw significant activity: candidate proposals scored, shaping docs created, and a safety checklist added. The current proposal slate includes:

    • An open-source issue triage assistant
    • A digital-access assistant for elderly users
    • A small charity automation kit
    • A public-good agent template library

    A Claude Code handover document was added for the team, covering setup, sync, and triage workflows — suggesting the project is moving from concept to collaborative execution.

    HamMediaLabs: Governance Deepening

    HamMediaLabs shipped a wave of internal governance: onboarding guides, a development guide, risk register, branch hygiene policy, and a PR review dashboard with dependency health reporting. For a small team, this is the scaffolding that prevents chaos as contributors scale up.

    Key Takeaways

    1. Production security is becoming operational, not aspirational. SecureScore’s live activation is a signal that local-first security scoring is viable for small teams without enterprise tooling budgets.

    2. The agent ops story is shifting from “can we build it?” to “can we keep it running?” Flood protection, cost dashboards, failover tiers, and tool audits — this is the maintenance phase of the agentic infrastructure lifecycle.

    3. Multi-model cost visibility is now a first-class concern. Langfuse cost analytics with cross-provider reporting, OpenRouter credit-awareness, and tiered escalation — these all reflect the reality that running multiple AI models costs real money and needs real monitoring.

    4. Governance-as-code is emerging in smaller projects. HamMediaLabs and Atlas Foundation both shipped policy documents as code: branch hygiene, risk registers, safety checklists. This is where the industry is heading — compliance documentation that lives in the repo, not in a SharePoint graveyard.

    Code Snippet: Dynamic SSH Allowlist

    From the Hamnet infrastructure work, a pattern worth showing — dynamically resolving an ISP’s current IP for an SSH allowlist, with a failsafe that doesn’t lock you out:

    #!/usr/bin/env bash
# update-ssh-allowlist.sh — refresh dynamic IP in allowlist
set -euo pipefail

CURRENT_IP=$(curl -s --max-time 5 https://ifconfig.me)
KNOWN_FALLBACK="203.0.113.0/24"  # static backup range

if [[ "$CURRENT_IP" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    iptables -D INPUT -p tcp --dport 22 -j DROP 2>/dev/null || true
    iptables -A INPUT -s "$CURRENT_IP/32" -p tcp --dport 22 -j ACCEPT
    echo "Updated: $CURRENT_IP/32"
else
    iptables -A INPUT -s "$KNOWN_FALLBACK" -p tcp --dport 22 -j ACCEPT
    echo "Fallback applied: $KNOWN_FALLBACK"
fi

    The key insight: always have a fallback. Dynamic DNS is reliable until the day your ISP changes your IP at 2 AM and you can’t SSH in to fix it.

    Data source: commits, issues, and PRs from 49 repositories over the past 7 days. Collected via the GitHub CLI on 2026-06-25.

  • GitHub Weekly — SecureScore Goes Live, Agentic Ops Mature, and the Atlas Foundation Takes Shape

    Introduction

    Seventy-four repos, one week, and a surprising amount of shipped work. This week’s activity spans three themes: production security tooling crossing from plan into live operation, agentic infrastructure maturing with better observability and failover, and a new initiative — the Atlas Foundation — taking shape around public-good AI proposals. Here’s what happened, what shipped, and what it signals.

    What Happened

    SecureScore: From Dry-Run to Live

    The Hermes SecureScore project crossed a meaningful threshold this week. After install dry-runs and collector CLI fixes over the previous days, the project recorded its live activation on June 19. The Docker collector for SecureScore evidence shipped, a config collector bug for string provider entries was fixed, and the evidence pipeline moved from testing to production. On the dashboard side, the main Hermes Agent repo merged PR #1 adding a SecureScore view, confirming this isn’t a standalone experiment but an integrated part of operations.

    Why it matters: security评分 that runs locally, on your own infra, without sending data externally — that’s a pattern more teams will need as AI agent deployments multiply and audit requirements tighten.

    Hermes Mgmt: Dashboard v0.17 and the Work Behind It

    The management dashboard saw a methodical series of doc-and-script updates. Highlights:

    • v0.17.0 migration guide published, including a process restart requirement — a small detail that saves a lot of “why isn’t this working?” confusion.
    • Langfuse cost analytics landed as a cross-provider spend report with a weekly digest cron. Teams running multi-model setups can now track spend across providers without spreadsheet gymnastics.
    • Agent Tool Audit Report 2026-06-24 opened as a PR, systematically cataloguing what the agent toolkit actually contains and where the gaps are.
    • Telegram flood protection (Layers 3+4+5) merged, covering restart hygiene, notification deduplication, and rate-limit handling.
    • Security vulnerability remediation — 16 CVEs patched across nltk and starlette dependencies.

    The pattern here isn’t glamorous features; it’s the unglamorous maintainability work that keeps agent infrastructure from rotting.

    Agentic Coordination: Failover, Cost Awareness, and Escalation

    Three PRs from the Hermes Mgmt repo tell a story about agentic ops moving past the demo phase:

    1. Local-primary triage and tiered escalation (PR #624) — an EPIC outlining how a local model (Hermes3:8b) handles first-tier requests and escalates to stronger models when responses are weak or empty. This is the failover tier the project has been building toward.
    2. Credit-awareness fix plan for OpenRouter — because nothing kills an agent workflow faster than hitting a model credit limit mid-task without warning.
    3. Agent Radar got its initial commit with a roadmap of MVP tasks: detector rules for common agent frameworks, repo status badges, and schema examples. The idea is to build a tool that scans repos and identifies agent frameworks automatically.

    Infrastructure: Hardening and Monitoring

    On the infrastructure side:

    • Hamnet shipped SSH hardening for the VPS with a dynamic-IP allowlist failsafe — the kind of defensive depth that matters when IPs change and locks you out. Vhost routing was also hardened after an incident where a domain served the wrong site.
    • Ollama dashboard and metrics pipeline (Hamnet PR #134) — because if you’re running models locally, you need to know the server is alive without checking manually.
    • A WordPress blog import runbook was added, documenting the pipeline that keeps the blog publishing workflow reproducible.

    Atlas Foundation: Public-Good Agent Proposals

    The project-atlas-foundation repo saw significant activity: candidate proposals scored, shaping docs created, and a safety checklist added. The current proposal slate includes:

    • An open-source issue triage assistant
    • A digital-access assistant for elderly users
    • A small charity automation kit
    • A public-good agent template library

    A Claude Code handover document was added for the team, covering setup, sync, and triage workflows — suggesting the project is moving from concept to collaborative execution.

    HamMediaLabs: Governance Deepening

    HamMediaLabs shipped a wave of internal governance: onboarding guides, a development guide, risk register, branch hygiene policy, and a PR review dashboard with dependency health reporting. For a small team, this is the scaffolding that prevents chaos as contributors scale up.

    Key Takeaways

    1. Production security is becoming operational, not aspirational. SecureScore’s live activation is a signal that local-first security scoring is viable for small teams without enterprise tooling budgets.

    2. The agent ops story is shifting from “can we build it?” to “can we keep it running?” Flood protection, cost dashboards, failover tiers, and tool audits — this is the maintenance phase of the agentic infrastructure lifecycle.

    3. Multi-model cost visibility is now a first-class concern. Langfuse cost analytics with cross-provider reporting, OpenRouter credit-awareness, and tiered escalation — these all reflect the reality that running multiple AI models costs real money and needs real monitoring.

    4. Governance-as-code is emerging in smaller projects. HamMediaLabs and Atlas Foundation both shipped policy documents as code: branch hygiene, risk registers, safety checklists. This is where the industry is heading — compliance documentation that lives in the repo, not in a SharePoint graveyard.

    Code Snippet: Dynamic SSH Allowlist

    From the Hamnet infrastructure work, a pattern worth showing — dynamically resolving an ISP’s current IP for an SSH allowlist, with a failsafe that doesn’t lock you out:

    #!/usr/bin/env bash
# update-ssh-allowlist.sh — refresh dynamic IP in allowlist
set -euo pipefail

CURRENT_IP=$(curl -s --max-time 5 https://ifconfig.me)
KNOWN_FALLBACK="203.0.113.0/24"  # static backup range

if [[ "$CURRENT_IP" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    iptables -D INPUT -p tcp --dport 22 -j DROP 2>/dev/null || true
    iptables -A INPUT -s "$CURRENT_IP/32" -p tcp --dport 22 -j ACCEPT
    echo "Updated: $CURRENT_IP/32"
else
    iptables -A INPUT -s "$KNOWN_FALLBACK" -p tcp --dport 22 -j ACCEPT
    echo "Fallback applied: $KNOWN_FALLBACK"
fi

    The key insight: always have a fallback. Dynamic DNS is reliable until the day your ISP changes your IP at 2 AM and you can’t SSH in to fix it.

    Data source: commits, issues, and PRs from 49 repositories over the past 7 days. Collected via the GitHub CLI on 2026-06-25.

  • GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    g

    The week of June 15-22 was not defined by a single dramatic event. Instead, it was the kind of week that separates platforms that merely work from platforms that hold up under sustained pressure. Across 50 repositories, over 100 events — commits, pull requests, issues — pushed forward several long-running threads: memory architecture, health monitoring, security governance, and the unglamorous but essential work of keeping production infrastructure honest.

    What Happened

    hermes-mgmt — Hardening the Core

    The hermes-mgmt repository remained the busiest node in the network, with 20-plus commits, 20 PRs, and 13 issues. The dominant theme was memory system reliability. A significant fix (PR covering issues #573, #574, #575) hardened dual_memory.py across three dimensions: Ollama-first Mem0 configuration, deterministic Letta archival behaviour, and Qdrant vector dimension alignment. These are the kinds of fixes that don’t make headlines but prevent the subtle data corruption that erodes trust in AI systems over time.

    A related fix (issue #411) corrected a memory drift check that was comparing the wrong Letta data — core-memory passages instead of ARCHIVAL passages. This is a telling detail: as memory architectures grow more layered (core, archival, vector), the surface area for misaligned reads increases. Catching this before it caused silent data degradation matters.

    Health probes got a major upgrade with PR #568, which introduced functional health checks for services that report as “green” but are actually broken — Letta, n8n, Langfuse, Qdrant, Gateway, and Ollama. This is a pattern anyone running distributed systems will recognise: the dashboard shows all green, but something is quietly failing. Functional probes go beyond “is the process running?” to “does the service actually respond correctly?” — a distinction that matters enormously in production.

    Secrets management continued to mature. PR #552 delivered a comprehensive secrets management architecture built around Bitwarden and HashiCorp Vault, while PR #527 removed committed default secrets from the Langfuse compose configuration. These are the foundational security practices that need to be in place before scale makes them painful to retrofit.

    On the cron and scheduling side, the system now runs 38 cron jobs with 36 healthy — a 95% health rate that reflects the cumulative effect of weeks of hardening work. PR #504 corrected cron exit-code semantics so that findings and alerts are no longer misinterpreted as failures, which was causing unnecessary noise in the monitoring pipeline.

    Several open issues point to the next layer of work: Telegram flood-control protection (#583), gateway ungraceful shutdown (#582), Ollama-agent sequential dispatch performance (#580), and a Langfuse Cost Report cron that’s been disabled due to stale API keys (#576). None of these are emergencies, but they represent the known gaps that get addressed in order of priority.

    hamnet — Infrastructure Truth

    The hamnet repository, which manages infrastructure automation, saw 9 commits and 10 PRs with a focus on hosting hardening and monitoring.

    SSH hardening was codified for the VPS fleet (PR #131), including a dynamic-IP allowlist failsafe — a practical safeguard for environments where IP addresses shift. Vhost routing was hardened after www.theitrevolution.co.uk was found serving the wrong site, a class of misconfiguration that can be difficult to spot without deliberate verification.

    On the monitoring side, an Alertmanager webhook receiver was added with X-Webhook-Token authentication (PR #87), and a new Mac AI Server dashboard was built for the Ollama era (PR #134). The dashboard work reflects a broader shift: as the AI infrastructure stack evolves (Ollama replacing previous model servers), the monitoring layer needs to evolve in parallel or it becomes a liability rather than an asset.

    An open issue (#95) flags an SSL certificate SAN mismatch for www.richardham.co.uk — the kind of thing that works until it doesn’t, usually at the worst possible moment.

    hermes-agent — Gateway Stability

    The hermes-agent repository received 10 commits focused on gateway reliability. The command-line matcher was hardened, Windows restart no longer causes a silent outage, and the gateway now refreshes its cached agent max_iterations from the current config rather than stale values. A fix for dict choice unwrapping in the clarify function rounds out a set of small but meaningful stability improvements.

    These are the fixes that users never notice — because they prevent the failures that would have been noticed. Silent outages and stale config caches are particularly insidious because they can persist for extended periods before manifesting as user-visible problems.

    hermes-securescore — Evidence Collection

    The SecureScore project advanced with 9 commits and 2 PRs, adding both a Docker collector and a Hermes config collector for security evidence. A high-risk action approval playbook was also added, formalising the governance process for sensitive operations. The project recorded its live activation this week — a milestone that moves it from development into operational use.

    hermes-voice-satellite — Laying Groundwork

    The voice satellite project saw 9 commits establishing the MVP build plan, an operations runbook, a Hermes voice satellite API contract, and a Termux bootstrap script for Android (S24). A native Android app placeholder was also added. This is early-stage infrastructure work — the kind of foundation that needs to exist before feature development can accelerate.

    project-atlas-foundation — Launch Readiness

    Project Atlas received 10 commits focused on governance and launch preparation: a PR template, MIT licence, security policy, CODEOWNERS, a safety checklist, research workflow, promotion process, CI configuration, and hardened lifecycle guides. Scoring of proposals #3-#7 was completed alongside shaping docs and a launch checklist update. This is the organisational scaffolding that turns a codebase into a project other people can contribute to.

    AgentRadar — New Arrival

    AgentRadar received its initial commit this week. Details are sparse, but a new repository appearing in the portfolio is always worth noting — it represents a new thread that will either find its place or be retired honestly.

    richardham-web-and-Brand — Content Pipeline

    The web-and-brand repository saw 10 commits and 5 PRs, primarily focused on blog publishing and content pipeline maintenance. Four new blog posts in the agentic AI series were added with proper date spacing, and a build fix declared window.__calComLoaded to resolve a strict type-check failure. A WordPress publishing blocker was documented (VPS SSH key issue), which is the kind of honest infrastructure transparency that keeps content pipelines reliable.

    Key Takeaways

    Memory architecture is the new frontier. The volume of work on dual-memory hardening, Letta archival alignment, and Qdrant dimension matching signals that the memory subsystem has become a first-class concern. As AI agents handle longer and more complex tasks, the reliability of their memory layer directly determines the reliability of everything built on top of it. The drift-check fix — comparing the right data — is a small change with outsized implications.

    Functional health probes close a critical gap. The distinction between “process is running” and “service is working” is one of the most common blind spots in monitoring. Adding functional probes for six core services moves the observability stack from surface-level to meaningful. This is the kind of investment that pays for itself the first time it catches a green-but-broken service before a user does.

    Security governance is becoming systematic, not reactive. Between the secrets management architecture, the SecureScore evidence collectors, the high-risk action approval playbook, and the removal of committed defaults, the pattern is clear: security is being built into the development process rather than bolted on after incidents. This is the maturation path every platform needs to follow.

    Infrastructure truth matters. The hamnet work — SSH hardening, vhost routing fixes, SSL certificate monitoring, dashboard reconciliation — is the unglamorous foundation that everything else depends on. When www.theitrevolution.co.uk serves the wrong site, no amount of AI sophistication compensates. Keeping the infrastructure layer honest is a continuous discipline, not a one-time project.

    The content pipeline is converging with the platform. The parallel work on blog publishing, brand positioning, and the agentic AI content series is not separate from the technical work — it is how the technical work becomes visible and valuable. A hardened platform with strong governance needs an equally strong narrative around it.

    Looking Ahead

    The open issues across the portfolio paint a clear picture of next week’s priorities: resolve the Telegram flood-control design, address the gateway ungraceful shutdown path, fix the Langfuse Cost Report cron’s stale API keys, and close the SSL certificate SAN mismatch. On the infrastructure side, the Mac AI Server Ollama dashboard and metrics pipeline need to land, and the remaining open PRs in the web-and-brand repository need to progress through review.

    The velocity of the last week is notable not for its drama but for its consistency. Every repository moved forward. No single event dominated. That is what a healthy development portfolio looks like in practice — not a single sprint, but sustained, disciplined progress across every layer of the stack.

  • GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    The week of June 15-22 was not defined by a single dramatic event. Instead, it was the kind of week that separates platforms that merely work from platforms that hold up under sustained pressure. Across 50 repositories, over 100 events — commits, pull requests, issues — pushed forward several long-running threads: memory architecture, health monitoring, security governance, and the unglamorous but essential work of keeping production infrastructure honest.

    What Happened

    hermes-mgmt — Hardening the Core

    The hermes-mgmt repository remained the busiest node in the network, with 20-plus commits, 20 PRs, and 13 issues. The dominant theme was memory system reliability. A significant fix (PR covering issues #573, #574, #575) hardened dual_memory.py across three dimensions: Ollama-first Mem0 configuration, deterministic Letta archival behaviour, and Qdrant vector dimension alignment. These are the kinds of fixes that don’t make headlines but prevent the subtle data corruption that erodes trust in AI systems over time.

    A related fix (issue #411) corrected a memory drift check that was comparing the wrong Letta data — core-memory passages instead of ARCHIVAL passages. This is a telling detail: as memory architectures grow more layered (core, archival, vector), the surface area for misaligned reads increases. Catching this before it caused silent data degradation matters.

    Health probes got a major upgrade with PR #568, which introduced functional health checks for services that report as “green” but are actually broken — Letta, n8n, Langfuse, Qdrant, Gateway, and Ollama. This is a pattern anyone running distributed systems will recognise: the dashboard shows all green, but something is quietly failing. Functional probes go beyond “is the process running?” to “does the service actually respond correctly?” — a distinction that matters enormously in production.

    Secrets management continued to mature. PR #552 delivered a comprehensive secrets management architecture built around Bitwarden and HashiCorp Vault, while PR #527 removed committed default secrets from the Langfuse compose configuration. These are the foundational security practices that need to be in place before scale makes them painful to retrofit.

    On the cron and scheduling side, the system now runs 38 cron jobs with 36 healthy — a 95% health rate that reflects the cumulative effect of weeks of hardening work. PR #504 corrected cron exit-code semantics so that findings and alerts are no longer misinterpreted as failures, which was causing unnecessary noise in the monitoring pipeline.

    Several open issues point to the next layer of work: Telegram flood-control protection (#583), gateway ungraceful shutdown (#582), Ollama-agent sequential dispatch performance (#580), and a Langfuse Cost Report cron that’s been disabled due to stale API keys (#576). None of these are emergencies, but they represent the known gaps that get addressed in order of priority.

    hamnet — Infrastructure Truth

    The hamnet repository, which manages infrastructure automation, saw 9 commits and 10 PRs with a focus on hosting hardening and monitoring.

    SSH hardening was codified for the VPS fleet (PR #131), including a dynamic-IP allowlist failsafe — a practical safeguard for environments where IP addresses shift. Vhost routing was hardened after www.theitrevolution.co.uk was found serving the wrong site, a class of misconfiguration that can be difficult to spot without deliberate verification.

    On the monitoring side, an Alertmanager webhook receiver was added with X-Webhook-Token authentication (PR #87), and a new Mac AI Server dashboard was built for the Ollama era (PR #134). The dashboard work reflects a broader shift: as the AI infrastructure stack evolves (Ollama replacing previous model servers), the monitoring layer needs to evolve in parallel or it becomes a liability rather than an asset.

    An open issue (#95) flags an SSL certificate SAN mismatch for www.richardham.co.uk — the kind of thing that works until it doesn’t, usually at the worst possible moment.

    hermes-agent — Gateway Stability

    The hermes-agent repository received 10 commits focused on gateway reliability. The command-line matcher was hardened, Windows restart no longer causes a silent outage, and the gateway now refreshes its cached agent max_iterations from the current config rather than stale values. A fix for dict choice unwrapping in the clarify function rounds out a set of small but meaningful stability improvements.

    These are the fixes that users never notice — because they prevent the failures that would have been noticed. Silent outages and stale config caches are particularly insidious because they can persist for extended periods before manifesting as user-visible problems.

    hermes-securescore — Evidence Collection

    The SecureScore project advanced with 9 commits and 2 PRs, adding both a Docker collector and a Hermes config collector for security evidence. A high-risk action approval playbook was also added, formalising the governance process for sensitive operations. The project recorded its live activation this week — a milestone that moves it from development into operational use.

    hermes-voice-satellite — Laying Groundwork

    The voice satellite project saw 9 commits establishing the MVP build plan, an operations runbook, a Hermes voice satellite API contract, and a Termux bootstrap script for Android (S24). A native Android app placeholder was also added. This is early-stage infrastructure work — the kind of foundation that needs to exist before feature development can accelerate.

    project-atlas-foundation — Launch Readiness

    Project Atlas received 10 commits focused on governance and launch preparation: a PR template, MIT licence, security policy, CODEOWNERS, a safety checklist, research workflow, promotion process, CI configuration, and hardened lifecycle guides. Scoring of proposals #3-#7 was completed alongside shaping docs and a launch checklist update. This is the organisational scaffolding that turns a codebase into a project other people can contribute to.

    AgentRadar — New Arrival

    AgentRadar received its initial commit this week. Details are sparse, but a new repository appearing in the portfolio is always worth noting — it represents a new thread that will either find its place or be retired honestly.

    richardham-web-and-Brand — Content Pipeline

    The web-and-brand repository saw 10 commits and 5 PRs, primarily focused on blog publishing and content pipeline maintenance. Four new blog posts in the agentic AI series were added with proper date spacing, and a build fix declared window.__calComLoaded to resolve a strict type-check failure. A WordPress publishing blocker was documented (VPS SSH key issue), which is the kind of honest infrastructure transparency that keeps content pipelines reliable.

    Key Takeaways

    Memory architecture is the new frontier. The volume of work on dual-memory hardening, Letta archival alignment, and Qdrant dimension matching signals that the memory subsystem has become a first-class concern. As AI agents handle longer and more complex tasks, the reliability of their memory layer directly determines the reliability of everything built on top of it. The drift-check fix — comparing the right data — is a small change with outsized implications.

    Functional health probes close a critical gap. The distinction between “process is running” and “service is working” is one of the most common blind spots in monitoring. Adding functional probes for six core services moves the observability stack from surface-level to meaningful. This is the kind of investment that pays for itself the first time it catches a green-but-broken service before a user does.

    Security governance is becoming systematic, not reactive. Between the secrets management architecture, the SecureScore evidence collectors, the high-risk action approval playbook, and the removal of committed defaults, the pattern is clear: security is being built into the development process rather than bolted on after incidents. This is the maturation path every platform needs to follow.

    Infrastructure truth matters. The hamnet work — SSH hardening, vhost routing fixes, SSL certificate monitoring, dashboard reconciliation — is the unglamorous foundation that everything else depends on. When www.theitrevolution.co.uk serves the wrong site, no amount of AI sophistication compensates. Keeping the infrastructure layer honest is a continuous discipline, not a one-time project.

    The content pipeline is converging with the platform. The parallel work on blog publishing, brand positioning, and the agentic AI content series is not separate from the technical work — it is how the technical work becomes visible and valuable. A hardened platform with strong governance needs an equally strong narrative around it.

    Looking Ahead

    The open issues across the portfolio paint a clear picture of next week’s priorities: resolve the Telegram flood-control design, address the gateway ungraceful shutdown path, fix the Langfuse Cost Report cron’s stale API keys, and close the SSL certificate SAN mismatch. On the infrastructure side, the Mac AI Server Ollama dashboard and metrics pipeline need to land, and the remaining open PRs in the web-and-brand repository need to progress through review.

    The velocity of the last week is notable not for its drama but for its consistency. Every repository moved forward. No single event dominated. That is what a healthy development portfolio looks like in practice — not a single sprint, but sustained, disciplined progress across every layer of the stack.

  • GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    GitHub Weekly: Memory Architecture, Health Probes, and the Quiet Work of Production Hardening

    The week of June 15-22 was not defined by a single dramatic event. Instead, it was the kind of week that separates platforms that merely work from platforms that hold up under sustained pressure. Across 50 repositories, over 100 events — commits, pull requests, issues — pushed forward several long-running threads: memory architecture, health monitoring, security governance, and the unglamorous but essential work of keeping production infrastructure honest.

    What Happened

    hermes-mgmt — Hardening the Core

    The hermes-mgmt repository remained the busiest node in the network, with 20-plus commits, 20 PRs, and 13 issues. The dominant theme was memory system reliability. A significant fix (PR covering issues #573, #574, #575) hardened dual_memory.py across three dimensions: Ollama-first Mem0 configuration, deterministic Letta archival behaviour, and Qdrant vector dimension alignment. These are the kinds of fixes that don’t make headlines but prevent the subtle data corruption that erodes trust in AI systems over time.

    A related fix (issue #411) corrected a memory drift check that was comparing the wrong Letta data — core-memory passages instead of ARCHIVAL passages. This is a telling detail: as memory architectures grow more layered (core, archival, vector), the surface area for misaligned reads increases. Catching this before it caused silent data degradation matters.

    Health probes got a major upgrade with PR #568, which introduced functional health checks for services that report as “green” but are actually broken — Letta, n8n, Langfuse, Qdrant, Gateway, and Ollama. This is a pattern anyone running distributed systems will recognise: the dashboard shows all green, but something is quietly failing. Functional probes go beyond “is the process running?” to “does the service actually respond correctly?” — a distinction that matters enormously in production.

    Secrets management continued to mature. PR #552 delivered a comprehensive secrets management architecture built around Bitwarden and HashiCorp Vault, while PR #527 removed committed default secrets from the Langfuse compose configuration. These are the foundational security practices that need to be in place before scale makes them painful to retrofit.

    On the cron and scheduling side, the system now runs 38 cron jobs with 36 healthy — a 95% health rate that reflects the cumulative effect of weeks of hardening work. PR #504 corrected cron exit-code semantics so that findings and alerts are no longer misinterpreted as failures, which was causing unnecessary noise in the monitoring pipeline.

    Several open issues point to the next layer of work: Telegram flood-control protection (#583), gateway ungraceful shutdown (#582), Ollama-agent sequential dispatch performance (#580), and a Langfuse Cost Report cron that’s been disabled due to stale API keys (#576). None of these are emergencies, but they represent the known gaps that get addressed in order of priority.

    hamnet — Infrastructure Truth

    The hamnet repository, which manages infrastructure automation, saw 9 commits and 10 PRs with a focus on hosting hardening and monitoring.

    SSH hardening was codified for the VPS fleet (PR #131), including a dynamic-IP allowlist failsafe — a practical safeguard for environments where IP addresses shift. Vhost routing was hardened after www.theitrevolution.co.uk was found serving the wrong site, a class of misconfiguration that can be difficult to spot without deliberate verification.

    On the monitoring side, an Alertmanager webhook receiver was added with X-Webhook-Token authentication (PR #87), and a new Mac AI Server dashboard was built for the Ollama era (PR #134). The dashboard work reflects a broader shift: as the AI infrastructure stack evolves (Ollama replacing previous model servers), the monitoring layer needs to evolve in parallel or it becomes a liability rather than an asset.

    An open issue (#95) flags an SSL certificate SAN mismatch for www.richardham.co.uk — the kind of thing that works until it doesn’t, usually at the worst possible moment.

    hermes-agent — Gateway Stability

    The hermes-agent repository received 10 commits focused on gateway reliability. The command-line matcher was hardened, Windows restart no longer causes a silent outage, and the gateway now refreshes its cached agent max_iterations from the current config rather than stale values. A fix for dict choice unwrapping in the clarify function rounds out a set of small but meaningful stability improvements.

    These are the fixes that users never notice — because they prevent the failures that would have been noticed. Silent outages and stale config caches are particularly insidious because they can persist for extended periods before manifesting as user-visible problems.

    hermes-securescore — Evidence Collection

    The SecureScore project advanced with 9 commits and 2 PRs, adding both a Docker collector and a Hermes config collector for security evidence. A high-risk action approval playbook was also added, formalising the governance process for sensitive operations. The project recorded its live activation this week — a milestone that moves it from development into operational use.

    hermes-voice-satellite — Laying Groundwork

    The voice satellite project saw 9 commits establishing the MVP build plan, an operations runbook, a Hermes voice satellite API contract, and a Termux bootstrap script for Android (S24). A native Android app placeholder was also added. This is early-stage infrastructure work — the kind of foundation that needs to exist before feature development can accelerate.

    project-atlas-foundation — Launch Readiness

    Project Atlas received 10 commits focused on governance and launch preparation: a PR template, MIT licence, security policy, CODEOWNERS, a safety checklist, research workflow, promotion process, CI configuration, and hardened lifecycle guides. Scoring of proposals #3-#7 was completed alongside shaping docs and a launch checklist update. This is the organisational scaffolding that turns a codebase into a project other people can contribute to.

    AgentRadar — New Arrival

    AgentRadar received its initial commit this week. Details are sparse, but a new repository appearing in the portfolio is always worth noting — it represents a new thread that will either find its place or be retired honestly.

    richardham-web-and-Brand — Content Pipeline

    The web-and-brand repository saw 10 commits and 5 PRs, primarily focused on blog publishing and content pipeline maintenance. Four new blog posts in the agentic AI series were added with proper date spacing, and a build fix declared window.__calComLoaded to resolve a strict type-check failure. A WordPress publishing blocker was documented (VPS SSH key issue), which is the kind of honest infrastructure transparency that keeps content pipelines reliable.

    Key Takeaways

    Memory architecture is the new frontier. The volume of work on dual-memory hardening, Letta archival alignment, and Qdrant dimension matching signals that the memory subsystem has become a first-class concern. As AI agents handle longer and more complex tasks, the reliability of their memory layer directly determines the reliability of everything built on top of it. The drift-check fix — comparing the right data — is a small change with outsized implications.

    Functional health probes close a critical gap. The distinction between “process is running” and “service is working” is one of the most common blind spots in monitoring. Adding functional probes for six core services moves the observability stack from surface-level to meaningful. This is the kind of investment that pays for itself the first time it catches a green-but-broken service before a user does.

    Security governance is becoming systematic, not reactive. Between the secrets management architecture, the SecureScore evidence collectors, the high-risk action approval playbook, and the removal of committed defaults, the pattern is clear: security is being built into the development process rather than bolted on after incidents. This is the maturation path every platform needs to follow.

    Infrastructure truth matters. The hamnet work — SSH hardening, vhost routing fixes, SSL certificate monitoring, dashboard reconciliation — is the unglamorous foundation that everything else depends on. When www.theitrevolution.co.uk serves the wrong site, no amount of AI sophistication compensates. Keeping the infrastructure layer honest is a continuous discipline, not a one-time project.

    The content pipeline is converging with the platform. The parallel work on blog publishing, brand positioning, and the agentic AI content series is not separate from the technical work — it is how the technical work becomes visible and valuable. A hardened platform with strong governance needs an equally strong narrative around it.

    Looking Ahead

    The open issues across the portfolio paint a clear picture of next week’s priorities: resolve the Telegram flood-control design, address the gateway ungraceful shutdown path, fix the Langfuse Cost Report cron’s stale API keys, and close the SSL certificate SAN mismatch. On the infrastructure side, the Mac AI Server Ollama dashboard and metrics pipeline need to land, and the remaining open PRs in the web-and-brand repository need to progress through review.

    The velocity of the last week is notable not for its drama but for its consistency. Every repository moved forward. No single event dominated. That is what a healthy development portfolio looks like in practice — not a single sprint, but sustained, disciplined progress across every layer of the stack.

  • What Multi-Agent Operations Actually Look Like in Practice

    What Multi-Agent Operations Actually Look Like in Practice

    Most organisations experimenting with AI agents are still operating them like a single chat window. Someone opens a prompt, asks the agent to do something, waits for the output, and moves on. That works for demos. It does not work when you are running agents against production systems or trying to get consistent results across a team.

    The gap is not technical. It is operational. The organisations getting genuine value from AI agents are not the ones with the most advanced models. They are the ones that figured out how to coordinate agents the way you would coordinate a team: clear roles, defined handoffs, checkpoint reviews, and someone accountable for the outcome.

    The Governance Gap Nobody Talks About

    The current wave of AI agent tooling is impressive. You can spin up an agent that writes code, another that reviews it, another that runs tests, and a fourth that deploys. The demos are compelling. The problem is that most organisations have not thought about what happens when these agents operate together at scale.

    Who coordinates them? What happens when two agents make conflicting changes? Where is the state stored, and who can inspect it? If an agent fails halfway through a task, what recovers? If an agent produces an incorrect output that another agent consumes, how do you trace the error back?

    These are the same questions you would ask about any multi-person production system. The difference is that agents do not have common sense, do not ask clarifying questions by default, and do not stop when something looks wrong unless you have built in the checks.

    The governance gap is this: most teams have moved from “can we run an agent?” to “we are running agents” without establishing the coordination layer in between.

    The Pattern That Actually Works

    After running autonomous coding agents in production for several months, the pattern that has proven reliable is a hybrid orchestration model. It has four parts.

    A coordinator role. One agent, or one human, owns the overall task. This role does not do the detailed work. It defines the objective, breaks it into independent subtasks, assigns each to a worker, and reviews the results. In practice, this is the role I occupy when running Hermes Agent, Claude Code, or Codex on a project. I set the direction, handle security decisions and state management, and delegate the pure coding work.

    Parallel worker agents. When subtasks are independent, they run simultaneously. Three agents working on three separate services at the same time complete in minutes what a single agent would handle sequentially in an hour. The key requirement is that the subtasks must be genuinely independent. If agent B depends on agent A’s output, running them in parallel creates conflicts, not speed.

    State machines for complex flows. When a task has sequential dependencies, a simple state machine prevents chaos. Each agent picks up the task at a defined state, does its work, writes output to a known location, and transitions the task forward. If an agent fails, the state does not advance. The next agent picks up the failed state and either retries or escalates.

    Checkpoint reviews. At defined points in the flow, a human reviews the output before the next stage begins. This is not a bottleneck. It is a safety mechanism. The review confirms that the output is sane, the state is correct, and the next stage has what it needs. In practice, these reviews take seconds when things are going well and save hours when they are not.

    A Concrete Example: Diagnosing Three Services at Once

    Suppose three independent services are exhibiting issues simultaneously. A traditional approach investigates them sequentially: diagnose service A, fix it, move to service B, fix it, move to service C.

    With a multi-agent setup, the coordinator defines the diagnostic task for each service and spins up three parallel subagents. Each agent gets the same instructions: examine the logs, identify the root cause, propose a fix, and write its findings to a shared state file. The agents do not communicate with each other. They do not need to. They are working on independent systems.

    When all three agents have completed their tasks, the coordinator reviews the findings, checks for conflicts (two agents proposing changes to a shared dependency, for example), and either approves the fixes or escalates for human review.

    A diagnostic process that would take a single engineer most of a day takes under thirty minutes. The quality is not lower — each agent focuses on a single problem without context-switching. The risk is not higher — the checkpoint review catches anything anomalous before it reaches production.

    This is not theoretical. It is a routine operational pattern that runs on free-tier models for the worker agents. The expensive model is the coordinator, and even that role can be handled by a human with a clear framework.

    The Cost Conversation

    There is a persistent misconception that running AI agents at scale requires expensive API subscriptions. In practice, the opposite is true. Worker agents doing diagnostics, code generation, and testing do not need frontier models. They need competent instruction-following, and that is available on free tiers or at very low cost.

    The coordinator role is where model quality matters. This is the agent making decisions about task decomposition, conflict resolution, and escalation. It needs to reason well. But there is only one coordinator, and it does relatively little token-heavy work compared to the workers.

    The cost structure in a well-designed multi-agent system is front-loaded into the coordination layer and minimal in the execution layer. You are paying for one good decision-maker and many cheap workers. The economics favour this model, which is one reason it works for cost-conscious organisations, not just well-funded ones.

    Failure recovery follows the same logic. When an agent fails on a free tier, the cost of retry is zero. When an agent fails on an expensive tier, every retry is a budget event. Putting cheap agents on high-volume work and the expensive agent on high-judgement work is not just an architectural decision. It is a cost optimisation.

    What Organisations Should Do Next

    If you are running or planning to run AI agents in production, the operational model matters more than model selection. Here is where to start.

    Define the coordinator role first. Decide whether a human or an agent owns task decomposition and review. Document what this role is responsible for and what decisions require escalation. This is your governance layer.

    Identify independent subtasks. Look at your current agent workflows and find the tasks that can run in parallel. Sequential workflows where tasks are independent are leaving time on the table.

    Build state into your workflows. Every agent should write its output to a known location in a known format. Every downstream agent should read from that location. If you cannot inspect workflow state at any point without replaying the entire execution, your state management is insufficient.

    Set checkpoint reviews at decision points. Not at every step — that defeats the purpose. At points where an incorrect output would propagate downstream and cause real damage. A review that takes five seconds and prevents a two-hour debugging session is time well spent.

    Use the right model for the right role. Do not pay frontier-model prices for tasks that a free-tier model handles competently. Reserve your budget for the coordination and review layers where reasoning quality directly affects outcomes.

    If your organisation is moving from AI experimentation to production agent operations, the coordination layer is where the value is — and where the risk lives. The AI & Automation Architecture service covers the design of multi-agent systems with proper governance, state management, and cost controls. Or get in touch for a conversation about what your agent operations should look like before they scale.

  • What Multi-Agent Operations Actually Look Like in Practice

    What Multi-Agent Operations Actually Look Like in Practice

    Most organisations experimenting with AI agents are still operating them like a single chat window. Someone opens a prompt, asks the agent to do something, waits for the output, and moves on. That works for demos. It does not work when you are running agents against production systems or trying to get consistent results across a team.

    The gap is not technical. It is operational. The organisations getting genuine value from AI agents are not the ones with the most advanced models. They are the ones that figured out how to coordinate agents the way you would coordinate a team: clear roles, defined handoffs, checkpoint reviews, and someone accountable for the outcome.

    The Governance Gap Nobody Talks About

    The current wave of AI agent tooling is impressive. You can spin up an agent that writes code, another that reviews it, another that runs tests, and a fourth that deploys. The demos are compelling. The problem is that most organisations have not thought about what happens when these agents operate together at scale.

    Who coordinates them? What happens when two agents make conflicting changes? Where is the state stored, and who can inspect it? If an agent fails halfway through a task, what recovers? If an agent produces an incorrect output that another agent consumes, how do you trace the error back?

    These are the same questions you would ask about any multi-person production system. The difference is that agents do not have common sense, do not ask clarifying questions by default, and do not stop when something looks wrong unless you have built in the checks.

    The governance gap is this: most teams have moved from “can we run an agent?” to “we are running agents” without establishing the coordination layer in between.

    The Pattern That Actually Works

    After running autonomous coding agents in production for several months, the pattern that has proven reliable is a hybrid orchestration model. It has four parts.

    A coordinator role. One agent, or one human, owns the overall task. This role does not do the detailed work. It defines the objective, breaks it into independent subtasks, assigns each to a worker, and reviews the results. In practice, this is the role I occupy when running Hermes Agent, Claude Code, or Codex on a project. I set the direction, handle security decisions and state management, and delegate the pure coding work.

    Parallel worker agents. When subtasks are independent, they run simultaneously. Three agents working on three separate services at the same time complete in minutes what a single agent would handle sequentially in an hour. The key requirement is that the subtasks must be genuinely independent. If agent B depends on agent A’s output, running them in parallel creates conflicts, not speed.

    State machines for complex flows. When a task has sequential dependencies, a simple state machine prevents chaos. Each agent picks up the task at a defined state, does its work, writes output to a known location, and transitions the task forward. If an agent fails, the state does not advance. The next agent picks up the failed state and either retries or escalates.

    Checkpoint reviews. At defined points in the flow, a human reviews the output before the next stage begins. This is not a bottleneck. It is a safety mechanism. The review confirms that the output is sane, the state is correct, and the next stage has what it needs. In practice, these reviews take seconds when things are going well and save hours when they are not.

    A Concrete Example: Diagnosing Three Services at Once

    Suppose three independent services are exhibiting issues simultaneously. A traditional approach investigates them sequentially: diagnose service A, fix it, move to service B, fix it, move to service C.

    With a multi-agent setup, the coordinator defines the diagnostic task for each service and spins up three parallel subagents. Each agent gets the same instructions: examine the logs, identify the root cause, propose a fix, and write its findings to a shared state file. The agents do not communicate with each other. They do not need to. They are working on independent systems.

    When all three agents have completed their tasks, the coordinator reviews the findings, checks for conflicts (two agents proposing changes to a shared dependency, for example), and either approves the fixes or escalates for human review.

    A diagnostic process that would take a single engineer most of a day takes under thirty minutes. The quality is not lower — each agent focuses on a single problem without context-switching. The risk is not higher — the checkpoint review catches anything anomalous before it reaches production.

    This is not theoretical. It is a routine operational pattern that runs on free-tier models for the worker agents. The expensive model is the coordinator, and even that role can be handled by a human with a clear framework.

    The Cost Conversation

    There is a persistent misconception that running AI agents at scale requires expensive API subscriptions. In practice, the opposite is true. Worker agents doing diagnostics, code generation, and testing do not need frontier models. They need competent instruction-following, and that is available on free tiers or at very low cost.

    The coordinator role is where model quality matters. This is the agent making decisions about task decomposition, conflict resolution, and escalation. It needs to reason well. But there is only one coordinator, and it does relatively little token-heavy work compared to the workers.

    The cost structure in a well-designed multi-agent system is front-loaded into the coordination layer and minimal in the execution layer. You are paying for one good decision-maker and many cheap workers. The economics favour this model, which is one reason it works for cost-conscious organisations, not just well-funded ones.

    Failure recovery follows the same logic. When an agent fails on a free tier, the cost of retry is zero. When an agent fails on an expensive tier, every retry is a budget event. Putting cheap agents on high-volume work and the expensive agent on high-judgement work is not just an architectural decision. It is a cost optimisation.

    What Organisations Should Do Next

    If you are running or planning to run AI agents in production, the operational model matters more than model selection. Here is where to start.

    Define the coordinator role first. Decide whether a human or an agent owns task decomposition and review. Document what this role is responsible for and what decisions require escalation. This is your governance layer.

    Identify independent subtasks. Look at your current agent workflows and find the tasks that can run in parallel. Sequential workflows where tasks are independent are leaving time on the table.

    Build state into your workflows. Every agent should write its output to a known location in a known format. Every downstream agent should read from that location. If you cannot inspect workflow state at any point without replaying the entire execution, your state management is insufficient.

    Set checkpoint reviews at decision points. Not at every step — that defeats the purpose. At points where an incorrect output would propagate downstream and cause real damage. A review that takes five seconds and prevents a two-hour debugging session is time well spent.

    Use the right model for the right role. Do not pay frontier-model prices for tasks that a free-tier model handles competently. Reserve your budget for the coordination and review layers where reasoning quality directly affects outcomes.

    If your organisation is moving from AI experimentation to production agent operations, the coordination layer is where the value is — and where the risk lives. The AI & Automation Architecture service covers the design of multi-agent systems with proper governance, state management, and cost controls. Or get in touch for a conversation about what your agent operations should look like before they scale.

  • What Multi-Agent Operations Actually Look Like in Practice

    What Multi-Agent Operations Actually Look Like in Practice

    Most organisations experimenting with AI agents are still operating them like a single chat window. Someone opens a prompt, asks the agent to do something, waits for the output, and moves on. That works for demos. It does not work when you are running agents against production systems or trying to get consistent results across a team.

    The gap is not technical. It is operational. The organisations getting genuine value from AI agents are not the ones with the most advanced models. They are the ones that figured out how to coordinate agents the way you would coordinate a team: clear roles, defined handoffs, checkpoint reviews, and someone accountable for the outcome.

    The Governance Gap Nobody Talks About

    The current wave of AI agent tooling is impressive. You can spin up an agent that writes code, another that reviews it, another that runs tests, and a fourth that deploys. The demos are compelling. The problem is that most organisations have not thought about what happens when these agents operate together at scale.

    Who coordinates them? What happens when two agents make conflicting changes? Where is the state stored, and who can inspect it? If an agent fails halfway through a task, what recovers? If an agent produces an incorrect output that another agent consumes, how do you trace the error back?

    These are the same questions you would ask about any multi-person production system. The difference is that agents do not have common sense, do not ask clarifying questions by default, and do not stop when something looks wrong unless you have built in the checks.

    The governance gap is this: most teams have moved from “can we run an agent?” to “we are running agents” without establishing the coordination layer in between.

    The Pattern That Actually Works

    After running autonomous coding agents in production for several months, the pattern that has proven reliable is a hybrid orchestration model. It has four parts.

    A coordinator role. One agent, or one human, owns the overall task. This role does not do the detailed work. It defines the objective, breaks it into independent subtasks, assigns each to a worker, and reviews the results. In practice, this is the role I occupy when running Hermes Agent, Claude Code, or Codex on a project. I set the direction, handle security decisions and state management, and delegate the pure coding work.

    Parallel worker agents. When subtasks are independent, they run simultaneously. Three agents working on three separate services at the same time complete in minutes what a single agent would handle sequentially in an hour. The key requirement is that the subtasks must be genuinely independent. If agent B depends on agent A’s output, running them in parallel creates conflicts, not speed.

    State machines for complex flows. When a task has sequential dependencies, a simple state machine prevents chaos. Each agent picks up the task at a defined state, does its work, writes output to a known location, and transitions the task forward. If an agent fails, the state does not advance. The next agent picks up the failed state and either retries or escalates.

    Checkpoint reviews. At defined points in the flow, a human reviews the output before the next stage begins. This is not a bottleneck. It is a safety mechanism. The review confirms that the output is sane, the state is correct, and the next stage has what it needs. In practice, these reviews take seconds when things are going well and save hours when they are not.

    A Concrete Example: Diagnosing Three Services at Once

    Suppose three independent services are exhibiting issues simultaneously. A traditional approach investigates them sequentially: diagnose service A, fix it, move to service B, fix it, move to service C.

    With a multi-agent setup, the coordinator defines the diagnostic task for each service and spins up three parallel subagents. Each agent gets the same instructions: examine the logs, identify the root cause, propose a fix, and write its findings to a shared state file. The agents do not communicate with each other. They do not need to. They are working on independent systems.

    When all three agents have completed their tasks, the coordinator reviews the findings, checks for conflicts (two agents proposing changes to a shared dependency, for example), and either approves the fixes or escalates for human review.

    A diagnostic process that would take a single engineer most of a day takes under thirty minutes. The quality is not lower — each agent focuses on a single problem without context-switching. The risk is not higher — the checkpoint review catches anything anomalous before it reaches production.

    This is not theoretical. It is a routine operational pattern that runs on free-tier models for the worker agents. The expensive model is the coordinator, and even that role can be handled by a human with a clear framework.

    The Cost Conversation

    There is a persistent misconception that running AI agents at scale requires expensive API subscriptions. In practice, the opposite is true. Worker agents doing diagnostics, code generation, and testing do not need frontier models. They need competent instruction-following, and that is available on free tiers or at very low cost.

    The coordinator role is where model quality matters. This is the agent making decisions about task decomposition, conflict resolution, and escalation. It needs to reason well. But there is only one coordinator, and it does relatively little token-heavy work compared to the workers.

    The cost structure in a well-designed multi-agent system is front-loaded into the coordination layer and minimal in the execution layer. You are paying for one good decision-maker and many cheap workers. The economics favour this model, which is one reason it works for cost-conscious organisations, not just well-funded ones.

    Failure recovery follows the same logic. When an agent fails on a free tier, the cost of retry is zero. When an agent fails on an expensive tier, every retry is a budget event. Putting cheap agents on high-volume work and the expensive agent on high-judgement work is not just an architectural decision. It is a cost optimisation.

    What Organisations Should Do Next

    If you are running or planning to run AI agents in production, the operational model matters more than model selection. Here is where to start.

    Define the coordinator role first. Decide whether a human or an agent owns task decomposition and review. Document what this role is responsible for and what decisions require escalation. This is your governance layer.

    Identify independent subtasks. Look at your current agent workflows and find the tasks that can run in parallel. Sequential workflows where tasks are independent are leaving time on the table.

    Build state into your workflows. Every agent should write its output to a known location in a known format. Every downstream agent should read from that location. If you cannot inspect workflow state at any point without replaying the entire execution, your state management is insufficient.

    Set checkpoint reviews at decision points. Not at every step — that defeats the purpose. At points where an incorrect output would propagate downstream and cause real damage. A review that takes five seconds and prevents a two-hour debugging session is time well spent.

    Use the right model for the right role. Do not pay frontier-model prices for tasks that a free-tier model handles competently. Reserve your budget for the coordination and review layers where reasoning quality directly affects outcomes.

    If your organisation is moving from AI experimentation to production agent operations, the coordination layer is where the value is — and where the risk lives. The AI & Automation Architecture service covers the design of multi-agent systems with proper governance, state management, and cost controls. Or get in touch for a conversation about what your agent operations should look like before they scale.

Powered by CookieAdmin