RichardHam.co.uk Blog

Tag: infrastructure

  • The Hidden Cost Structure of Multi-Model AI

    The Hidden Cost Structure of Multi-Model AI

    You set a budget. You picked a model. You shipped the feature. Then the bill arrived.

    If you are running AI at scale across a product or platform, you have probably noticed that costs do not behave the way you expected. The per-token price looked reasonable in the dashboard. The monthly total did not.

    The problem is not that AI is expensive. The problem is that the cost structure is hidden.

    Why this matters now

    The shift from single-model to multi-model architectures is well underway. Most serious AI-powered products now route between several providers: a frontier model for complex reasoning, a cheaper one for classification and extraction, maybe a local model for high-volume, low-judgement work.

    That architecture is the right move. But it introduces a cost surface that almost nobody tracks properly. Different models have wildly different pricing. Some tasks genuinely need GPT-4o or Claude Sonnet. Most do not. And without visibility, you end up paying frontier prices for commodity work.

    The organisations that figure this out early will have a structural advantage. The ones that do not will burn budget and wonder why.

    What is really going on

    There are four hidden cost drivers in most multi-model setups. They compound each other.

    1. Token waste

    Every call to an LLM carries context: system prompts, conversation history, tool definitions, retrieved documents. In a multi-agent or multi-step workflow, that context grows at each hop. It is common to see workflows where 70 to 80 percent of the tokens sent are redundant or repeated across steps.

    At scale, that waste is not trivial. A workflow running 10,000 times a day with an average of 4,000 input tokens per call, where half of those tokens are unnecessary, can easily cost hundreds of pounds per month in tokens that contributed nothing to the output.

    2. Model mismatch

    This is the single biggest lever. Using Claude Sonnet for a task that Llama 3.2 handles just as well is a 10x cost difference for no quality gain. Using GPT-4o to classify support tickets is not just expensive, it is architecturally unnecessary.

    The pattern I see repeatedly: teams default to the most capable model available, then never revisit that decision once the feature is in production. The model that was right for prototyping is not necessarily right for day 10,000 of production.

    3. Lack of visibility

    Most AI billing is provider-level. You can see what you spent on OpenAI versus Anthropic versus a smaller provider. What you cannot see, without deliberate instrumentation, is the cost per workflow, per agent, per feature, or per team.

    That means nobody can answer basic questions. Which feature is the most expensive to run? Which team’s AI usage is growing fastest? Which workflow has the worst token-to-value ratio? Without per-workflow cost tracking, you are managing AI spend with a monthly credit card statement.

    4. API sprawl

    When multiple teams adopt AI independently, you end up with overlapping credentials, redundant integrations, and no centralised billing. One team uses OpenAI directly. Another wraps it in a different SDK. A third spins up a separate provider for a narrow use case. Each team optimises locally. Nobody optimises globally.

    This is not a technology problem. It is a governance gap, and it is expensive.

    What to do next

    The good news is that the solution pattern is well understood and does not require ripping out what you have already built.

    1. Route cheap tasks to free or local models.

    If a task does not require frontier reasoning, run it locally. Tools like Ollama make this practical for classification, extraction, summarisation, and other high-volume, low-judgement work. The cost difference is not marginal. It is the difference between paying per token and paying nothing beyond your compute.

    2. Reserve expensive models for high-judgement work.

    Define clear criteria for when a task warrants a frontier model. Complex reasoning, nuanced language generation, multi-step planning, and safety-critical decisions qualify. Everything else should default to the cheapest model that meets your quality bar.

    3. Track costs per workflow, not just per provider.

    Instrument your routing layer to log token usage and cost at the workflow or agent level. This does not require a complex platform. Even a lightweight middleware that tags each request with a workflow identifier and records input/output token counts will give you the visibility you need.

    4. Set budget alerts before you need them.

    Per-workflow budgets with hard and soft alerts let you catch runaway costs before they become line items in a board report. A soft alert at 70 percent of budget gives you time to investigate. A hard cap at 100 percent protects you from the unexpected.

    5. Audit your context.

    Review the prompts and context you are sending at each step of every workflow. Remove redundant system prompts, collapse repeated context, and set maximum token limits. This is the fastest way to reduce waste without changing any model choices.

    6. Centralise provider management.

    One set of credentials. One billing view. One team responsible for provider relationships and rate negotiations. This is not about slowing teams down. It is about making sure the organisation can see and manage what it is spending.

    Where Richard can help

    I work with technical leaders who are building AI-powered products and need the cost structure to be as well-designed as the architecture. That means routing strategies, cost tracking, and practical governance that does not get in the way of delivery.

    If multi-model cost management is on your roadmap, get in touch or review the relevant services.

  • Why AI Control Planes Need Observability From Day One

    Why AI Control Planes Need Observability From Day One

    AI agents are getting plugged into email, CRM, finance, and document systems faster than most teams can build the operations around them.

    That is the mismatch.

    The agent might be fine. The prompt might be fine. The integration might even work. What is often missing is the layer that says who can do what, what happened, what it cost, and how the system recovers when something breaks.

    What the control plane does

    Think of it as the operational wrapper around the agent. It handles:

    • authentication and scoped access
    • structured logging
    • approval gates for risky actions
    • observability and cost tracking
    • failure recovery

    That is not overkill. That is the minimum if the agent is touching production data.

    A concrete example

    Imagine a workflow that reads a form, enriches a record, updates a CRM, and sends a confirmation email.

    Without the wrapper, you get a static key, weak logs, and no real visibility when something goes wrong.

    With it, you get scoped access, review points, full traces, and a clearer answer when someone asks, “What did the system actually do?”

    Why day one matters

    Retrofitting this later is painful. It means backfilling logs, reworking workflows, and explaining to people why the guardrails arrived after the risk did.

    It is much easier to build the control plane alongside the agents and treat it as part of the product, not a patch.

    What to do first

    1. List every agent and workflow that can touch a real system.
    2. Log the full chain of inputs, decisions, and actions.
    3. Add human approval where the blast radius is high.
    4. Track failures and costs as first-class signals.

    That is enough to start building something you can actually trust.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • GitHub Weekly — Inventory Reconciliation, Safer Automation, and Pilot Delivery

    GitHub Weekly — Inventory Reconciliation, Safer Automation, and Pilot Delivery

    When I reviewed this week’s GitHub activity, one pattern kept showing up across very different repos: the work was not really about adding more moving parts. It was about making the existing parts easier to trust.

    That showed up in infrastructure work, in the agent and governance layer, in product scaffolding, and even in website and brand updates. The common thread was operational credibility. Not “can this be built?” but “can this be run, understood, and improved without guesswork?”

    I think that distinction matters more than most teams admit. Plenty of systems can be made to work for a day. Far fewer are built to survive handovers, edge cases, and the quiet failure modes that only show up once the initial excitement wears off.

    What happened

    1. Inventory and infrastructure work moved from assumptions to reconciliation

    The clearest technical thread this week sat in the infrastructure estate.

    A cluster of commits and pull requests focused on inventory reconciliation, NetBox alignment, deployment timers, backup coverage, and preserving state correctly during synchronisation. The details matter here because they point to a mature kind of problem.

    This was not “set up monitoring” or “add a backup.” It was more specific than that:

    • preserving existing custom fields during sync instead of bluntly overwriting them
    • wiring host variables so the live inventory reflects the real estate more faithfully
    • adding a reconciliation timer so drift is checked regularly rather than relying on memory
    • tightening the documentation around port management and incident handling
    • adding backup paths around Git hosting and PostgreSQL exports so recovery is not left to best intentions

    That is serious operational work.

    A lot of teams stop once the first integration works. But once you have lived with an estate for a while, the harder problem is not connectivity — it is fidelity. Does your inventory still describe reality? Does your synchronisation preserve the parts of the system that humans added for a reason? Do your backups exist as a runnable path, not just a sentence in a plan?

    I also noticed a Terraform validation gate land in the same broader operating context. Again, that is a small change on paper, but it says something useful about the direction of travel: the systems are being nudged toward earlier feedback and fewer silent mistakes.

    That is usually a good sign. Mature platforms do not just automate more; they fail sooner and more visibly.

    2. Safer automation is becoming a design principle rather than a patch

    A second pattern was the continued tightening of automation boundaries.

    In the management and agent repos, the work touched cron behaviour, gateway restart safety, regression coverage, secret-scanning governance, prompt and model hygiene, and more explicit handling of runtime assumptions. There was also activity around daily “Decision Desk” issues and weekly cost rollups, which reinforces the sense that operational review is becoming a routine surface rather than an occasional scramble.

    What stood out to me was not any one fix in isolation. It was the posture behind them.

    The posture seems to be:

    • make hidden dependencies visible
    • stop false-green checks from looking healthy when they are not
    • separate human-only actions from safe automation paths
    • keep governance records close to the implementation work
    • add tests around the boundaries that matter most

    That is the right instinct for any agentic or semi-autonomous system.

    There is a temptation in AI and automation work to obsess over capability and underinvest in control. But the systems that earn trust over time are usually the opposite. They may look less flashy at first, but they are the ones people keep using because the failure modes are legible.

    I often find that the best progress in these environments comes from boring-sounding work: a better guard, a clearer runbook, a fix that prevents a check from hiding a broken path, or a cleaner boundary between what the machine can do alone and what still needs a person.

    That kind of work compounds.

    3. New product and pilot work is being framed with real operational shape from the start

    There was also a healthy amount of activity around new product and pilot work.

    One stream built out an AI consultancy-oriented assessment flow with issue scaffolding for the API, persistence, report generation, visitor-safe rendering, lead notifications, privacy controls, and admin protection. Another stream pushed a pilot roadmap forward with legal review notes, request packs, costing artefacts, rehearsal runbooks, and status-gate updates.

    This is the sort of work I like to see early.

    It suggests the projects are not being treated as presentation-layer exercises. They are being built with the surrounding machinery in mind:

    • how the workflow stores and protects data
    • how output gets generated with fallbacks
    • what supporting documents are needed before a pilot becomes real
    • what commercial and legal edges need handling before delivery starts
    • what a rehearsal path looks like before someone is relying on it

    That is a much stronger way to start an AI project than simply chasing a polished demo.

    The same practical mindset also showed up in the website work. The brand alignment and navigation adjustments in the main site repo, along with the redesign and deployment handover work in a separate website project, both point to an important truth: delivery is never just code. It is also handover, consistency, content structure, and operational clarity once the thing is live.

    Why this week matters

    What connects all of this is a shift from implementation to operability.

    I do not mean that the build phase is over. Clearly it is not. There is still plenty being created. But the work is increasingly shaped by questions like:

    • Can this system survive drift?
    • Can somebody else understand the current state quickly?
    • Can an automated path be trusted not to hide the real failure?
    • Can a pilot be delivered without inventing the commercial and governance pieces at the last minute?
    • Can the visible front end stay aligned with the operational reality behind it?

    Those questions are where systems start becoming durable.

    They are also where a lot of technical teams quietly win or lose time. If you skip them, you pay later through rework, brittle deployments, unclear ownership, and incident response that starts with archaeology. If you handle them early, the platform becomes easier to change because it is easier to reason about.

    Key takeaways

    A few practical lessons came through clearly this week.

    • Reconciliation beats assumption. A live inventory is only useful if it keeps matching reality. Sync jobs and timers are not admin overhead; they are how trust is maintained.
    • State preservation matters as much as state collection. It is not enough to ingest live data if the process wipes the context humans added deliberately.
    • Guard rails are product work. In agent and automation systems, restart safety, explicit boundaries, and truthful checks are not secondary concerns.
    • Pilots need legal and operational scaffolding early. Rehearsal runbooks, request packs, privacy controls, and delivery notes are signs of seriousness, not bureaucracy.
    • Good delivery includes the handover path. Website and product work both improve when documentation, navigation, and deployment steps are treated as first-class.

    If I had to reduce the whole week to one line, it would be this: the strongest systems in the batch were the ones being made easier to trust, not merely easier to demo.

    Closing thought

    This week’s most interesting GitHub activity was not one dramatic launch. It was the repeated decision to replace ambiguity with structure.

    That happened in infrastructure reconciliation, in safer automation boundaries, in early-stage product scaffolding, and in content and website delivery work. Each change on its own might look incremental. Together, they point in a useful direction: systems that are easier to operate, easier to hand over, and harder to misunderstand.

    That is the sort of progress I pay attention to.

    If you are building AI workflows, internal tooling, or customer-facing systems and want them to be robust as well as impressive, that is exactly the kind of work I help with through services and more focused advisory conversations via contact.

  • What Hardening a Production Server Actually Looks Like

    What Hardening a Production Server Actually Looks Like

    There’s a version of “server hardening” that exists in compliance documents: a tidy checklist, a one-time audit, a box ticked. Then there’s what it actually looks like in production — messy, iterative, and never quite finished.

    I’ve spent the last several weeks hardening infrastructure across multiple environments, and the pattern is always the same. What was secure at deployment drifts. Defaults get forgotten. Services get bolted on. And suddenly the thing you trusted is quietly doing something you never intended.

    Here’s what the real work looks like.

    The Problem Nobody Warns You About: Configuration Drift

    You deploy a server with a locked-down SSH config, a clean firewall, and sensible defaults. Three months later, someone adds a web server, opens a port for debugging, and forgets to close it. Or an Nginx vhost gets added without an explicit server_name, and the default catch-all starts serving the wrong site to the wrong visitors.

    This isn’t hypothetical. I recently found a production edge server where the Nginx default_server block was silently intercepting requests meant for a different virtual host. The site was working — but it was serving the wrong content to a subset of visitors. No errors in the logs. No alerts. Just quiet, invisible misconfiguration that had been running who knows how long.

    The fix was straightforward: explicit server_name directives on every vhost, and removing the catch-all entirely. But finding it required actually looking, which is the part that doesn’t happen often enough.

    SSH Access With a Dynamic IP: The Failsafe Pattern

    Here’s another one that comes up constantly. You lock SSH down to specific source IPs — best practice, absolutely. But your home IP is assigned by your ISP via DHCP, and it changes. Now you’re locked out of your own server, or you’re tempted to leave the firewall wide open “just until I update it.”

    The solution I implemented on a production VPS was an automated failsafe script. It runs periodically, detects the current public IP, and updates the firewall allowlist if the IP has changed. The key design principles:

    • The script authenticates outbound — it calls a known endpoint to discover the current IP, then pushes the update.
    • It only modifies the specific allowlist rule — it doesn’t touch any other firewall configuration.
    • It logs every change — so there’s an audit trail of when and why the IP was updated.
    • It fails closed — if the script can’t determine the current IP, it doesn’t open anything up.

    This pattern means you get the security of IP-restricted SSH without the operational risk of locking yourself out when your ISP rotates your address. It’s not fancy. It’s just honest about the reality that infrastructure has to be operable by humans.

    Multi-Hop SSH: Key Management Across Trust Boundaries

    The more complex the infrastructure, the more carefully you need to think about SSH key propagation. In one environment, the access path runs through multiple hops: local machine to management host, management host to container runtime, container runtime to VPS.

    Each hop is a trust boundary. The question at each one is: what keys exist here, who can use them, and what happens if this host is compromised?

    The hardening approach:

    • Separate keys per hop — no single key traverses the entire chain. If one host is compromised, the blast radius is limited to the next hop, not the entire path.
    • Keys are never copied manually — they’re provisioned through automation, with expiry and rotation baked in.
    • Agent forwarding is scoped, not blanket — it’s enabled only for specific connections and disabled by default.
    • Every key has a known owner and purpose — if you can’t explain why a key exists, it gets removed.

    This is the kind of thing that feels excessive until the day it prevents a lateral move during an incident. Then it feels like the most important work you did.

    The Hardening Checklist You Can Actually Use

    If you’re responsible for production infrastructure — whether you’re a CTO, a technical founder, or the person who just ended up owning the servers — here’s a practical checklist drawn from real hardening work:

    1. Audit your firewall rules quarterly. Every rule should have a comment explaining why it exists. If you can’t explain it, remove it.
    2. Remove default_server catch-alls from web servers. Every vhost should have an explicit server_name. If a request doesn’t match a known vhost, it should get a 444 or a meaningful error — not silently served by the wrong site.
    3. Automate your SSH access management. If your source IP can change, build the failsafe before you get locked out, not after.
    4. Segment your SSH keys by trust boundary. One key per hop. No exceptions for convenience.
    5. Review listening services monthly. Run ss -tlnp and verify every open port is intentional. You will find surprises.
    6. Check for configuration drift after every change. The best time to catch a misconfiguration is right after someone made a different change nearby.
    7. Log access and changes. If you can’t tell who connected, when, and what they changed, you don’t have a hardened server — you have a shared secret.

    The Uncomfortable Truth

    Hardening isn’t a project with a finish line. It’s a discipline. The servers that get compromised aren’t usually the ones that were never hardened — they’re the ones that were hardened once and then left to drift.

    The work is unglamorous. It’s reviewing firewall rules on a Friday afternoon. It’s removing a default_server block that “seems to be working.” It’s writing a small script so your SSH access doesn’t break when your ISP does its thing.

    But this is the work that keeps production infrastructure trustworthy. And it’s the work that separates infrastructure that looks secure from infrastructure that is secure.

    If this is on your roadmap — or if you’d rather someone else owned it — get in touch or review the relevant services.

  • Weekly GitHub Activity — 22–29 June 2026

    Introduction

    Another busy week in the repositories. Between infrastructure automation, platform hardening, commercial product development, and a small mountain of documentation, the commit logs tell the story of a systemthat’s rapidly moving from “hand-crafted and held together with SSH sessions” toward something far more repeatable and governed. Let’s walk through what landed.

    What Happened

    hermes-mgmt: Hardening, Governance, and AI Spend Controls

    The hermes-mgmt repository saw 10 commits and 20 pull requests this week, almost all of them orbiting two themes: Tier 2 operational hardening and AI spend governance.

    On the hardening side, a suite of operator runbooks and governance records landed — documenting the execution of Tier 2 hardening scripts, session decisions from backlog triage, and a full operator TODO instruction document. PR #694 tackled memory infrastructure head-on, covering a Qdrant deployment, gateway shutdown procedures, and Telegram 429 rate-limit handling. PR #695 addressed bug remediation across Spotify integration, n8n authentication, vision/video backend issues, and a 429 retry storm that was causing more harm than good.

    The AI spend work is particularly interesting. PR #696 introduced budget policies, cost telemetry, context-optimisation strategies, and prompt-caching controls — essentially an AI spend optimisation control plane. As LLM-powered workloads grow, having a formal mechanism to track, cap, and optimise spend is no longer nice-to-have; it’s operational hygiene. The design docs for child components that accompanied this work sketch out a system where every agent invocation carries a cost context, and budget alarms fire before the invoice does.

    Security compliance also got attention. PR #691 documented progress against SecureScore controls SEC-002, SEC-006, RES-002, and MAT-001 — a reminder that even in a solo-operator environment, treating compliance as code keeps the audit trail honest.

    A quieter but important fix: PR #640 made the Mem0→Letta memory backfill idempotent, preventing duplicate records on re-runs. Small change, big impact on data integrity.

    hamnet: The Control Plane Takes Shape

    hamnet was the most active repository this week — 10 commits, 18 pull requests, and 40 open issues. The headline is the emergence of a single-pane control-plane dashboard (PR #183), backed by a growing collection of infrastructure collectors.

    The control-plane work is being built incrementally and deliberately:

    • PR #177 established the identity model, a static inventory MVP, and the first batch of workload collectors.
    • PR #179 added a second batch — AgentRadar, Ollama status, repository state, and NetBox integration.
    • PR #176 fixed Grafana drift authentication and a tvheadend timeout issue that was causing monitoring gaps.

    On the infrastructure-as-code front, PR #178 delivered a comprehensive IaC roadmap with schemas, runbooks, and a system-classification model. PR #194 proposed enforcing unattended-upgrades policies on external cloud hosts through Ansible — a defensive measure that ensures security patches land even when nobody’s watching.

    Several fixes made day-to-day operations smoother: a script to tag missing VMs with ansible:managed in NetBox (commit 33d2fd9), a UTF-8 locale fix for ansible-inventory (commit 198d9b8), and adding Gitea to the Docker container detection logic (commit a950c67) — a gap that meant the monitoring stack was blind to a self-hosted service.

    A Docker image drift management runbook (commit 81fa372) and operator runbooks for 11 infrastructure issues (commit 8f09076) round out the picture. hamnet is building the operational backbone: not just running services, but knowing what’s running, whether it should be, and what to do when it isn’t.

    The 40 open issues deserve a mention — they’re a candid inventory of what needs attention, from an Oracle scanner that’s out of sync and a PVE exporter reporting zero metrics to Loki missing log-shipping and stale NetBox entries. Filing them is half the battle.

    ms365-agentic-ai: Collectors, Cost Controls, and Production Readiness

    The ms365-agentic-ai repository is accelerating. Five commits this week laid significant groundwork:

    • A deployable Azure Functions host with a run-history sink and live-path tests (commit 5465364).
    • Live read-only fetch wiring for all collectors (commit 416c537).
    • A cost-management component combining an estimator with an Azure cost collector (commit 86ab202).
    • Production-readiness runbook with dry-run and preflight tooling (commit 5da09c4).
    • A simple start guide, docs index, and FAQ/troubleshooting section (commit 212729a).

    The 10 open issues (#6–#15) are essentially the product roadmap: approval-gated remediation models, audit logging, Teams health digests, Azure resource health and Defender XDR collectors, Graph permissions matrices, and licensing/cost matrices. What’s notable is the breadth — this isn’t just a tool, it’s an operational platform being designed with governance from the start.

    richardham-co-uk-ConsultancyOS: Commercial Foundations

    ConsultancyOS — the commercial operating-system template — got its infrastructure in order this week. PR #12 added CI, licensing, a contributing guide, and a docs handbook. A SessionStart hook for web sessions (commit 1a54d22) shows the product thinking extending into the user experience.

    The 9 open issues map out the productisation journey: operational dashboards, a GitHub delivery-repo template, LLM integration, n8n workflow scaffolding, accountant review processes, company identity, and integrations with Dolibarr CRM and Invoice Ninja. This is a project that’s explicitly moving from “personal tooling” toward “repeatable commercial offering.”

    on-maintenance-ai-roadmap: Client-Facing Delivery

    The on-maintenance-ai-roadmap repository focused on communication and deployment model this week. A client-facing project plan (PR #23), an Architecture Decision Record for production hosting in client-owned Azure (PR #21), and a deploy-from-GitHub infrastructure model (PR #24) all point to a delivery pattern where the client retains ownership and control of their environment while the project provides the automation and intelligence layer.

    ai-cloud-credits-grant: Cloud Funding Applications

    Six commits across the ai-cloud-credits-grant repository added usage plans for NVIDIA Inception, Cloudflare, Oracle OCI, AWS, and Google Cloud. These are structured applications for cloud-provider credit programmes — the kind of unglamorous but essential work that unlocks compute capacity for projects that need it. The 6 open issues track the remaining work: a master company profile, applications to the major providers, and an evidence pack.

    Key Takeaways

    1. The control plane is real. hamnet’s dashboard, collectors, and identity model aren’t prototypes — they’re an incremental build toward a system that provides visibility and governance across the entire infrastructure. When you can see everything from one pane, you can fix things before they break.

    2. AI spend is now a first-class concern. The hermes-mgmt AI spend control plane — with budget policies, cost telemetry, and prompt caching — reflects a mature approach to LLM operations. The era of “just call the API and hope the bill is reasonable” is over, at least in this stack.

    3. Documentation as a delivery artifact. Across every repository this week, documentation wasn’t an afterthought — it was a first-class deliverable. Operator runbooks, governance records, ADRs, client-facing plans, and onboarding guides shipped alongside the code. This is the behaviour that separates sustainable projects from weekend hacks.

    4. Compliance isn’t optional, even for small teams. The SecureScore work in hermes-mgmt and the approval-gated remediation model in ms365-agentic-ai show that compliance and auditability are being designed in, not bolted on. That’s cheaper and more effective at any scale.

    5. Open issues are a feature, not a bug. hamnet’s 40 open issues and ms365-agentic-ai’s 10 open issues aren’t backlogs — they’re transparent roadmaps. Filing what needs doing, and making it visible, is how solo operators avoid the “I’ll remember that later” trap.

    This post is part of a weekly series summarising GitHub activity across the project portfolio. Previous entries are available in the blog archive.

  • Weekly GitHub Activity — 22–29 June 2026

    Introduction

    Another busy week in the repositories. Between infrastructure automation, platform hardening, commercial product development, and a small mountain of documentation, the commit logs tell the story of a systemthat’s rapidly moving from “hand-crafted and held together with SSH sessions” toward something far more repeatable and governed. Let’s walk through what landed.

    What Happened

    hermes-mgmt: Hardening, Governance, and AI Spend Controls

    The hermes-mgmt repository saw 10 commits and 20 pull requests this week, almost all of them orbiting two themes: Tier 2 operational hardening and AI spend governance.

    On the hardening side, a suite of operator runbooks and governance records landed — documenting the execution of Tier 2 hardening scripts, session decisions from backlog triage, and a full operator TODO instruction document. PR #694 tackled memory infrastructure head-on, covering a Qdrant deployment, gateway shutdown procedures, and Telegram 429 rate-limit handling. PR #695 addressed bug remediation across Spotify integration, n8n authentication, vision/video backend issues, and a 429 retry storm that was causing more harm than good.

    The AI spend work is particularly interesting. PR #696 introduced budget policies, cost telemetry, context-optimisation strategies, and prompt-caching controls — essentially an AI spend optimisation control plane. As LLM-powered workloads grow, having a formal mechanism to track, cap, and optimise spend is no longer nice-to-have; it’s operational hygiene. The design docs for child components that accompanied this work sketch out a system where every agent invocation carries a cost context, and budget alarms fire before the invoice does.

    Security compliance also got attention. PR #691 documented progress against SecureScore controls SEC-002, SEC-006, RES-002, and MAT-001 — a reminder that even in a solo-operator environment, treating compliance as code keeps the audit trail honest.

    A quieter but important fix: PR #640 made the Mem0→Letta memory backfill idempotent, preventing duplicate records on re-runs. Small change, big impact on data integrity.

    hamnet: The Control Plane Takes Shape

    hamnet was the most active repository this week — 10 commits, 18 pull requests, and 40 open issues. The headline is the emergence of a single-pane control-plane dashboard (PR #183), backed by a growing collection of infrastructure collectors.

    The control-plane work is being built incrementally and deliberately:

    • PR #177 established the identity model, a static inventory MVP, and the first batch of workload collectors.
    • PR #179 added a second batch — AgentRadar, Ollama status, repository state, and NetBox integration.
    • PR #176 fixed Grafana drift authentication and a tvheadend timeout issue that was causing monitoring gaps.

    On the infrastructure-as-code front, PR #178 delivered a comprehensive IaC roadmap with schemas, runbooks, and a system-classification model. PR #194 proposed enforcing unattended-upgrades policies on external cloud hosts through Ansible — a defensive measure that ensures security patches land even when nobody’s watching.

    Several fixes made day-to-day operations smoother: a script to tag missing VMs with ansible:managed in NetBox (commit 33d2fd9), a UTF-8 locale fix for ansible-inventory (commit 198d9b8), and adding Gitea to the Docker container detection logic (commit a950c67) — a gap that meant the monitoring stack was blind to a self-hosted service.

    A Docker image drift management runbook (commit 81fa372) and operator runbooks for 11 infrastructure issues (commit 8f09076) round out the picture. hamnet is building the operational backbone: not just running services, but knowing what’s running, whether it should be, and what to do when it isn’t.

    The 40 open issues deserve a mention — they’re a candid inventory of what needs attention, from an Oracle scanner that’s out of sync and a PVE exporter reporting zero metrics to Loki missing log-shipping and stale NetBox entries. Filing them is half the battle.

    ms365-agentic-ai: Collectors, Cost Controls, and Production Readiness

    The ms365-agentic-ai repository is accelerating. Five commits this week laid significant groundwork:

    • A deployable Azure Functions host with a run-history sink and live-path tests (commit 5465364).
    • Live read-only fetch wiring for all collectors (commit 416c537).
    • A cost-management component combining an estimator with an Azure cost collector (commit 86ab202).
    • Production-readiness runbook with dry-run and preflight tooling (commit 5da09c4).
    • A simple start guide, docs index, and FAQ/troubleshooting section (commit 212729a).

    The 10 open issues (#6–#15) are essentially the product roadmap: approval-gated remediation models, audit logging, Teams health digests, Azure resource health and Defender XDR collectors, Graph permissions matrices, and licensing/cost matrices. What’s notable is the breadth — this isn’t just a tool, it’s an operational platform being designed with governance from the start.

    richardham-co-uk-ConsultancyOS: Commercial Foundations

    ConsultancyOS — the commercial operating-system template — got its infrastructure in order this week. PR #12 added CI, licensing, a contributing guide, and a docs handbook. A SessionStart hook for web sessions (commit 1a54d22) shows the product thinking extending into the user experience.

    The 9 open issues map out the productisation journey: operational dashboards, a GitHub delivery-repo template, LLM integration, n8n workflow scaffolding, accountant review processes, company identity, and integrations with Dolibarr CRM and Invoice Ninja. This is a project that’s explicitly moving from “personal tooling” toward “repeatable commercial offering.”

    on-maintenance-ai-roadmap: Client-Facing Delivery

    The on-maintenance-ai-roadmap repository focused on communication and deployment model this week. A client-facing project plan (PR #23), an Architecture Decision Record for production hosting in client-owned Azure (PR #21), and a deploy-from-GitHub infrastructure model (PR #24) all point to a delivery pattern where the client retains ownership and control of their environment while the project provides the automation and intelligence layer.

    ai-cloud-credits-grant: Cloud Funding Applications

    Six commits across the ai-cloud-credits-grant repository added usage plans for NVIDIA Inception, Cloudflare, Oracle OCI, AWS, and Google Cloud. These are structured applications for cloud-provider credit programmes — the kind of unglamorous but essential work that unlocks compute capacity for projects that need it. The 6 open issues track the remaining work: a master company profile, applications to the major providers, and an evidence pack.

    Key Takeaways

    1. The control plane is real. hamnet’s dashboard, collectors, and identity model aren’t prototypes — they’re an incremental build toward a system that provides visibility and governance across the entire infrastructure. When you can see everything from one pane, you can fix things before they break.

    2. AI spend is now a first-class concern. The hermes-mgmt AI spend control plane — with budget policies, cost telemetry, and prompt caching — reflects a mature approach to LLM operations. The era of “just call the API and hope the bill is reasonable” is over, at least in this stack.

    3. Documentation as a delivery artifact. Across every repository this week, documentation wasn’t an afterthought — it was a first-class deliverable. Operator runbooks, governance records, ADRs, client-facing plans, and onboarding guides shipped alongside the code. This is the behaviour that separates sustainable projects from weekend hacks.

    4. Compliance isn’t optional, even for small teams. The SecureScore work in hermes-mgmt and the approval-gated remediation model in ms365-agentic-ai show that compliance and auditability are being designed in, not bolted on. That’s cheaper and more effective at any scale.

    5. Open issues are a feature, not a bug. hamnet’s 40 open issues and ms365-agentic-ai’s 10 open issues aren’t backlogs — they’re transparent roadmaps. Filing what needs doing, and making it visible, is how solo operators avoid the “I’ll remember that later” trap.

    This post is part of a weekly series summarising GitHub activity across the project portfolio. Previous entries are available in the blog archive.

Powered by