RichardHam.co.uk Blog

Tag: observability

  • The Case for Explicit Policies

    The Case for Explicit Policies

    Reliable systems do not emerge from good intentions. They emerge when the rules are explicit enough that another operator can understand what the system is supposed to do without reverse-engineering its behaviour from the wreckage.

    That sounds obvious, but it is still one of the most common gaps I see in automation and AI work. Teams build the workflow, connect the services, and get something working end to end. Then they leave the important decisions half-stated. Which provider is preferred? When should the fallback fire? What counts as a real health check? Which version of a process note is authoritative? The system may run, but the operating model is still fuzzy.

    The problem is not that people are careless. It is that policy work often looks less urgent than delivery work. Until something breaks, the invisible rule feels good enough.

    Where ambiguity shows up first

    The first place ambiguity appears is usually routing.

    A stack with multiple models, providers, queues, or execution paths always contains policy whether the team writes it down or not. If the preferred provider is too expensive for low-value tasks, that is policy. If one model is allowed for drafting but not for final output, that is policy. If a workflow should fall back only on timeout and not on quality failure, that is policy too.

    When none of that is written down clearly, people start inferring intent from whatever happened last.

    That is how teams end up with arguments that sound technical but are really operational:

    • “I thought the cheaper path was the default.”
    • “I assumed the fallback only applied during outages.”
    • “I didn’t realise this job was meant to stay on the private model.”
    • “I thought the dashboard alert meant the workflow had already rerouted.”

    None of those are bugs in isolation. They are symptoms of unstated policy.

    Why observability is part of policy

    The same issue appears in monitoring.

    A lot of dashboards tell you that a process is alive. That is not the same as proving the service is doing the right thing.

    For AI and automation systems, a truthful check usually needs to answer something more useful:

    • did the workflow complete the task it was supposed to complete?
    • did it use the intended path?
    • did it return data that looks structurally valid?
    • did the fallback stay dormant when the primary path was healthy?
    • can the operator see enough detail to explain the outcome afterward?

    If the check cannot answer those questions, the dashboard may still be visually tidy, but it is not giving the operator what they need.

    This is why I think observability should be treated as policy, not just instrumentation. The team has to decide what “working” actually means. Otherwise the monitoring layer simply reflects a vague assumption instead of a deliberate standard.

    Reuse is how policy survives handover

    The other quiet benefit of explicit policy is reuse.

    If a team has to rediscover the same routing rule, the same rollback sequence, or the same publishing checklist every time, then the policy is not really part of the system yet. It still lives in memory.

    That is expensive in a small team and dangerous in a growing one.

    Good reuse does not have to be elaborate. Often it is just a set of plain habits:

    • keep one canonical source of truth for important workflows
    • write fallback conditions near the implementation
    • keep short runbooks for the obvious failure modes
    • use the same naming and review patterns across similar jobs
    • record decisions before context evaporates

    None of that feels exciting while you are doing it. But it changes the quality of handover completely. A new operator no longer has to absorb the entire history of the stack before they can act safely.

    What explicit policy looks like in practice

    In practical terms, I look for a few simple signals.

    1. The preferred path is obvious

    The system should make it clear what happens first, what happens second, and under which conditions the fallback is allowed to take over.

    2. The checks reflect user reality

    A green dashboard should mean more than “something is listening on a port”. It should tell the operator whether the real job still works.

    3. Recovery paths exist before the incident

    If the first time a team documents the rollback sequence is during a failure, the policy work happened too late.

    4. Repeated patterns are actually reusable

    If the same kind of workflow appears three times, there should be a shared pattern instead of three slightly different tribal versions.

    Why this matters more with AI systems

    AI systems raise the cost of ambiguity because they turn a fuzzy rule into machine-speed inconsistency.

    In a manual process, unclear policy wastes time. In an automated one, it can silently change outputs, route work to the wrong provider, or create a trail that is too vague to audit later.

    That is why I think trustworthy AI is less about magic prompts and more about explicit operating rules.

    If the rules matter, write them down.

    If the outcome matters, check the real behaviour.

    If the workflow repeats, make it reusable.

    That does not make the system flashy. It makes it dependable.

    And in production, dependable usually wins.

    If you are building automation that needs to survive handover, escalation, and real operational scrutiny, the AI & Automation Architecture work is designed for exactly that. Or get in touch if you want a second pair of eyes on the operating model before the ambiguity becomes expensive.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Why AI Control Planes Need Observability From Day One

    Why AI Control Planes Need Observability From Day One

    AI agents are getting plugged into email, CRM, finance, and document systems faster than most teams can build the operations around them.

    That is the mismatch.

    The agent might be fine. The prompt might be fine. The integration might even work. What is often missing is the layer that says who can do what, what happened, what it cost, and how the system recovers when something breaks.

    What the control plane does

    Think of it as the operational wrapper around the agent. It handles:

    • authentication and scoped access
    • structured logging
    • approval gates for risky actions
    • observability and cost tracking
    • failure recovery

    That is not overkill. That is the minimum if the agent is touching production data.

    A concrete example

    Imagine a workflow that reads a form, enriches a record, updates a CRM, and sends a confirmation email.

    Without the wrapper, you get a static key, weak logs, and no real visibility when something goes wrong.

    With it, you get scoped access, review points, full traces, and a clearer answer when someone asks, “What did the system actually do?”

    Why day one matters

    Retrofitting this later is painful. It means backfilling logs, reworking workflows, and explaining to people why the guardrails arrived after the risk did.

    It is much easier to build the control plane alongside the agents and treat it as part of the product, not a patch.

    What to do first

    1. List every agent and workflow that can touch a real system.
    2. Log the full chain of inputs, decisions, and actions.
    3. Add human approval where the blast radius is high.
    4. Track failures and costs as first-class signals.

    That is enough to start building something you can actually trust.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • The Case for Explicit Policies

    Reliable systems do not emerge from good intentions. They emerge when the rules are explicit enough that another operator can understand what the system is supposed to do without reverse-engineering its behaviour from the wreckage.

    That sounds obvious, but it is still one of the most common gaps I see in automation and AI work. Teams build the workflow, connect the services, and get something working end to end. Then they leave the important decisions half-stated. Which provider is preferred? When should the fallback fire? What counts as a real health check? Which version of a process note is authoritative? The system may run, but the operating model is still fuzzy.

    The problem is not that people are careless. It is that policy work often looks less urgent than delivery work. Until something breaks, the invisible rule feels good enough.

    Where ambiguity shows up first

    The first place ambiguity appears is usually routing.

    A stack with multiple models, providers, queues, or execution paths always contains policy whether the team writes it down or not. If the preferred provider is too expensive for low-value tasks, that is policy. If one model is allowed for drafting but not for final output, that is policy. If a workflow should fall back only on timeout and not on quality failure, that is policy too.

    When none of that is written down clearly, people start inferring intent from whatever happened last.

    That is how teams end up with arguments that sound technical but are really operational:

    • “I thought the cheaper path was the default.”
    • “I assumed the fallback only applied during outages.”
    • “I didn’t realise this job was meant to stay on the private model.”
    • “I thought the dashboard alert meant the workflow had already rerouted.”

    None of those are bugs in isolation. They are symptoms of unstated policy.

    Why observability is part of policy

    The same issue appears in monitoring.

    A lot of dashboards tell you that a process is alive. That is not the same as proving the service is doing the right thing.

    For AI and automation systems, a truthful check usually needs to answer something more useful:

    • did the workflow complete the task it was supposed to complete?
    • did it use the intended path?
    • did it return data that looks structurally valid?
    • did the fallback stay dormant when the primary path was healthy?
    • can the operator see enough detail to explain the outcome afterward?

    If the check cannot answer those questions, the dashboard may still be visually tidy, but it is not giving the operator what they need.

    This is why I think observability should be treated as policy, not just instrumentation. The team has to decide what “working” actually means. Otherwise the monitoring layer simply reflects a vague assumption instead of a deliberate standard.

    Reuse is how policy survives handover

    The other quiet benefit of explicit policy is reuse.

    If a team has to rediscover the same routing rule, the same rollback sequence, or the same publishing checklist every time, then the policy is not really part of the system yet. It still lives in memory.

    That is expensive in a small team and dangerous in a growing one.

    Good reuse does not have to be elaborate. Often it is just a set of plain habits:

    • keep one canonical source of truth for important workflows
    • write fallback conditions near the implementation
    • keep short runbooks for the obvious failure modes
    • use the same naming and review patterns across similar jobs
    • record decisions before context evaporates

    None of that feels exciting while you are doing it. But it changes the quality of handover completely. A new operator no longer has to absorb the entire history of the stack before they can act safely.

    What explicit policy looks like in practice

    In practical terms, I look for a few simple signals.

    1. The preferred path is obvious

    The system should make it clear what happens first, what happens second, and under which conditions the fallback is allowed to take over.

    2. The checks reflect user reality

    A green dashboard should mean more than “something is listening on a port”. It should tell the operator whether the real job still works.

    3. Recovery paths exist before the incident

    If the first time a team documents the rollback sequence is during a failure, the policy work happened too late.

    4. Repeated patterns are actually reusable

    If the same kind of workflow appears three times, there should be a shared pattern instead of three slightly different tribal versions.

    Why this matters more with AI systems

    AI systems raise the cost of ambiguity because they turn a fuzzy rule into machine-speed inconsistency.

    In a manual process, unclear policy wastes time. In an automated one, it can silently change outputs, route work to the wrong provider, or create a trail that is too vague to audit later.

    That is why I think trustworthy AI is less about magic prompts and more about explicit operating rules.

    If the rules matter, write them down.

    If the outcome matters, check the real behaviour.

    If the workflow repeats, make it reusable.

    That does not make the system flashy. It makes it dependable.

    And in production, dependable usually wins.

    If you are building automation that needs to survive handover, escalation, and real operational scrutiny, the AI & Automation Architecture work is designed for exactly that. Or get in touch if you want a second pair of eyes on the operating model before the ambiguity becomes expensive.

  • GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    Introduction

    Some weeks are about shipping something obvious. Other weeks are about making the systems around the work easier to trust.

    This week looked like the second kind of week. Across the repos I reviewed, the change was direction, not just volume. Model routing became more deliberate. Monitoring became more accurate. Reusable delivery patterns became more explicit. Creative systems got clearer guardrails. And the small operational details that usually sit in the background were pulled into the foreground where they belong.

    That matters because the hardest part of running AI-assisted or automation-heavy systems is rarely the first prototype. The hard part is the handoff from “this works on my machine” to “this can be operated, explained, and improved without guesswork.” This week’s activity was a good reminder that trustworthy systems are built from a lot of very ordinary decisions made consistently.

    What happened

    Model routing stopped being an assumption and became a policy

    The clearest example came from hermes-mgmt. There was a real shift in how model choice is being handled: routing policy updates were merged, a stable default was pinned down, and provider-diverse fallback and outage behaviour were documented more explicitly. On top of that, role-based routing for research, coding, and review was defined instead of leaving model selection to whatever happened to be convenient in the moment.

    A lot of teams start with a simple belief: pick the biggest or newest model and let it do everything. That works until it doesn’t. Once you begin relying on models for different kinds of work, the distinction between “best model” and “best model for this job” becomes the difference between a healthy control plane and an expensive guessing game.

    I like this change because it turns model selection into an operating decision. It acknowledges that research work, coding work, and review work are not the same thing. It also gives the system a way to recover when a provider has issues, rather than forcing every incident into a manual exception.

    The surrounding work supports that direction too. A stale current-state snapshot was refreshed, the wider documentation was merged, and an earlier activation runbook for mixture-of-agents work shows that the system is being treated as something that needs operational memory, not just clever prompts.

    Observability was corrected, not merely expanded

    The biggest lesson from hamnet was the same one I have seen many times in production environments: a dashboard is only useful if it is telling the truth.

    This week’s work there focused on dashboard reliability rather than cosmetic additions. Grafana datasource timing was corrected, time series queries were given explicit intervals, default time windows were adjusted, and Pushgateway queries were wrapped so they render consistently. There was also a practical backlog item to track services that still need HTTPS equivalents, which is the kind of operational debt that gets forgotten unless somebody writes it down.

    I think this is a good example of how observability should be approached. It is tempting to treat monitoring as a collection of panels: add some graphs, colour some thresholds, and call it “visibility.” But real observability is more demanding. It means the query window is right, the step size is right, the defaults are right, and the chart is actually showing the thing you think it is showing.

    That is not a minor distinction. If your dashboard is technically live but operationally misleading, it can create more risk than it removes.

    There was also a useful signal in the issues around connecting a second Hermes instance as a remote subagent. That kind of work says the control plane is no longer just about one local setup. It is beginning to think in terms of distributed operation, where trust, connectivity, and monitoring all need to survive outside a single happy path.

    Creative systems became more governable

    HamMediaLabs showed a similar pattern, but from a different angle. The work there focused on creative-quality instrumentation, creative-ops playbooks, a per-brand visual identity layer, and a short-form voice system.

    That sounds very different from routing policies and dashboard fixes, but the underlying principle is the same: if you want something to scale, you need a repeatable operating model.

    In creative work, people often assume the value lives in taste alone. Taste matters, of course, but once you are managing multiple brands or content streams, taste has to be supported by structure. Otherwise every output becomes a one-off argument.

    The interesting thing here is that the repo is not just producing creative assets. It is creating a system for how those assets should be judged and delivered. That is the right order. First you define the identity. Then you define the checks. Then you can move quickly without drifting away from what the work is supposed to feel like.

    Delivery got more reusable, not just more active

    richardham-co-uk-ConsultancyOS was another good example of this week’s theme. CI was added, licensing and contributing guidance were written, a docs handbook appeared, and a SessionStart hook was introduced. In parallel, there was work on reusable delivery-repo patterns, operational dashboards, and prompt-pack integration.

    That is the sort of repo work that pays off over time in a way that is easy to underestimate.

    Most teams can write a one-off project. Fewer teams can turn that project into a template that makes the next one easier. The difference is not just technical polish. It is whether the project has been made reusable. When the CI, contribution rules, documentation, and bootstrap hooks are all captured explicitly, the next engagement starts with less friction and fewer assumptions.

    I see the same thing in the planning and monitoring work happening in lk-ai-roadmap. The hosting and runtime map was clarified, with a clear “no standing servers” direction. Token usage and cost-versus-budget monitoring were documented. Network and security monitoring agent work moved forward as a pilot spec. And the baseline capture kit and endorsement tracking suggest the project is being built with traceability in mind rather than as an ad hoc collection of tasks.

    That is what mature delivery looks like: fewer hidden decisions, more explicit rules, and less dependence on individual memory.

    Key takeaways

    The pattern across all of this work was remarkably consistent.

    • Policies are better than assumptions. Model routing, fallback behaviour, and role-based usage are clearer when they are written down.
    • Monitoring is only useful when it is accurate. A dashboard with the wrong time window or query behaviour is not a shortcut to clarity.
    • Reuse is a force multiplier. CI, docs, hooks, and templates make the next delivery easier than the last one.
    • Creative work benefits from the same discipline as infrastructure. Identity and quality checks are what make scale possible without drift.
    • Governance works best when it becomes ordinary. The more decisions are captured in the system itself, the less they depend on memory or heroics.

    What I take from this week is simple: the work is moving from experimentation to governable systems. That is a good sign. It means the platform is getting more predictable without losing momentum, and the people operating it are making fewer decisions in the dark.

    That is the kind of progress that compounds.

    If you are building AI systems, automation, or operational dashboards and want help turning the moving parts into something easier to trust, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to run.

  • GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    Introduction

    Some weeks are about shipping something obvious. Other weeks are about making the systems around the work easier to trust.

    This week looked like the second kind of week. Across the repos I reviewed, the change was direction, not just volume. Model routing became more deliberate. Monitoring became more accurate. Reusable delivery patterns became more explicit. Creative systems got clearer guardrails. And the small operational details that usually sit in the background were pulled into the foreground where they belong.

    That matters because the hardest part of running AI-assisted or automation-heavy systems is rarely the first prototype. The hard part is the handoff from “this works on my machine” to “this can be operated, explained, and improved without guesswork.” This week’s activity was a good reminder that trustworthy systems are built from a lot of very ordinary decisions made consistently.

    What happened

    Model routing stopped being an assumption and became a policy

    The clearest example came from hermes-mgmt. There was a real shift in how model choice is being handled: routing policy updates were merged, a stable default was pinned down, and provider-diverse fallback and outage behaviour were documented more explicitly. On top of that, role-based routing for research, coding, and review was defined instead of leaving model selection to whatever happened to be convenient in the moment.

    A lot of teams start with a simple belief: pick the biggest or newest model and let it do everything. That works until it doesn’t. Once you begin relying on models for different kinds of work, the distinction between “best model” and “best model for this job” becomes the difference between a healthy control plane and an expensive guessing game.

    I like this change because it turns model selection into an operating decision. It acknowledges that research work, coding work, and review work are not the same thing. It also gives the system a way to recover when a provider has issues, rather than forcing every incident into a manual exception.

    The surrounding work supports that direction too. A stale current-state snapshot was refreshed, the wider documentation was merged, and an earlier activation runbook for mixture-of-agents work shows that the system is being treated as something that needs operational memory, not just clever prompts.

    Observability was corrected, not merely expanded

    The biggest lesson from hamnet was the same one I have seen many times in production environments: a dashboard is only useful if it is telling the truth.

    This week’s work there focused on dashboard reliability rather than cosmetic additions. Grafana datasource timing was corrected, time series queries were given explicit intervals, default time windows were adjusted, and Pushgateway queries were wrapped so they render consistently. There was also a practical backlog item to track services that still need HTTPS equivalents, which is the kind of operational debt that gets forgotten unless somebody writes it down.

    I think this is a good example of how observability should be approached. It is tempting to treat monitoring as a collection of panels: add some graphs, colour some thresholds, and call it “visibility.” But real observability is more demanding. It means the query window is right, the step size is right, the defaults are right, and the chart is actually showing the thing you think it is showing.

    That is not a minor distinction. If your dashboard is technically live but operationally misleading, it can create more risk than it removes.

    There was also a useful signal in the issues around connecting a second Hermes instance as a remote subagent. That kind of work says the control plane is no longer just about one local setup. It is beginning to think in terms of distributed operation, where trust, connectivity, and monitoring all need to survive outside a single happy path.

    Creative systems became more governable

    HamMediaLabs showed a similar pattern, but from a different angle. The work there focused on creative-quality instrumentation, creative-ops playbooks, a per-brand visual identity layer, and a short-form voice system.

    That sounds very different from routing policies and dashboard fixes, but the underlying principle is the same: if you want something to scale, you need a repeatable operating model.

    In creative work, people often assume the value lives in taste alone. Taste matters, of course, but once you are managing multiple brands or content streams, taste has to be supported by structure. Otherwise every output becomes a one-off argument.

    The interesting thing here is that the repo is not just producing creative assets. It is creating a system for how those assets should be judged and delivered. That is the right order. First you define the identity. Then you define the checks. Then you can move quickly without drifting away from what the work is supposed to feel like.

    Delivery got more reusable, not just more active

    richardham-co-uk-ConsultancyOS was another good example of this week’s theme. CI was added, licensing and contributing guidance were written, a docs handbook appeared, and a SessionStart hook was introduced. In parallel, there was work on reusable delivery-repo patterns, operational dashboards, and prompt-pack integration.

    That is the sort of repo work that pays off over time in a way that is easy to underestimate.

    Most teams can write a one-off project. Fewer teams can turn that project into a template that makes the next one easier. The difference is not just technical polish. It is whether the project has been made reusable. When the CI, contribution rules, documentation, and bootstrap hooks are all captured explicitly, the next engagement starts with less friction and fewer assumptions.

    I see the same thing in the planning and monitoring work happening in lk-ai-roadmap. The hosting and runtime map was clarified, with a clear “no standing servers” direction. Token usage and cost-versus-budget monitoring were documented. Network and security monitoring agent work moved forward as a pilot spec. And the baseline capture kit and endorsement tracking suggest the project is being built with traceability in mind rather than as an ad hoc collection of tasks.

    That is what mature delivery looks like: fewer hidden decisions, more explicit rules, and less dependence on individual memory.

    Key takeaways

    The pattern across all of this work was remarkably consistent.

    • Policies are better than assumptions. Model routing, fallback behaviour, and role-based usage are clearer when they are written down.
    • Monitoring is only useful when it is accurate. A dashboard with the wrong time window or query behaviour is not a shortcut to clarity.
    • Reuse is a force multiplier. CI, docs, hooks, and templates make the next delivery easier than the last one.
    • Creative work benefits from the same discipline as infrastructure. Identity and quality checks are what make scale possible without drift.
    • Governance works best when it becomes ordinary. The more decisions are captured in the system itself, the less they depend on memory or heroics.

    What I take from this week is simple: the work is moving from experimentation to governable systems. That is a good sign. It means the platform is getting more predictable without losing momentum, and the people operating it are making fewer decisions in the dark.

    That is the kind of progress that compounds.

    If you are building AI systems, automation, or operational dashboards and want help turning the moving parts into something easier to trust, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to run.

  • The Case for Explicit Policies

    The Case for Explicit Policies

    Reliable systems do not emerge from good intentions. They emerge when the rules are explicit enough that another operator can understand what the system is supposed to do without reverse-engineering its behaviour from the wreckage.

    That sounds obvious, but it is still one of the most common gaps I see in automation and AI work. Teams build the workflow, connect the services, and get something working end to end. Then they leave the important decisions half-stated. Which provider is preferred? When should the fallback fire? What counts as a real health check? Which version of a process note is authoritative? The system may run, but the operating model is still fuzzy.

    The problem is not that people are careless. It is that policy work often looks less urgent than delivery work. Until something breaks, the invisible rule feels good enough.

    Where ambiguity shows up first

    The first place ambiguity appears is usually routing.

    A stack with multiple models, providers, queues, or execution paths always contains policy whether the team writes it down or not. If the preferred provider is too expensive for low-value tasks, that is policy. If one model is allowed for drafting but not for final output, that is policy. If a workflow should fall back only on timeout and not on quality failure, that is policy too.

    When none of that is written down clearly, people start inferring intent from whatever happened last.

    That is how teams end up with arguments that sound technical but are really operational:

    • “I thought the cheaper path was the default.”
    • “I assumed the fallback only applied during outages.”
    • “I didn’t realise this job was meant to stay on the private model.”
    • “I thought the dashboard alert meant the workflow had already rerouted.”

    None of those are bugs in isolation. They are symptoms of unstated policy.

    Why observability is part of policy

    The same issue appears in monitoring.

    A lot of dashboards tell you that a process is alive. That is not the same as proving the service is doing the right thing.

    For AI and automation systems, a truthful check usually needs to answer something more useful:

    • did the workflow complete the task it was supposed to complete?
    • did it use the intended path?
    • did it return data that looks structurally valid?
    • did the fallback stay dormant when the primary path was healthy?
    • can the operator see enough detail to explain the outcome afterward?

    If the check cannot answer those questions, the dashboard may still be visually tidy, but it is not giving the operator what they need.

    This is why I think observability should be treated as policy, not just instrumentation. The team has to decide what “working” actually means. Otherwise the monitoring layer simply reflects a vague assumption instead of a deliberate standard.

    Reuse is how policy survives handover

    The other quiet benefit of explicit policy is reuse.

    If a team has to rediscover the same routing rule, the same rollback sequence, or the same publishing checklist every time, then the policy is not really part of the system yet. It still lives in memory.

    That is expensive in a small team and dangerous in a growing one.

    Good reuse does not have to be elaborate. Often it is just a set of plain habits:

    • keep one canonical source of truth for important workflows
    • write fallback conditions near the implementation
    • keep short runbooks for the obvious failure modes
    • use the same naming and review patterns across similar jobs
    • record decisions before context evaporates

    None of that feels exciting while you are doing it. But it changes the quality of handover completely. A new operator no longer has to absorb the entire history of the stack before they can act safely.

    What explicit policy looks like in practice

    In practical terms, I look for a few simple signals.

    1. The preferred path is obvious

    The system should make it clear what happens first, what happens second, and under which conditions the fallback is allowed to take over.

    2. The checks reflect user reality

    A green dashboard should mean more than “something is listening on a port”. It should tell the operator whether the real job still works.

    3. Recovery paths exist before the incident

    If the first time a team documents the rollback sequence is during a failure, the policy work happened too late.

    4. Repeated patterns are actually reusable

    If the same kind of workflow appears three times, there should be a shared pattern instead of three slightly different tribal versions.

    Why this matters more with AI systems

    AI systems raise the cost of ambiguity because they turn a fuzzy rule into machine-speed inconsistency.

    In a manual process, unclear policy wastes time. In an automated one, it can silently change outputs, route work to the wrong provider, or create a trail that is too vague to audit later.

    That is why I think trustworthy AI is less about magic prompts and more about explicit operating rules.

    If the rules matter, write them down.

    If the outcome matters, check the real behaviour.

    If the workflow repeats, make it reusable.

    That does not make the system flashy. It makes it dependable.

    And in production, dependable usually wins.

    If you are building automation that needs to survive handover, escalation, and real operational scrutiny, the AI & Automation Architecture work is designed for exactly that. Or get in touch if you want a second pair of eyes on the operating model before the ambiguity becomes expensive.

Powered by