RichardHam.co.uk Blog

Tag: automation

  • GitHub Weekly — Security Hardening, Governance Cleanup, and Website Delivery

    GitHub Weekly — Security Hardening, Governance Cleanup, and Website Delivery

    This week’s GitHub activity had a very familiar shape: a lot of small changes, but a very clear direction.

    When I pulled the last seven days together, I ended up with 250 events across 64 repositories. The detail varied, but the pattern was hard to miss. The most useful work clustered around three themes: making systems safer to operate, making decisions easier to trace, and moving real work out of drafts and into something usable.

    That combination matters. It is easy to celebrate a feature release and overlook the quieter work that makes the next release easier, safer, and less dependent on memory. This week was mostly about exactly that.

    What happened

    1. The hardening work kept moving

    The busiest thread was around the Hermes management side of the stack. There were issues and pull requests covering gateway restart behaviour, profile-scoped tools and memory surfaces, local Vault health, OAuth token expiry, dependency CVEs, and a few routing and performance concerns.

    That is the kind of activity I like to see in a living system, even when it is uncomfortable. The work was not cosmetic. It was focused on failure modes:

    • what happens when the secrets backend is sealed
    • what happens when a token expires unexpectedly
    • what happens when dependency drift creates exposure in the runtime
    • what happens when routing logic gets too expensive to keep running blindly
    • what happens when multiple operators need clean boundaries around tools and state

    In other words, the week was spent asking the right questions before the answers became incidents.

    There was also a useful operational thread around keeping the system honest: adding safety nets, reviewing carry-forward behaviour, and tightening the control plane around restarts and state. That kind of work rarely looks dramatic in a changelog, but it is often where the real reliability gains come from.

    2. Governance became more concrete

    A second pattern showed up in the roadmap and control repositories. There were commits and pull requests for provisioning scripts, README links, alert-rule tuning, and monitoring documentation.

    That matters because strategy is only useful when it can be executed repeatedly.

    A plan is just a plan until it has:

    • a provisioning path
    • a documented handover
    • a clear monitoring expectation
    • a change history that another person can follow

    The activity this week pushed in that direction. Roadmap work became more operational. Monitoring guidance became more explicit. Decision-making became easier to track. Even the recurring “decision desk” style updates are useful in that sense: they turn vague progress into a traceable record.

    That is particularly important in AI and automation work, where teams can move quickly but still leave behind unclear assumptions. The more complex the stack gets, the more valuable it becomes to treat governance as part of the delivery process rather than as a separate admin task.

    3. Website work moved from intention to delivery

    The third thread was more visible: website work.

    There was a full redesign path on one service site, including layout work, deploy tooling, and a clear move from mockups to implemented pages. On the content side, the blog workflow itself also kept moving, with a new weekly roundup drafted and the editorial queue updated.

    I think this is an underrated signal. People often talk about code, but delivery includes the path around the code too:

    • the build steps
    • the draft content
    • the publication workflow
    • the editorial queue
    • the handoff between “done locally” and “live somewhere useful”

    If those pieces are weak, the site may look finished while still being awkward to maintain. If they are strong, the site becomes easier to update, easier to trust, and easier to keep current.

    That same lesson shows up in content systems, operations tooling, and AI workflows. The system is only as strong as the path from intent to output.

    Key takeaways

    A week like this leaves a few practical lessons.

    • Security debt is easiest to fix before it becomes visible. The moment a sealed Vault, an expired token, or a dependency CVE shows up in a weekly review is the moment to deal with it.
    • Operational boundaries matter more as systems grow. Profile-scoped tools, explicit memory surfaces, and predictable restart behaviour are all examples of the same idea: reduce ambiguity.
    • Roadmaps need executable steps, not just aspirations. Provisioning scripts, alert-rule updates, and readable docs make a roadmap real.
    • Delivery includes the publishing path. A draft that never reaches the right place is only half a result.
    • The best week-over-week improvement is often cumulative, not flashy. Small fixes across security, governance, and publishing add up to a stronger operating model.

    A practical standard

    If I had to compress the week into one rule, it would be this:

    • if it can fail, define the failure mode
    • if it repeats, make it traceable
    • if it matters, write it down
    • if it ships, make the path to shipping reliable

    That is a good standard for AI work, but it is just as useful for infrastructure, websites, and internal operations. The details change. The principle does not.

    The goal is not to eliminate all uncertainty. The goal is to make the important parts of the system clear enough that they can be operated without guesswork.

    Closing thought

    The most useful work this week was not a single large feature. It was the accumulation of smaller changes that make a system easier to run: better failure handling, clearer governance, more repeatable delivery, and a stronger publishing path.

    That is usually where durable progress lives.

    If you are working through a similar mix of security, automation, and delivery problems, the AI & Automation Architecture service is a good starting point. Or get in touch if you want to talk through the shape of the system before it becomes the problem.

  • When the Rules Become the Product

    When the Rules Become the Product

    This week kept returning to the same idea: the most useful engineering work is often the work that makes a system easier to trust.

    That does not always look exciting from the outside. It is not always a new feature, a flashy demo, or a dramatic redesign. More often it is the quieter work of making rules explicit, tightening feedback loops, removing ambiguity, and making the next decision easier than the last one.

    Across the repos I watched this week, that pattern showed up again and again. The common thread was less about adding novelty and more about turning guesswork into something people can actually operate.

    What happened

    1. hermes-mgmt kept pushing on cost, routing, and safety

    hermes-mgmt was the busiest repo in the set, and the signal was very clear: the platform is maturing by making its own guard rails stronger.

    A few of the issues were a good reminder that reliability starts with honesty. One thread called out how expensive large context windows become when autonomous loops keep calling them. Another flagged a provider routing problem where fallback behaviour was not matching the intent of the system. There was also a security issue around dependency CVEs and overly permissive local state handling, plus a series of local service defects that needed attention before they could become bigger problems.

    The pull requests told the same story from the implementation side. There were updates around shared spend visibility, stronger routing defaults, helper scripts for operator actions, secret-finding classifiers, carry guard hooks, resilient re-apply logic, memory activation, and better health checks. In plain English: this is the part of the work where a system stops relying on optimism and starts relying on policy.

    That matters because most production problems are not caused by one spectacular failure. They come from small inconsistencies that accumulate until the platform becomes harder to predict than it should be. The work in hermes-mgmt was a good example of the opposite: make the rules visible, make the fallback paths deliberate, and make the expensive behaviour harder to trigger by accident.

    2. ricambio-ai-roadmap moved product behaviour into the open

    The second big theme lived in ricambio-ai-roadmap, where the work on pilot2_email and the surrounding platform showed a lot more operational maturity than a casual observer might expect.

    There were fixes for transient IMAP failures, a fallback provider chain, and intent-based routing. There was also a 48-hour expiry policy, an in-app help section, and review-session documentation that makes the decision path much easier to follow later. At the same time, the live-platform review surfaced issues around basic-auth defaults, placeholder leakage risk, console labelling, and filter gaps.

    That mix is important. It means the project is no longer just building features; it is building a way to reason about those features.

    If a workflow is going to make decisions on behalf of a user, then the decision rules need to be legible. If a review finds a weak spot, the fix should not just patch the symptom. It should make the system easier to explain the next time somebody has to operate it.

    That is why the recent work here feels significant. It is not only improving the product. It is making the product harder to misunderstand.

    3. richardham-web-and-brand treated content and navigation as part of the system

    The web and brand repo had a very different surface area, but the same underlying pattern.

    There was work on missing sector pages, menu ordering, the AI stack page, homepage hero messaging, and proof-point content. There was also a rewrite of a blog post that had been flagged for confidential or internal detail. That kind of content cleanup is easy to underestimate, but it is exactly the sort of work that keeps a site coherent and credible.

    The best websites do not just look polished. They make it obvious where to go next.

    When navigation is clear, a visitor does not have to guess. When page structure is consistent, a service line is easier to understand. When internal detail is removed from public-facing copy, the story becomes more focused and more trustworthy. That is why I think of this sort of content work as part of the system, not a separate marketing task.

    It is also a useful reminder that design discipline and operational discipline are cousins. Both are about reducing friction. Both are about making the important thing easier to find. And both become more valuable as the system grows.

    4. Smaller repos kept reinforcing the same direction

    A few other repos pointed in the same direction even if they were not as noisy.

    dh-electrical-uk-website had redesign work and deploy tooling, which is a nice example of making a site feel more coherent and easier to ship.

    HamMediaLabs, lk-ai-roadmap, control-tower, and ai-cost-tracker all contributed to the broader picture as well: the more a platform matures, the more the useful work shifts toward clarity, repeatability, and control.

    That is a pattern I like because it is easy to miss in the moment. From far away, all of this can look like a list of unrelated tasks. Up close, it is really one story told in different repos: make the rules explicit, make the system observable, and make the next step easier to take.

    Key takeaways

    A few lessons stood out this week:

    • Routing should be policy, not folklore. If a system needs to choose between paths, the choice should be deliberate and explainable.
    • Security work belongs in the main workflow. Review findings, dependency checks, and permission problems are not side quests; they are part of keeping the platform honest.
    • Visibility is worth more than cleverness. Spend tracking, health checks, and honest dashboards make a system easier to run than vague confidence ever will.
    • Content operations matter. Navigation, structure, and public copy are part of the user experience and deserve the same discipline as the code behind them.
    • The best guard rail is a clear rule. The less people have to remember, the less likely the system is to depend on luck.

    Closing thought

    The strongest theme this week was not speed. It was trust.

    A system becomes easier to trust when the important choices are written down, the fallback paths are deliberate, and the rough edges are visible before they become incidents. That is true for routing policies, for security reviews, for website navigation, and for the way content moves from draft to publish.

    If you are trying to make an AI or automation workflow feel less like improvisation and more like operations, the next step usually is not more complexity. It is more clarity.

    And if that is the kind of cleanup you are wrestling with, get in touch.

  • When Systems Stop Relying on Guesswork

    A lot of avoidable operational pain comes from one simple source: somebody had to guess.

    They had to guess which model the workflow would choose. They had to guess whether a green dashboard meant the service was actually working. They had to guess how to recover something because the only real runbook lived in one person’s head. They had to guess which version of a draft was the right one to publish.

    That is a small sentence with expensive consequences.

    In my experience, many reliability problems are not caused by one dramatic bug. They come from layers of ambiguity that were tolerated because the system appeared to work often enough. The workflow usually picked the expected path. The dashboard was usually telling the truth. The operator usually remembered the recovery sequence. The content pipeline usually published the right version.

    Usually is not a safe operating model.

    Where the friction really sits

    The obvious failure is the outage. The more common failure is ambiguity that keeps making the whole stack harder to trust.

    Routing decisions

    If a system can choose between providers, models, or execution paths, then the decision logic needs to be durable and visible. Otherwise behaviour changes by accident.

    That is how teams end up with a platform that feels inconsistent even though each individual component is technically healthy. One person thinks the cheap path is the default. Another thinks the reliable path is. A third assumes the fallback only applies in a timeout case. None of that is a software problem at first. It is a policy problem.

    The fix is rarely glamorous. Write the rule. Keep it near the implementation. Make it obvious what happens first, what happens second, and what happens when the preferred path fails.

    Monitoring that tells the truth

    There is a big difference between “the process is running” and “the service is working”.

    A healthy-looking process table can still hide a broken dependency, a dead API path, a stale token, or an empty data plane. If the check only proves that something is listening on a port, it is not telling the operator what they actually need to know.

    Useful monitoring is usually the boring kind. Does the real workflow respond, with the right output, in the way the operator expects? Can someone tell whether a fallback fired? Can they see whether the service remained functionally correct rather than merely alive?

    If not, the dashboard is decoration.

    Recovery and repeatability

    A lot of infrastructure looks solid right up until someone else has to operate it.

    That is where repeatable recovery paths matter. Clear defaults. Known fallbacks. A documented sequence for fixing the obvious failure modes. The point is not to eliminate human judgement. The point is to stop every incident from turning into archaeology.

    Runbooks, checklists, and decision notes are not admin overhead for serious systems. They are part of the product if the product has to survive handover.

    Publishing discipline

    This applies to content as much as infrastructure.

    If the publishing path is fuzzy, you get duplicate posts, mismatched versions, archive drift, and copy that sounds assembled rather than written. That is not only a branding issue. It is the same operational problem wearing different clothes.

    The draft, the review step, the canonical source file, and the final publish target all need to be explicit. Otherwise the system eventually ships the wrong thing.

    Why AI multiplies the cost of ambiguity

    AI and automation raise the stakes because they convert vague process into machine-speed inconsistency.

    When a human operator is doing everything manually, weak process often shows up as wasted time. When workflows start routing, deciding, and acting on their own, the same vagueness becomes a production risk. The system can now repeat the wrong assumption quickly, quietly, and at scale.

    That is why I keep coming back to the same standard:

    • if it matters, write it down
    • if it repeats, make it reusable
    • if it can fail, define the fallback
    • if it is monitored, test the real behaviour
    • if it is published, keep one clear source of truth

    None of that is flashy. All of it matters.

    The useful kind of progress

    The best weeks are not always the dramatic ones.

    Sometimes the real progress is a tighter routing rule. A dashboard that finally reflects reality. A recovery path that no longer depends on memory. A publishing workflow that stops sending the wrong version into the public archive.

    That kind of work rarely looks exciting from the outside. It just makes the system calmer, safer, and easier to trust.

    And in practice, that is often the difference between something that demos well and something that survives production.

    If you want help turning AI or automation from a pile of moving parts into an operating model people can actually trust, start with AI & Automation Architecture or get in touch.

  • Why AI Workflows Need Audit Trails

    Why AI Workflows Need Audit Trails

    The conversation around AI has shifted. It is no longer just about drafting text or summarising meetings. More and more often, these systems are taking actions on live business processes.

    That is where the risk changes shape.

    A clever prompt can produce a good-looking result. It cannot tell you what happened after the fact if the output was wrong.

    The gap

    When a person makes a decision in a process, there is usually some trace of it. An email, a ticket, a sign-off, a log entry. With an AI system, that trace is often thin or missing.

    If the system updates the wrong record or sends the wrong message, the team is left with a result and very little explanation.

    What breaks without logs

    • You cannot reconstruct the sequence of events.
    • You cannot show who approved what.
    • You cannot improve the workflow with confidence.

    That is not just a debugging problem. It is an accountability problem.

    What a useful audit trail looks like

    At minimum, every execution should capture:

    • the input it received
    • the steps it took
    • the action it actually executed
    • the output it produced
    • the timestamp and identity of the run

    That is enough to answer the questions that matter later.

    The practical bit

    The tooling is already there. Structured logs, append-only storage, and reviewable traces are all enough to get started. The main thing is to design for visibility before the workflow is under pressure.

    If the system is allowed to act, it should also be required to explain itself.

  • The Case for Explicit Policies

    The Case for Explicit Policies

    Reliable systems do not emerge from good intentions. They emerge when the rules are explicit enough that another operator can understand what the system is supposed to do without reverse-engineering its behaviour from the wreckage.

    That sounds obvious, but it is still one of the most common gaps I see in automation and AI work. Teams build the workflow, connect the services, and get something working end to end. Then they leave the important decisions half-stated. Which provider is preferred? When should the fallback fire? What counts as a real health check? Which version of a process note is authoritative? The system may run, but the operating model is still fuzzy.

    The problem is not that people are careless. It is that policy work often looks less urgent than delivery work. Until something breaks, the invisible rule feels good enough.

    Where ambiguity shows up first

    The first place ambiguity appears is usually routing.

    A stack with multiple models, providers, queues, or execution paths always contains policy whether the team writes it down or not. If the preferred provider is too expensive for low-value tasks, that is policy. If one model is allowed for drafting but not for final output, that is policy. If a workflow should fall back only on timeout and not on quality failure, that is policy too.

    When none of that is written down clearly, people start inferring intent from whatever happened last.

    That is how teams end up with arguments that sound technical but are really operational:

    • “I thought the cheaper path was the default.”
    • “I assumed the fallback only applied during outages.”
    • “I didn’t realise this job was meant to stay on the private model.”
    • “I thought the dashboard alert meant the workflow had already rerouted.”

    None of those are bugs in isolation. They are symptoms of unstated policy.

    Why observability is part of policy

    The same issue appears in monitoring.

    A lot of dashboards tell you that a process is alive. That is not the same as proving the service is doing the right thing.

    For AI and automation systems, a truthful check usually needs to answer something more useful:

    • did the workflow complete the task it was supposed to complete?
    • did it use the intended path?
    • did it return data that looks structurally valid?
    • did the fallback stay dormant when the primary path was healthy?
    • can the operator see enough detail to explain the outcome afterward?

    If the check cannot answer those questions, the dashboard may still be visually tidy, but it is not giving the operator what they need.

    This is why I think observability should be treated as policy, not just instrumentation. The team has to decide what “working” actually means. Otherwise the monitoring layer simply reflects a vague assumption instead of a deliberate standard.

    Reuse is how policy survives handover

    The other quiet benefit of explicit policy is reuse.

    If a team has to rediscover the same routing rule, the same rollback sequence, or the same publishing checklist every time, then the policy is not really part of the system yet. It still lives in memory.

    That is expensive in a small team and dangerous in a growing one.

    Good reuse does not have to be elaborate. Often it is just a set of plain habits:

    • keep one canonical source of truth for important workflows
    • write fallback conditions near the implementation
    • keep short runbooks for the obvious failure modes
    • use the same naming and review patterns across similar jobs
    • record decisions before context evaporates

    None of that feels exciting while you are doing it. But it changes the quality of handover completely. A new operator no longer has to absorb the entire history of the stack before they can act safely.

    What explicit policy looks like in practice

    In practical terms, I look for a few simple signals.

    1. The preferred path is obvious

    The system should make it clear what happens first, what happens second, and under which conditions the fallback is allowed to take over.

    2. The checks reflect user reality

    A green dashboard should mean more than “something is listening on a port”. It should tell the operator whether the real job still works.

    3. Recovery paths exist before the incident

    If the first time a team documents the rollback sequence is during a failure, the policy work happened too late.

    4. Repeated patterns are actually reusable

    If the same kind of workflow appears three times, there should be a shared pattern instead of three slightly different tribal versions.

    Why this matters more with AI systems

    AI systems raise the cost of ambiguity because they turn a fuzzy rule into machine-speed inconsistency.

    In a manual process, unclear policy wastes time. In an automated one, it can silently change outputs, route work to the wrong provider, or create a trail that is too vague to audit later.

    That is why I think trustworthy AI is less about magic prompts and more about explicit operating rules.

    If the rules matter, write them down.

    If the outcome matters, check the real behaviour.

    If the workflow repeats, make it reusable.

    That does not make the system flashy. It makes it dependable.

    And in production, dependable usually wins.

    If you are building automation that needs to survive handover, escalation, and real operational scrutiny, the AI & Automation Architecture work is designed for exactly that. Or get in touch if you want a second pair of eyes on the operating model before the ambiguity becomes expensive.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Why AI Control Planes Need Observability From Day One

    Why AI Control Planes Need Observability From Day One

    AI agents are getting plugged into email, CRM, finance, and document systems faster than most teams can build the operations around them.

    That is the mismatch.

    The agent might be fine. The prompt might be fine. The integration might even work. What is often missing is the layer that says who can do what, what happened, what it cost, and how the system recovers when something breaks.

    What the control plane does

    Think of it as the operational wrapper around the agent. It handles:

    • authentication and scoped access
    • structured logging
    • approval gates for risky actions
    • observability and cost tracking
    • failure recovery

    That is not overkill. That is the minimum if the agent is touching production data.

    A concrete example

    Imagine a workflow that reads a form, enriches a record, updates a CRM, and sends a confirmation email.

    Without the wrapper, you get a static key, weak logs, and no real visibility when something goes wrong.

    With it, you get scoped access, review points, full traces, and a clearer answer when someone asks, “What did the system actually do?”

    Why day one matters

    Retrofitting this later is painful. It means backfilling logs, reworking workflows, and explaining to people why the guardrails arrived after the risk did.

    It is much easier to build the control plane alongside the agents and treat it as part of the product, not a patch.

    What to do first

    1. List every agent and workflow that can touch a real system.
    2. Log the full chain of inputs, decisions, and actions.
    3. Add human approval where the blast radius is high.
    4. Track failures and costs as first-class signals.

    That is enough to start building something you can actually trust.

Powered by