RichardHam.co.uk Blog

Tag: security

  • GitHub Weekly — Security Hardening, Governance Cleanup, and Website Delivery

    GitHub Weekly — Security Hardening, Governance Cleanup, and Website Delivery

    This week’s GitHub activity had a very familiar shape: a lot of small changes, but a very clear direction.

    When I pulled the last seven days together, I ended up with 250 events across 64 repositories. The detail varied, but the pattern was hard to miss. The most useful work clustered around three themes: making systems safer to operate, making decisions easier to trace, and moving real work out of drafts and into something usable.

    That combination matters. It is easy to celebrate a feature release and overlook the quieter work that makes the next release easier, safer, and less dependent on memory. This week was mostly about exactly that.

    What happened

    1. The hardening work kept moving

    The busiest thread was around the Hermes management side of the stack. There were issues and pull requests covering gateway restart behaviour, profile-scoped tools and memory surfaces, local Vault health, OAuth token expiry, dependency CVEs, and a few routing and performance concerns.

    That is the kind of activity I like to see in a living system, even when it is uncomfortable. The work was not cosmetic. It was focused on failure modes:

    • what happens when the secrets backend is sealed
    • what happens when a token expires unexpectedly
    • what happens when dependency drift creates exposure in the runtime
    • what happens when routing logic gets too expensive to keep running blindly
    • what happens when multiple operators need clean boundaries around tools and state

    In other words, the week was spent asking the right questions before the answers became incidents.

    There was also a useful operational thread around keeping the system honest: adding safety nets, reviewing carry-forward behaviour, and tightening the control plane around restarts and state. That kind of work rarely looks dramatic in a changelog, but it is often where the real reliability gains come from.

    2. Governance became more concrete

    A second pattern showed up in the roadmap and control repositories. There were commits and pull requests for provisioning scripts, README links, alert-rule tuning, and monitoring documentation.

    That matters because strategy is only useful when it can be executed repeatedly.

    A plan is just a plan until it has:

    • a provisioning path
    • a documented handover
    • a clear monitoring expectation
    • a change history that another person can follow

    The activity this week pushed in that direction. Roadmap work became more operational. Monitoring guidance became more explicit. Decision-making became easier to track. Even the recurring “decision desk” style updates are useful in that sense: they turn vague progress into a traceable record.

    That is particularly important in AI and automation work, where teams can move quickly but still leave behind unclear assumptions. The more complex the stack gets, the more valuable it becomes to treat governance as part of the delivery process rather than as a separate admin task.

    3. Website work moved from intention to delivery

    The third thread was more visible: website work.

    There was a full redesign path on one service site, including layout work, deploy tooling, and a clear move from mockups to implemented pages. On the content side, the blog workflow itself also kept moving, with a new weekly roundup drafted and the editorial queue updated.

    I think this is an underrated signal. People often talk about code, but delivery includes the path around the code too:

    • the build steps
    • the draft content
    • the publication workflow
    • the editorial queue
    • the handoff between “done locally” and “live somewhere useful”

    If those pieces are weak, the site may look finished while still being awkward to maintain. If they are strong, the site becomes easier to update, easier to trust, and easier to keep current.

    That same lesson shows up in content systems, operations tooling, and AI workflows. The system is only as strong as the path from intent to output.

    Key takeaways

    A week like this leaves a few practical lessons.

    • Security debt is easiest to fix before it becomes visible. The moment a sealed Vault, an expired token, or a dependency CVE shows up in a weekly review is the moment to deal with it.
    • Operational boundaries matter more as systems grow. Profile-scoped tools, explicit memory surfaces, and predictable restart behaviour are all examples of the same idea: reduce ambiguity.
    • Roadmaps need executable steps, not just aspirations. Provisioning scripts, alert-rule updates, and readable docs make a roadmap real.
    • Delivery includes the publishing path. A draft that never reaches the right place is only half a result.
    • The best week-over-week improvement is often cumulative, not flashy. Small fixes across security, governance, and publishing add up to a stronger operating model.

    A practical standard

    If I had to compress the week into one rule, it would be this:

    • if it can fail, define the failure mode
    • if it repeats, make it traceable
    • if it matters, write it down
    • if it ships, make the path to shipping reliable

    That is a good standard for AI work, but it is just as useful for infrastructure, websites, and internal operations. The details change. The principle does not.

    The goal is not to eliminate all uncertainty. The goal is to make the important parts of the system clear enough that they can be operated without guesswork.

    Closing thought

    The most useful work this week was not a single large feature. It was the accumulation of smaller changes that make a system easier to run: better failure handling, clearer governance, more repeatable delivery, and a stronger publishing path.

    That is usually where durable progress lives.

    If you are working through a similar mix of security, automation, and delivery problems, the AI & Automation Architecture service is a good starting point. Or get in touch if you want to talk through the shape of the system before it becomes the problem.

  • When the Rules Become the Product

    When the Rules Become the Product

    This week kept returning to the same idea: the most useful engineering work is often the work that makes a system easier to trust.

    That does not always look exciting from the outside. It is not always a new feature, a flashy demo, or a dramatic redesign. More often it is the quieter work of making rules explicit, tightening feedback loops, removing ambiguity, and making the next decision easier than the last one.

    Across the repos I watched this week, that pattern showed up again and again. The common thread was less about adding novelty and more about turning guesswork into something people can actually operate.

    What happened

    1. hermes-mgmt kept pushing on cost, routing, and safety

    hermes-mgmt was the busiest repo in the set, and the signal was very clear: the platform is maturing by making its own guard rails stronger.

    A few of the issues were a good reminder that reliability starts with honesty. One thread called out how expensive large context windows become when autonomous loops keep calling them. Another flagged a provider routing problem where fallback behaviour was not matching the intent of the system. There was also a security issue around dependency CVEs and overly permissive local state handling, plus a series of local service defects that needed attention before they could become bigger problems.

    The pull requests told the same story from the implementation side. There were updates around shared spend visibility, stronger routing defaults, helper scripts for operator actions, secret-finding classifiers, carry guard hooks, resilient re-apply logic, memory activation, and better health checks. In plain English: this is the part of the work where a system stops relying on optimism and starts relying on policy.

    That matters because most production problems are not caused by one spectacular failure. They come from small inconsistencies that accumulate until the platform becomes harder to predict than it should be. The work in hermes-mgmt was a good example of the opposite: make the rules visible, make the fallback paths deliberate, and make the expensive behaviour harder to trigger by accident.

    2. ricambio-ai-roadmap moved product behaviour into the open

    The second big theme lived in ricambio-ai-roadmap, where the work on pilot2_email and the surrounding platform showed a lot more operational maturity than a casual observer might expect.

    There were fixes for transient IMAP failures, a fallback provider chain, and intent-based routing. There was also a 48-hour expiry policy, an in-app help section, and review-session documentation that makes the decision path much easier to follow later. At the same time, the live-platform review surfaced issues around basic-auth defaults, placeholder leakage risk, console labelling, and filter gaps.

    That mix is important. It means the project is no longer just building features; it is building a way to reason about those features.

    If a workflow is going to make decisions on behalf of a user, then the decision rules need to be legible. If a review finds a weak spot, the fix should not just patch the symptom. It should make the system easier to explain the next time somebody has to operate it.

    That is why the recent work here feels significant. It is not only improving the product. It is making the product harder to misunderstand.

    3. richardham-web-and-brand treated content and navigation as part of the system

    The web and brand repo had a very different surface area, but the same underlying pattern.

    There was work on missing sector pages, menu ordering, the AI stack page, homepage hero messaging, and proof-point content. There was also a rewrite of a blog post that had been flagged for confidential or internal detail. That kind of content cleanup is easy to underestimate, but it is exactly the sort of work that keeps a site coherent and credible.

    The best websites do not just look polished. They make it obvious where to go next.

    When navigation is clear, a visitor does not have to guess. When page structure is consistent, a service line is easier to understand. When internal detail is removed from public-facing copy, the story becomes more focused and more trustworthy. That is why I think of this sort of content work as part of the system, not a separate marketing task.

    It is also a useful reminder that design discipline and operational discipline are cousins. Both are about reducing friction. Both are about making the important thing easier to find. And both become more valuable as the system grows.

    4. Smaller repos kept reinforcing the same direction

    A few other repos pointed in the same direction even if they were not as noisy.

    dh-electrical-uk-website had redesign work and deploy tooling, which is a nice example of making a site feel more coherent and easier to ship.

    HamMediaLabs, lk-ai-roadmap, control-tower, and ai-cost-tracker all contributed to the broader picture as well: the more a platform matures, the more the useful work shifts toward clarity, repeatability, and control.

    That is a pattern I like because it is easy to miss in the moment. From far away, all of this can look like a list of unrelated tasks. Up close, it is really one story told in different repos: make the rules explicit, make the system observable, and make the next step easier to take.

    Key takeaways

    A few lessons stood out this week:

    • Routing should be policy, not folklore. If a system needs to choose between paths, the choice should be deliberate and explainable.
    • Security work belongs in the main workflow. Review findings, dependency checks, and permission problems are not side quests; they are part of keeping the platform honest.
    • Visibility is worth more than cleverness. Spend tracking, health checks, and honest dashboards make a system easier to run than vague confidence ever will.
    • Content operations matter. Navigation, structure, and public copy are part of the user experience and deserve the same discipline as the code behind them.
    • The best guard rail is a clear rule. The less people have to remember, the less likely the system is to depend on luck.

    Closing thought

    The strongest theme this week was not speed. It was trust.

    A system becomes easier to trust when the important choices are written down, the fallback paths are deliberate, and the rough edges are visible before they become incidents. That is true for routing policies, for security reviews, for website navigation, and for the way content moves from draft to publish.

    If you are trying to make an AI or automation workflow feel less like improvisation and more like operations, the next step usually is not more complexity. It is more clarity.

    And if that is the kind of cleanup you are wrestling with, get in touch.

  • Containment Patterns for Autonomous AI Agents

    Containment Patterns for Autonomous AI Agents

    You wouldn’t give a new employee the keys to every office, the admin password to every system, and a company credit card on day one. You’d onboard them gradually — scoped access, supervised actions, and a clear audit trail of what they did and why.

    Autonomous AI agents deserve the same treatment. Arguably more, because they can act faster, at greater scale, and with less inherent judgment than any human hire.

    Yet the pattern I see repeatedly is teams wiring agents directly into production systems with broad API keys, unrestricted network access, and no oversight. The agent works brilliantly — until it doesn’t. And when it goes wrong, nobody can explain what happened or why.

    This is a containment problem. And it’s solvable.

    Why containment matters now

    Autonomous agents are moving from demos into production. They’re reading emails, writing database records, triggering deployments, and calling external APIs. Each of those actions carries real business risk.

    The uncomfortable truth is that today’s agents are semi-trusted at best. Hallucinations aren’t just a chatbot annoyance — they’re an action an agent might take on a live system. A misinterpreted prompt can become a bulk delete. A confused tool call can cascade into an outage.

    Regulators are watching. The EU AI Act’s transparency and risk management requirements are already in force for high-risk systems. UK boards are starting to ask pointed questions about AI governance. If you can’t explain what your agents did and what guardrails were in place, you have a problem that no amount of prompt engineering will fix.

    The five containment patterns

    The patterns below are drawn from how we’d sandbox any untrusted or semi-trusted process — adapted for the specific risks that autonomous agents introduce. I’ve been implementing these in SentinelForge, a platform for running agents with strict governance, auditing, and observability.

    1. Process isolation

    Agents should run in their own isolated environment — a separate container or VM, not on a shared host alongside your production workloads.

    This means:

    • Dedicated runtime per agent or agent class, with its own filesystem and process namespace
    • Resource limits (CPU, memory, disk) so a runaway agent can’t starve other services
    • Immutable infrastructure — the agent’s environment is defined in code, rebuilt from scratch each run, and never patched in place

    If the agent is compromised or behaves unexpectedly, the blast radius stops at the container boundary.

    2. Network segmentation

    An agent should only be able to reach the endpoints it explicitly needs. Nothing more.

    In practice:

    • Egress filtering — whitelist approved domains and IP ranges; block everything else
    • No direct access to internal networks, metadata services (like cloud instance metadata endpoints), or other agents’ environments
    • DNS-level controls to prevent DNS exfiltration or calls to unexpected services

    Think of it like a corporate network with VLANs. The marketing intern doesn’t need access to the finance database. Your agent doesn’t need access to the public internet.

    3. Credential scoping

    This is where most agent deployments go wrong. A single long-lived API key with broad permissions is the agent equivalent of a master key.

    Instead:

    • Time-limited tokens — credentials that expire after a task or session, not months
    • Minimum privilege — if the agent only needs to read from a database, it doesn’t get write access
    • Brokered access — credentials are injected at runtime by a secrets manager, never stored in the agent’s environment or prompt
    • Per-task credentials — different actions get different tokens, so a compromise in one area doesn’t cascade

    4. Output validation

    Before an agent’s action hits a real system, it should pass through a validation layer.

    This means:

    • Schema validation — does the agent’s requested action match an expected structure?
    • Policy checks — is this action allowed given the current context, user, and risk level?
    • Human-in-the-loop gates for high-risk actions (bulk operations, deletions, external communications)
    • Dry-run capability — the ability to simulate what the agent would do before it does it

    This isn’t about slowing agents down. It’s about making sure the action that executes is the action you intended.

    5. Audit logging

    Every action an agent takes should be recorded in a tamper-evident log that’s separate from the agent itself.

    Capture:

    • What the agent did (the action, the target, the parameters)
    • Why it did it (the prompt, the reasoning chain, the tool call context)
    • When it happened (with reliable, centralised timestamps)
    • What the outcome was (success, failure, validation result)

    This log is your forensic trail when something goes wrong. It’s also your evidence for compliance, board reporting, and continuous improvement.

    A practical containment checklist

    If you’re deploying autonomous agents today, here’s a quick audit:

    • [ ] Each agent runs in an isolated container or VM with resource limits
    • [ ] Network egress is restricted to an explicit allowlist
    • [ ] No long-lived credentials — tokens are time-limited and minimum-privilege
    • [ ] Credentials are brokered at runtime, never hardcoded or stored in prompts
    • [ ] Agent actions pass through a validation or policy layer before execution
    • [ ] High-risk actions require human approval
    • [ ] All actions are logged to a tamper-evident, centralised audit trail
    • [ ] Logs are reviewed regularly, not just when something goes wrong

    If you can’t check at least six of those boxes, your agents are running with more freedom than most of your employees.

    The principle is simple

    Autonomous agents are powerful. That power needs guardrails — not because the technology is malicious, but because it’s fast, scalable, and imperfect. The same qualities that make agents valuable are the qualities that make containment essential.

    The patterns aren’t exotic. They’re the same ones we’ve used for decades to manage untrusted code, third-party integrations, and new employees. The difference is that agents can act at machine speed, so your guardrails need to be just as fast and just as automated.

    This is the problem I’m working on with SentinelForge — making these patterns practical and deployable, not just theoretical.

    If agent containment is on your roadmap, get in touch or take a look at the projects I’m building in this space.

  • The M365 Audit You Can Run Before Calling a Consultant

    The M365 Audit You Can Run Before Calling a Consultant

    Last month I published a post about the M365 security baseline most SMEs skip — the default tenant configuration that leaves businesses exposed because no one with security expertise has reviewed it since the initial migration.

    That post explained what to check. This one gives you the tool to check it.

    I built an open-source M365 Audit Toolkit and put it on GitHub. It is a set of read-only PowerShell modules that connect to your tenant using your credentials, interrogate the configuration, and produce a structured report of what is misconfigured, missing, or ranging beyond what you would reasonably accept. It does not change anything. It does not write data. It reads, evaluates, and reports.

    And it is designed so that you can run it yourself — before you pick up the phone to a consultant.

    Why a Self-Service Audit

    Most UK SMEs operate in a state of confident uncertainty about their M365 tenant. The IT provider set it up. It works. Email flows. SharePoint files sync. Teams calls connect. Everything appears fine.

    But “fine” is not a security posture. It is an absence of visible failure.

    The problem is that the gaps in an M365 tenant are invisible until they are not. A compromised account exposes every SharePoint site the user can reach. A forgotten guest account with access to a shared folder persists indefinitely. A mailbox that has never been audited leaves no trail when something goes wrong. These gaps do not generate alerts. They do not appear in the admin console with a red warning icon. They sit quietly until a breach unmasks them, a subject access request forces you to find every piece of personal data in the tenant, or a client’s due diligence questionnaire asks questions you cannot answer.

    I built the toolkit because the cost of a structured audit should not be a barrier to knowing where you stand. You should not need to engage a consultant just to find out whether your conditional access policies are configured, your admin roles are clean, or your retention policies align with your data retention schedule.

    Run the audit first. Understand your gaps. Then decide whether you need help closing them.

    What the Toolkit Covers

    The M365 Audit Toolkit is modular. You can run the full audit or pick the modules relevant to your situation currently and build from there. At present it covers five areas.

    Security Baseline Assessment. Checks MFA enforcement across all users, verifies that legacy authentication protocols is blocked, reviews conditional access policy configuration, mailbox auditing status, and DLP label deployment. This maps directly to the seven controls most SMEs skip.

    Admin Role Hygiene. Identifies every global administrator in the tenant, flags accounts that should not hold that role, checks for service principal permissions that grant equivalent access, and reports on role-based access control adoption for other administrative functions. If your tenant has four, six, or eight global admins, this module will tell you exactly who they are and what they can do.

    Guest Access Audit. Enumerates every external guest account in the tenant, what they can access, when they were last active, and whether access reviews are configured. This is the module that finds the former contractor who still has access to your shared folders three years after their engagement ended — which is more common than you would think.

    SharePoint and OneDrive Governance. Reviews site-level sharing settings, detects externally shared content, checks for orphaned sites, and reports on storage allocation and versioning configuration. If you have ever wondered how much of your organisation’s data is sitting in personal OneDrive accounts that the business does not back up and cannot access, this module provides the answer.

    AI Readiness Evaluation. Assesses your tenant’s readiness to adopt Copilot and other AI services safely — reviewing sensitivity label coverage, data loss prevention configuration, and access controls that would govern what an AI service can surface to users. This is increasingly relevant as Microsoft bundles Copilot into enterprise and business tiers.

    How to Run It

    The toolkit uses interactive MFA authentication, meaning it connects as you and inherits your permissions. It does not require app registrations, client secrets, or service accounts. If you have global reader access to the tenant, you can run the audit.

    The general workflow:

    1. Clone the repository from GitHub and review the modules.
    2. Connect to your tenant using interactive browser-based MFA authentication.
    3. Select the modules relevant to your assessment — security baseline, guest access, governance, AI readiness, or all of them.
    4. Review the output as a structured summary showing what passed, what failed, and what needs attention.

    The entire process can be completed in under an hour for a typical SME tenant. The output is a point-in-time snapshot — it does not persist data, and it does not leave artefacts in your tenant beyond the sign-in logs generated by your own authentication.

    When to Call a Consultant

    Running the audit yourself is the right first step. But there are situations where the findings will tell you that it is time to bring in expertise.

    • The report is long and you do not know where to start. If every module returns significant findings, the remediation effort is material and benefits from prioritisation and sequencing — which is exactly what a structured engagement provides.
    • You lack the permissions or authority to make changes. If the audit reveals that conditional access policies need to be reconfigured but your IT provider controls the tenant, the value of the report is in telling you exactly what to ask for and why.
    • You are preparing for client due diligence, a cyber insurance application, or a regulatory review. The audit output gives you a factual baseline, but remediation may need to be documented, evidenced, and in some cases certified.
    • You are adopting Copilot or other AI services. The AI readiness module will tell you where your gaps are, but closing them — sensitivity labels, access controls, DLP policies — is a project that benefits from experience doing it.

    The point is not to avoid consultants. The point is to arrive at that conversation informed, with a clear picture of what needs to be done rather than starting from a blank page.

    The Checklist Before You Pick Up the Phone

    If you want to understand your M365 tenant before committing to an engagement, here is the minimum viable assessment:

    • [ ] Run the security baseline module and resolve any critical MFA or conditional access findings
    • [ ] Run the admin role module and reduce global administrators to two or three accounts
    • [ ] Run the guest access module and revoke access for accounts no longer needed
    • [ ] Review the SharePoint governance report for externally shared sites and orphaned content
    • [ ] Review the AI readiness output if Copilot adoption is on your roadmap

    If you can work through that list and resolve the findings, your tenant is in materially better shape. If you cannot, you now have a specific, evidenced brief for what needs to happen next.

    The M365 Audit Toolkit is available now on GitHub. It is free, read-only, and built for exactly this use case.

    If you run it and the findings warrant a conversation, the Security & Compliance Strategy service covers M365 tenant configuration as part of a broader engagement. Or get in touch with your report — I can tell you within a 30-minute call whether the gaps are ones you can close yourself or whether it is time to bring in support.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Building a Secure Agent Control Plane With Observability From Day One

    Building a Secure Agent Control Plane With Observability From Day One

    AI agents are being connected to real business systems at pace. Email inboxes. CRM records. Finance platforms. Document stores. In many organisations, agents now read, write, and decide across these systems — not as a pilot, but as operational infrastructure.

    The problem is that the operational infrastructure around those agents has not kept pace. Teams connect agents to sensitive systems the same way they would connect a chatbot: authenticate, test, deploy. There is no logging layer, no approval gate, no cost tracking, no failure recovery. The agent works until it does not, and when it breaks, nobody knows why.

    The agents themselves are often well-designed. The prompts are thoughtful. The integrations function. But there is no control plane — no layer between the agent and the systems it touches that handles authorisation, logging, rate limiting, approval, and monitoring. Without one, you are running production workloads with no operations team.

    What an Agent Control Plane Actually Is

    An agent control plane is the operational layer that sits between your AI agents and the systems they interact with. It is not a single tool. It is a set of components that collectively ensure agents act safely, their actions are visible, and failures are recoverable.

    Think of it the way you would think about any production system. A web application has authentication, logging, rate limiting, monitoring, and alerting. A database has access controls, backup procedures, and performance monitoring. An agent that reads client records, sends emails, and updates financial data needs the same — arguably more, because its behaviour is less deterministic than a traditional application.

    The control plane answers a set of questions that every production system must answer: Who is allowed to do what? What actually happened? How much did it cost? What do we do when it fails? And can we prove all of this to an auditor, a regulator, or a board?

    The Six Components You Need

    Authentication broker. Agents need credentials to access systems, but those credentials should not live inside the agent or its prompt. A central broker issues scoped, time-limited tokens with minimum permissions. When the token expires, access stops. This is service account management applied to agents.

    Audit logger. Every agent action must be logged in a structured, tamper-evident format. Not just the final output — the full chain: what the agent was asked, which tools it called, what decisions it made, what it executed, and what it returned. Teams log the input and output but miss the intermediate steps. When an agent takes an unexpected action, you need to trace the reasoning that led to it.

    Approval workflow. Not every action should be autonomous. A control plane defines which actions require human approval before execution. Sending an internal summary might be automated. Sending a client-facing email or modifying a financial record should require sign-off. The approval gate sits in the control plane, not in the agent’s prompt — a prompt is a request, not an enforcement mechanism.

    Observability. Tools like Langfuse are designed for this. Observability for agents means tracking execution traces, token usage, latency, error rates, and cost per execution. It means answering questions like: which workflow failed most often last week? Which agent is consuming the most tokens? Is the error rate increasing?

    Cost tracking. Agentic AI is not free. Every model call, every tool invocation, every token consumed has a cost. Without cost tracking per agent, per workflow, and per execution, you cannot budget, optimise, or identify waste.

    Failure recovery. Agents fail. Models time out. APIs return errors. A control plane defines what happens: does it retry? Escalate to a human? Roll back? Without explicit failure recovery, a failed agent either silently drops the task or retries indefinitely, burning tokens and creating duplicate actions.

    A Concrete Example

    Consider an n8n workflow that processes incoming client data: it reads a form submission, enriches the record with data from an external API, updates the CRM, and sends a confirmation email.

    Without a control plane, this workflow runs with a static API key, no logging beyond n8n’s default execution history, no approval gate for the email send, and no cost tracking. If the CRM API returns an error, the workflow fails silently. If the agent sends a confirmation email to the wrong address because the form data was malformed, there is no record of what data it saw or why it made that decision.

    With a control plane, the same workflow runs with a scoped token from the authentication broker, full execution tracing through Langfuse, an approval gate that holds the email send when the confidence score is below a threshold, and automatic retry with escalation on failure. Every execution is logged with its full decision chain. Cost per execution is tracked and surfaced on a dashboard.

    The workflow is the same. The operational posture is completely different.

    Why From Day One Matters

    Retrofitting observability and security into a live system is significantly more expensive than building it in from the start. Not just in money — in time, risk, and organisational friction.

    When you add a control plane after agents are already in production, you must modify every existing workflow to route through the new layer, backfill logs for actions that already happened, and negotiate with teams accustomed to unfettered agent access. In practice, this work costs roughly three times what it would have cost to build the control plane first — before accounting for the risk exposure during the period agents ran without controls.

    Building it from day one means the control plane is part of the deployment process. Every new agent is onboarded through it. Every execution is logged from the first run. Every cost is tracked from the first token. No migration, no retrofitting, no gap.

    What to Do Next

    If you are running agentic AI in your organisation — or planning to — here is where to start.

    1. Inventory your agents. List every agent, workflow, and automated process that takes action on a business system. Include what systems it accesses, what credentials it uses, and what actions it can perform. You cannot secure what you have not catalogued.

    2. Add structured logging. For every agent execution, log the input, the decision chain, the action taken, the output, and the timestamp. Use a consistent schema. Send these logs to a central, append-only store. If you are using n8n, configure each node to log its input and output. Integrate a tracing tool like Langfuse from the start.

    3. Implement approval gates for high-risk actions. Define which actions require human sign-off: anything client-facing, anything that modifies financial data, anything that changes system configuration. Build the approval gate into the control plane, not into the agent’s prompt.

    4. Set up observability. Deploy an observability layer that tracks execution traces, error rates, token usage, and cost per workflow. Build dashboards. Set alerts for anomalies — sudden cost increases, rising error rates, unusual execution patterns.

    5. Define failure recovery. For every workflow, document what happens on failure. Retry logic, escalation paths, rollback procedures. Test these before you need them.

    The organisations that will extract the most value from agentic AI are not the ones that move fastest. They are the ones that move safely — with the infrastructure to observe, control, and recover from the inevitable failures.

    If you are building agentic systems and need help designing the control plane, observability, and security architecture around them, the AI & Automation Architecture service covers exactly this. For examples of what this looks like in practice, see recent projects.

  • Why AI Control Planes Need Observability From Day One

    Why AI Control Planes Need Observability From Day One

    AI agents are getting plugged into email, CRM, finance, and document systems faster than most teams can build the operations around them.

    That is the mismatch.

    The agent might be fine. The prompt might be fine. The integration might even work. What is often missing is the layer that says who can do what, what happened, what it cost, and how the system recovers when something breaks.

    What the control plane does

    Think of it as the operational wrapper around the agent. It handles:

    • authentication and scoped access
    • structured logging
    • approval gates for risky actions
    • observability and cost tracking
    • failure recovery

    That is not overkill. That is the minimum if the agent is touching production data.

    A concrete example

    Imagine a workflow that reads a form, enriches a record, updates a CRM, and sends a confirmation email.

    Without the wrapper, you get a static key, weak logs, and no real visibility when something goes wrong.

    With it, you get scoped access, review points, full traces, and a clearer answer when someone asks, “What did the system actually do?”

    Why day one matters

    Retrofitting this later is painful. It means backfilling logs, reworking workflows, and explaining to people why the guardrails arrived after the risk did.

    It is much easier to build the control plane alongside the agents and treat it as part of the product, not a patch.

    What to do first

    1. List every agent and workflow that can touch a real system.
    2. Log the full chain of inputs, decisions, and actions.
    3. Add human approval where the blast radius is high.
    4. Track failures and costs as first-class signals.

    That is enough to start building something you can actually trust.

  • What Real Delivery Discipline Looks Like

    What Real Delivery Discipline Looks Like

    A lot of teams say they care about security and reliability. Fewer teams build delivery habits that prove it.

    That difference matters more than most strategy decks admit. Hardening work is easy to describe in principle. Everyone agrees that authentication should be stronger, fallbacks should be safer, and monitoring should be more truthful. The real question is whether those concerns are allowed to shape the delivery path itself.

    If they are not, the system ends up with good intentions and brittle behaviour.

    Hardening is not a final polish step

    One of the most persistent delivery mistakes is treating hardening as something that happens after the “real” build is done.

    In practice, the opposite is true. Security-sensitive choices usually need to be made while the implementation is still taking shape:

    • what credentials are allowed to reach which component
    • what action is taken when verification fails
    • whether a degraded path is acceptable or whether the job should stop
    • what evidence an operator needs before calling something healthy
    • how much trust the system gives to generated output by default

    Those are not edge details. They shape the behaviour of the whole stack.

    If the hardening conversation starts only once the workflow is already live, the team ends up retrofitting controls around assumptions that were never designed for scrutiny.

    Monitoring has to be able to disagree with the diagram

    A good architecture diagram can still hide a weak operating model.

    I see this most often in monitoring. Teams build checks that confirm whether the process is running, but not whether the real task is succeeding in a meaningful way. That produces the worst kind of comfort: the dashboard looks clean while the system is quietly failing at the thing that matters.

    Useful monitoring needs to be willing to contradict appearances.

    For example, I would rather know that:

    • the workflow completed but returned structurally bad output
    • the service stayed up but fell onto the wrong provider path
    • the API responded but the downstream data plane was empty
    • the fallback recovered execution but created a lower-trust result

    That kind of signal is less glamorous than a green uptime chart, but it is far more useful when somebody has to operate the platform at speed.

    Delivery discipline is mostly about reducing hidden surprises

    The phrase “delivery discipline” can sound heavier than it needs to. I do not mean process theatre. I mean building a path to change that does not rely on luck.

    In practical terms, the teams that do this well tend to share a few habits:

    They make verification part of the job

    A change is not finished because the command returned zero. It is finished when the operator can show that the intended behaviour is visible from the outside.

    They do not improvise every rollback

    If a deployment path matters, the recovery path should already exist before the incident.

    They keep release logic legible

    The more a release process depends on one person remembering unwritten exceptions, the less mature it is.

    They distinguish activity from progress

    A lot of work can happen in a sprint without improving the trustworthiness of the system. Good delivery discipline asks whether the change made the operating model clearer, safer, or easier to verify.

    Why this matters in AI and automation work

    Automation makes weak delivery habits more expensive because the system can now repeat them at scale.

    If a manual operator makes one bad judgement call, the damage is limited. If the workflow itself contains vague trust boundaries, shallow health checks, or unclear fallback rules, the same weakness can repeat every minute.

    That is why I treat discipline as a feature.

    Not in the moral sense. In the architectural sense.

    A disciplined delivery path gives the team:

    • safer defaults
    • clearer escalation points
    • more credible monitoring
    • easier handover between operators
    • fewer silent regressions

    That is not bureaucracy. It is what makes the rest of the system believable.

    What I would check first

    If I am assessing whether a delivery process is genuinely disciplined, I usually start with a small checklist:

    • are the trust boundaries obvious?
    • does the monitoring test real behaviour or just process existence?
    • can somebody explain the fallback path without guessing?
    • is there a documented way to verify the live result after a change?
    • would another operator know what “healthy” means from the artefacts alone?

    If the answer to those questions is mostly “not yet”, then the team probably does not have a tooling problem. It has a delivery-discipline problem.

    The good news is that this is fixable. Most of the gains come from clearer defaults, sharper verification, and a willingness to treat operational trust as part of the design instead of a layer you add afterward.

    That work is rarely flashy. It just means the system behaves like something serious.

    If your stack needs stronger delivery discipline around hardening, observability, or AI workflows, the AI & Automation Architecture and Security Strategy work is built around exactly those problems. Or get in touch if you want a practical review of where the delivery path is still carrying too much guesswork.

Powered by