RichardHam.co.uk Blog

Tag: AI

  • When Systems Stop Relying on Guesswork

    When Systems Stop Relying on Guesswork

    A system usually becomes more trustworthy for one simple reason: the people running it stop having to guess.

    That sounds obvious, but it is often the difference between something that demos well and something that survives real use. Early on, teams tend to depend on memory, informal habits, and whoever happens to know how the pieces fit together. The more a platform grows, the more fragile that becomes. Hidden choices turn into operational risk. Unwritten expectations become inconsistent behaviour. Small misunderstandings start to show up as outages, wasted time, or avoidable rework.

    The most useful progress is often not glamorous. It looks like defining defaults, documenting recovery paths, tightening feedback loops, and making the important steps repeatable. In other words: less mystery, more system.

    What changed

    Across a typical week of work, the strongest improvements usually fall into a few categories.

    1. Decisions become explicit

    A lot of problems come from the same place: a critical choice was never written down.

    That might be a model selection rule, a deployment expectation, a fallback path, or a review step. If the team needs the same answer more than once, it should probably live in a policy, not in somebody’s head.

    Explicit decisions are easier to audit, easier to improve, and easier to hand over. They also reduce the chance that the system behaves differently depending on who touched it last.

    2. Observability becomes truthful

    Dashboards are useful only when they reflect reality.

    It is very easy to build something that looks informative while quietly hiding the thing you actually need to know. Wrong time windows, weak queries, misleading defaults, and over-optimistic thresholds can all create the illusion of control. The result is a lot of visual noise and very little operational value.

    Good observability is boring in the best possible way. It tells you what happened, when it happened, and whether the current state matches the story the interface is telling.

    3. Reuse reduces friction

    Reusable process is one of the highest-leverage things a team can build.

    Runbooks, checklists, CI steps, templates, decision logs, and bootstrap scripts all do the same job: they reduce the amount of context that has to be remembered manually. That makes the next delivery faster, but more importantly it makes the next delivery less dependent on luck.

    A mature team does not just ship features. It also ships the scaffolding that makes future work safer.

    4. Governance becomes routine

    Governance only helps when it is part of the rhythm of work.

    If decisions are captured sporadically, the rationale gets lost. If they are recorded regularly, they start to form a usable memory for the organisation. That is especially important in AI and automation work, where the consequences of a shortcut can show up much later than the moment it was taken.

    Routine does not have to mean bureaucracy. It can simply mean that important questions are answered in the same place, the same way, every time.

    Why this matters for AI and automation

    AI systems are often judged by how clever they look in isolation. That is the wrong benchmark.

    The real test is whether the system can be operated reliably by other people. Can it recover when something fails? Can it explain what it is doing? Can the team change it without fear? Can the output be trusted enough to act on?

    Those questions are answered by architecture, process, and discipline more than by novelty.

    If a workflow depends on a model, the model choice should be deliberate. If a chart drives decisions, it should be accurate. If a process gets used repeatedly, it should be documented. If a system matters, its operation should not depend on tribal knowledge.

    A practical standard

    A useful rule of thumb is this:

    • if it matters, write it down
    • if it repeats, make it reusable
    • if it can fail, define the fallback
    • if it is monitored, make sure the monitoring is honest
    • if it is operationally important, keep the reasoning close to the work

    That standard is not flashy, but it works.

    It makes AI systems easier to run.
    It makes automation easier to trust.
    It makes teams less dependent on memory.
    And it turns a collection of clever individual actions into something more durable.

    Closing thought

    The best systems are not the ones that never need attention. They are the ones that make attention easier to apply.

    When the defaults are clear, the checks are real, and the process is reusable, the whole stack becomes calmer. That is the kind of progress that matters most: not dramatic, but lasting.

    If you are trying to make an AI or automation workflow more reliable, the first step is usually not adding more complexity. It is removing guesswork.

    If you want help turning a messy operational process into something clearer and easier to trust, the AI & Automation Architecture service is a good place to start. Or get in touch for a practical conversation.

  • The Case for Explicit Policies

    Reliable systems do not emerge from good intentions. They emerge when the rules are explicit enough that another operator can understand what the system is supposed to do without reverse-engineering its behaviour from the wreckage.

    That sounds obvious, but it is still one of the most common gaps I see in automation and AI work. Teams build the workflow, connect the services, and get something working end to end. Then they leave the important decisions half-stated. Which provider is preferred? When should the fallback fire? What counts as a real health check? Which version of a process note is authoritative? The system may run, but the operating model is still fuzzy.

    The problem is not that people are careless. It is that policy work often looks less urgent than delivery work. Until something breaks, the invisible rule feels good enough.

    Where ambiguity shows up first

    The first place ambiguity appears is usually routing.

    A stack with multiple models, providers, queues, or execution paths always contains policy whether the team writes it down or not. If the preferred provider is too expensive for low-value tasks, that is policy. If one model is allowed for drafting but not for final output, that is policy. If a workflow should fall back only on timeout and not on quality failure, that is policy too.

    When none of that is written down clearly, people start inferring intent from whatever happened last.

    That is how teams end up with arguments that sound technical but are really operational:

    • “I thought the cheaper path was the default.”
    • “I assumed the fallback only applied during outages.”
    • “I didn’t realise this job was meant to stay on the private model.”
    • “I thought the dashboard alert meant the workflow had already rerouted.”

    None of those are bugs in isolation. They are symptoms of unstated policy.

    Why observability is part of policy

    The same issue appears in monitoring.

    A lot of dashboards tell you that a process is alive. That is not the same as proving the service is doing the right thing.

    For AI and automation systems, a truthful check usually needs to answer something more useful:

    • did the workflow complete the task it was supposed to complete?
    • did it use the intended path?
    • did it return data that looks structurally valid?
    • did the fallback stay dormant when the primary path was healthy?
    • can the operator see enough detail to explain the outcome afterward?

    If the check cannot answer those questions, the dashboard may still be visually tidy, but it is not giving the operator what they need.

    This is why I think observability should be treated as policy, not just instrumentation. The team has to decide what “working” actually means. Otherwise the monitoring layer simply reflects a vague assumption instead of a deliberate standard.

    Reuse is how policy survives handover

    The other quiet benefit of explicit policy is reuse.

    If a team has to rediscover the same routing rule, the same rollback sequence, or the same publishing checklist every time, then the policy is not really part of the system yet. It still lives in memory.

    That is expensive in a small team and dangerous in a growing one.

    Good reuse does not have to be elaborate. Often it is just a set of plain habits:

    • keep one canonical source of truth for important workflows
    • write fallback conditions near the implementation
    • keep short runbooks for the obvious failure modes
    • use the same naming and review patterns across similar jobs
    • record decisions before context evaporates

    None of that feels exciting while you are doing it. But it changes the quality of handover completely. A new operator no longer has to absorb the entire history of the stack before they can act safely.

    What explicit policy looks like in practice

    In practical terms, I look for a few simple signals.

    1. The preferred path is obvious

    The system should make it clear what happens first, what happens second, and under which conditions the fallback is allowed to take over.

    2. The checks reflect user reality

    A green dashboard should mean more than “something is listening on a port”. It should tell the operator whether the real job still works.

    3. Recovery paths exist before the incident

    If the first time a team documents the rollback sequence is during a failure, the policy work happened too late.

    4. Repeated patterns are actually reusable

    If the same kind of workflow appears three times, there should be a shared pattern instead of three slightly different tribal versions.

    Why this matters more with AI systems

    AI systems raise the cost of ambiguity because they turn a fuzzy rule into machine-speed inconsistency.

    In a manual process, unclear policy wastes time. In an automated one, it can silently change outputs, route work to the wrong provider, or create a trail that is too vague to audit later.

    That is why I think trustworthy AI is less about magic prompts and more about explicit operating rules.

    If the rules matter, write them down.

    If the outcome matters, check the real behaviour.

    If the workflow repeats, make it reusable.

    That does not make the system flashy. It makes it dependable.

    And in production, dependable usually wins.

    If you are building automation that needs to survive handover, escalation, and real operational scrutiny, the AI & Automation Architecture work is designed for exactly that. Or get in touch if you want a second pair of eyes on the operating model before the ambiguity becomes expensive.

  • GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    Introduction

    Some weeks are about shipping something obvious. Other weeks are about making the systems around the work easier to trust.

    This week looked like the second kind of week. Across the repos I reviewed, the change was direction, not just volume. Model routing became more deliberate. Monitoring became more accurate. Reusable delivery patterns became more explicit. Creative systems got clearer guardrails. And the small operational details that usually sit in the background were pulled into the foreground where they belong.

    That matters because the hardest part of running AI-assisted or automation-heavy systems is rarely the first prototype. The hard part is the handoff from “this works on my machine” to “this can be operated, explained, and improved without guesswork.” This week’s activity was a good reminder that trustworthy systems are built from a lot of very ordinary decisions made consistently.

    What happened

    Model routing stopped being an assumption and became a policy

    The clearest example came from hermes-mgmt. There was a real shift in how model choice is being handled: routing policy updates were merged, a stable default was pinned down, and provider-diverse fallback and outage behaviour were documented more explicitly. On top of that, role-based routing for research, coding, and review was defined instead of leaving model selection to whatever happened to be convenient in the moment.

    A lot of teams start with a simple belief: pick the biggest or newest model and let it do everything. That works until it doesn’t. Once you begin relying on models for different kinds of work, the distinction between “best model” and “best model for this job” becomes the difference between a healthy control plane and an expensive guessing game.

    I like this change because it turns model selection into an operating decision. It acknowledges that research work, coding work, and review work are not the same thing. It also gives the system a way to recover when a provider has issues, rather than forcing every incident into a manual exception.

    The surrounding work supports that direction too. A stale current-state snapshot was refreshed, the wider documentation was merged, and an earlier activation runbook for mixture-of-agents work shows that the system is being treated as something that needs operational memory, not just clever prompts.

    Observability was corrected, not merely expanded

    The biggest lesson from hamnet was the same one I have seen many times in production environments: a dashboard is only useful if it is telling the truth.

    This week’s work there focused on dashboard reliability rather than cosmetic additions. Grafana datasource timing was corrected, time series queries were given explicit intervals, default time windows were adjusted, and Pushgateway queries were wrapped so they render consistently. There was also a practical backlog item to track services that still need HTTPS equivalents, which is the kind of operational debt that gets forgotten unless somebody writes it down.

    I think this is a good example of how observability should be approached. It is tempting to treat monitoring as a collection of panels: add some graphs, colour some thresholds, and call it “visibility.” But real observability is more demanding. It means the query window is right, the step size is right, the defaults are right, and the chart is actually showing the thing you think it is showing.

    That is not a minor distinction. If your dashboard is technically live but operationally misleading, it can create more risk than it removes.

    There was also a useful signal in the issues around connecting a second Hermes instance as a remote subagent. That kind of work says the control plane is no longer just about one local setup. It is beginning to think in terms of distributed operation, where trust, connectivity, and monitoring all need to survive outside a single happy path.

    Creative systems became more governable

    HamMediaLabs showed a similar pattern, but from a different angle. The work there focused on creative-quality instrumentation, creative-ops playbooks, a per-brand visual identity layer, and a short-form voice system.

    That sounds very different from routing policies and dashboard fixes, but the underlying principle is the same: if you want something to scale, you need a repeatable operating model.

    In creative work, people often assume the value lives in taste alone. Taste matters, of course, but once you are managing multiple brands or content streams, taste has to be supported by structure. Otherwise every output becomes a one-off argument.

    The interesting thing here is that the repo is not just producing creative assets. It is creating a system for how those assets should be judged and delivered. That is the right order. First you define the identity. Then you define the checks. Then you can move quickly without drifting away from what the work is supposed to feel like.

    Delivery got more reusable, not just more active

    richardham-co-uk-ConsultancyOS was another good example of this week’s theme. CI was added, licensing and contributing guidance were written, a docs handbook appeared, and a SessionStart hook was introduced. In parallel, there was work on reusable delivery-repo patterns, operational dashboards, and prompt-pack integration.

    That is the sort of repo work that pays off over time in a way that is easy to underestimate.

    Most teams can write a one-off project. Fewer teams can turn that project into a template that makes the next one easier. The difference is not just technical polish. It is whether the project has been made reusable. When the CI, contribution rules, documentation, and bootstrap hooks are all captured explicitly, the next engagement starts with less friction and fewer assumptions.

    I see the same thing in the planning and monitoring work happening in lk-ai-roadmap. The hosting and runtime map was clarified, with a clear “no standing servers” direction. Token usage and cost-versus-budget monitoring were documented. Network and security monitoring agent work moved forward as a pilot spec. And the baseline capture kit and endorsement tracking suggest the project is being built with traceability in mind rather than as an ad hoc collection of tasks.

    That is what mature delivery looks like: fewer hidden decisions, more explicit rules, and less dependence on individual memory.

    Key takeaways

    The pattern across all of this work was remarkably consistent.

    • Policies are better than assumptions. Model routing, fallback behaviour, and role-based usage are clearer when they are written down.
    • Monitoring is only useful when it is accurate. A dashboard with the wrong time window or query behaviour is not a shortcut to clarity.
    • Reuse is a force multiplier. CI, docs, hooks, and templates make the next delivery easier than the last one.
    • Creative work benefits from the same discipline as infrastructure. Identity and quality checks are what make scale possible without drift.
    • Governance works best when it becomes ordinary. The more decisions are captured in the system itself, the less they depend on memory or heroics.

    What I take from this week is simple: the work is moving from experimentation to governable systems. That is a good sign. It means the platform is getting more predictable without losing momentum, and the people operating it are making fewer decisions in the dark.

    That is the kind of progress that compounds.

    If you are building AI systems, automation, or operational dashboards and want help turning the moving parts into something easier to trust, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to run.

  • GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    Introduction

    Some weeks are about shipping something obvious. Other weeks are about making the systems around the work easier to trust.

    This week looked like the second kind of week. Across the repos I reviewed, the change was direction, not just volume. Model routing became more deliberate. Monitoring became more accurate. Reusable delivery patterns became more explicit. Creative systems got clearer guardrails. And the small operational details that usually sit in the background were pulled into the foreground where they belong.

    That matters because the hardest part of running AI-assisted or automation-heavy systems is rarely the first prototype. The hard part is the handoff from “this works on my machine” to “this can be operated, explained, and improved without guesswork.” This week’s activity was a good reminder that trustworthy systems are built from a lot of very ordinary decisions made consistently.

    What happened

    Model routing stopped being an assumption and became a policy

    The clearest example came from hermes-mgmt. There was a real shift in how model choice is being handled: routing policy updates were merged, a stable default was pinned down, and provider-diverse fallback and outage behaviour were documented more explicitly. On top of that, role-based routing for research, coding, and review was defined instead of leaving model selection to whatever happened to be convenient in the moment.

    A lot of teams start with a simple belief: pick the biggest or newest model and let it do everything. That works until it doesn’t. Once you begin relying on models for different kinds of work, the distinction between “best model” and “best model for this job” becomes the difference between a healthy control plane and an expensive guessing game.

    I like this change because it turns model selection into an operating decision. It acknowledges that research work, coding work, and review work are not the same thing. It also gives the system a way to recover when a provider has issues, rather than forcing every incident into a manual exception.

    The surrounding work supports that direction too. A stale current-state snapshot was refreshed, the wider documentation was merged, and an earlier activation runbook for mixture-of-agents work shows that the system is being treated as something that needs operational memory, not just clever prompts.

    Observability was corrected, not merely expanded

    The biggest lesson from hamnet was the same one I have seen many times in production environments: a dashboard is only useful if it is telling the truth.

    This week’s work there focused on dashboard reliability rather than cosmetic additions. Grafana datasource timing was corrected, time series queries were given explicit intervals, default time windows were adjusted, and Pushgateway queries were wrapped so they render consistently. There was also a practical backlog item to track services that still need HTTPS equivalents, which is the kind of operational debt that gets forgotten unless somebody writes it down.

    I think this is a good example of how observability should be approached. It is tempting to treat monitoring as a collection of panels: add some graphs, colour some thresholds, and call it “visibility.” But real observability is more demanding. It means the query window is right, the step size is right, the defaults are right, and the chart is actually showing the thing you think it is showing.

    That is not a minor distinction. If your dashboard is technically live but operationally misleading, it can create more risk than it removes.

    There was also a useful signal in the issues around connecting a second Hermes instance as a remote subagent. That kind of work says the control plane is no longer just about one local setup. It is beginning to think in terms of distributed operation, where trust, connectivity, and monitoring all need to survive outside a single happy path.

    Creative systems became more governable

    HamMediaLabs showed a similar pattern, but from a different angle. The work there focused on creative-quality instrumentation, creative-ops playbooks, a per-brand visual identity layer, and a short-form voice system.

    That sounds very different from routing policies and dashboard fixes, but the underlying principle is the same: if you want something to scale, you need a repeatable operating model.

    In creative work, people often assume the value lives in taste alone. Taste matters, of course, but once you are managing multiple brands or content streams, taste has to be supported by structure. Otherwise every output becomes a one-off argument.

    The interesting thing here is that the repo is not just producing creative assets. It is creating a system for how those assets should be judged and delivered. That is the right order. First you define the identity. Then you define the checks. Then you can move quickly without drifting away from what the work is supposed to feel like.

    Delivery got more reusable, not just more active

    richardham-co-uk-ConsultancyOS was another good example of this week’s theme. CI was added, licensing and contributing guidance were written, a docs handbook appeared, and a SessionStart hook was introduced. In parallel, there was work on reusable delivery-repo patterns, operational dashboards, and prompt-pack integration.

    That is the sort of repo work that pays off over time in a way that is easy to underestimate.

    Most teams can write a one-off project. Fewer teams can turn that project into a template that makes the next one easier. The difference is not just technical polish. It is whether the project has been made reusable. When the CI, contribution rules, documentation, and bootstrap hooks are all captured explicitly, the next engagement starts with less friction and fewer assumptions.

    I see the same thing in the planning and monitoring work happening in lk-ai-roadmap. The hosting and runtime map was clarified, with a clear “no standing servers” direction. Token usage and cost-versus-budget monitoring were documented. Network and security monitoring agent work moved forward as a pilot spec. And the baseline capture kit and endorsement tracking suggest the project is being built with traceability in mind rather than as an ad hoc collection of tasks.

    That is what mature delivery looks like: fewer hidden decisions, more explicit rules, and less dependence on individual memory.

    Key takeaways

    The pattern across all of this work was remarkably consistent.

    • Policies are better than assumptions. Model routing, fallback behaviour, and role-based usage are clearer when they are written down.
    • Monitoring is only useful when it is accurate. A dashboard with the wrong time window or query behaviour is not a shortcut to clarity.
    • Reuse is a force multiplier. CI, docs, hooks, and templates make the next delivery easier than the last one.
    • Creative work benefits from the same discipline as infrastructure. Identity and quality checks are what make scale possible without drift.
    • Governance works best when it becomes ordinary. The more decisions are captured in the system itself, the less they depend on memory or heroics.

    What I take from this week is simple: the work is moving from experimentation to governable systems. That is a good sign. It means the platform is getting more predictable without losing momentum, and the people operating it are making fewer decisions in the dark.

    That is the kind of progress that compounds.

    If you are building AI systems, automation, or operational dashboards and want help turning the moving parts into something easier to trust, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to run.

  • Why Agentic AI Needs Audit Trails, Not Just Clever Prompts

    Why Agentic AI Needs Audit Trails, Not Just Clever Prompts

    The conversation around AI in business has shifted. For the past two years, most organisations have experimented with generative AI as a productivity tool — drafting documents, summarising meetings, answering questions. The model receives input, produces output, and a human reviews the result.

    That is no longer the whole picture.

    Across the organisations I work with, AI is moving from answering questions to taking actions. Agents execute n8n workflows that move data between systems. They trigger Hermes agents that read, write, and decide. They call APIs, update records, send messages, and make operational decisions — sometimes with human approval, sometimes without.

    This is agentic AI: systems that do not just respond, but act. It creates a governance problem that clever prompts alone cannot solve.

    The Governance Gap

    When a human makes a decision in a business process, there is usually a trace. An email sent, a form submitted, a system log entry, a manager’s sign-off. When something goes wrong, you can reconstruct the sequence of events. You can ask: what were they asked to do, what did they do, and why?

    When an AI agent executes an action, that trace often does not exist. The agent receives a prompt, processes it through one or more model calls, and performs an action. If the action is wrong — if it updates the wrong record, sends a message to the wrong person, or executes a workflow it should not have — the organisation is left with a result and no explanation.

    This is not a theoretical risk. In my own infrastructure, I have built agentic workflows that interact with live systems. The difference between a safe deployment and an unsafe one is not the quality of the prompt. It is whether the system logs enough information to reconstruct what happened after the fact.

    What Happens Without Audit Trails

    Without audit trails, three things break down.

    You cannot reconstruct events. If an agent produces an incorrect output or takes an unintended action, you need to know what input it received, which model or tool it called, what intermediate decisions it made, and what action it executed. Without this, debugging is guesswork. You are trying to diagnose a problem without access to the patient’s notes.

    You cannot establish accountability. When an automated system causes harm — a data breach, a financial error, a compliance failure — someone needs to be able to explain what happened. Under UK GDPR, the accountability principle requires organisations to demonstrate compliance, not just claim it. If your AI agent processes personal data and you cannot show what it did with that data, you are not compliant. It does not matter how good the system is in theory.

    You cannot improve the system. Agentic AI systems iterate. You adjust prompts, change tool configurations, add guardrails. Without structured logs of what each execution actually did, you are optimising in the dark.

    What a Practical Audit Trail Looks Like

    An audit trail for an agentic AI system does not need to be complex. It needs to be consistent and complete. At minimum, each agent execution should capture:

    • Input received. What was the agent asked to do? This includes the user’s request, any system context, and the prompt that was constructed.
    • Decision chain. What steps did the agent take? Which tools did it call? What intermediate outputs did it produce? For multi-step agents, this is the sequence of reasoning that led to the final action.
    • Action taken. What did the agent actually do? Which API was called, which record was updated, which message was sent.
    • Output produced. What was the final result returned to the user or passed to the next step in the workflow.
    • Timestamp and identity. When did this happen, and which agent or workflow executed it?

    This is not excessive. It is the same information you would expect from any business system that takes actions on data. The fact that the system is powered by a language model does not change the requirement — it increases it, because the system’s behaviour is less deterministic and harder to predict.

    The Regulatory Dimension

    For UK organisations, this is not optional. UK GDPR Article 5(2) establishes the accountability principle: you must be able to demonstrate that you comply with data protection principles. If an AI agent processes personal data — and most business agents do — you need to show what data it accessed, what it did with that data, and on what basis.

    Article 30 requires records of processing activities. An agent that processes client records, employee data, or customer information is conducting processing activity. If you cannot produce a log of that activity, you do not have the records the regulation requires.

    For financial services firms, the FCA’s operational resilience framework adds another layer. Important business services must withstand disruption and recover. If your AI agents are part of an important business service — processing transactions, managing client communications, monitoring risk — you need to understand how they behave, what they depend on, and what happens when they fail. Audit trails are the evidence base for your resilience assessment.

    How to Implement Audit Trails in Agent Workflows

    The good news is that the tooling exists. You do not need to build this from scratch.

    Structured logging at every node. In n8n workflows, each node can be configured to log its input and output. For agentic workflows, you should log at minimum the trigger, each decision point, and the final action. Use a consistent schema — timestamp, node name, input summary, output summary, and execution status — so that logs are searchable and comparable.

    Observability platforms. Tools like Langfuse are designed for exactly this purpose. They capture the full execution trace of an agent: prompt, model response, tool calls, and final output. When connected to your workflow engine, they give you a queryable record of every agent execution without building custom logging infrastructure.

    Immutable storage. Audit logs must be tamper-evident. If the log can be modified after the fact, it is not an audit trail — it is a diary. Store logs in append-only storage with access controls that prevent modification. This can be as simple as writing to a write-once bucket or using a logging service that enforces retention policies.

    Structured output from agents. Design your agents to return structured output, not just free text. A JSON response that includes the action taken, the target system, and the rationale is far more useful for auditing than a paragraph of prose. This also makes it easier to validate agent behaviour programmatically — you can check that the action taken is within the set of permitted actions before it executes.

    Regular review. Audit trails are only useful if someone looks at them. Build a review cadence — weekly for high-risk agents, monthly for lower-risk ones — where you sample executions and check for anomalies.

    The Bottom Line

    The organisations that will get the most value from agentic AI are not the ones with the most sophisticated prompts. They are the ones that can trust their agents to act safely, verify what those agents did, and improve them over time.

    Audit trails are the foundation of that trust. They are how you move from hoping your agents behave to knowing they do. They are how you satisfy regulators, reassure boards, and sleep at night.

    If you are deploying agentic AI in your organisation — or planning to — audit infrastructure is not a phase-two consideration. It is a prerequisite.

    If you are building agentic AI systems and need help establishing the governance, architecture, and audit infrastructure to support them, the AI & Automation Architecture service covers exactly this. For a broader conversation about where your organisation stands, get in touch.

  • Why Agentic AI Needs Audit Trails, Not Just Clever Prompts

    Why Agentic AI Needs Audit Trails, Not Just Clever Prompts

    The conversation around AI in business has shifted. For the past two years, most organisations have experimented with generative AI as a productivity tool — drafting documents, summarising meetings, answering questions. The model receives input, produces output, and a human reviews the result.

    That is no longer the whole picture.

    Across the organisations I work with, AI is moving from answering questions to taking actions. Agents execute n8n workflows that move data between systems. They trigger Hermes agents that read, write, and decide. They call APIs, update records, send messages, and make operational decisions — sometimes with human approval, sometimes without.

    This is agentic AI: systems that do not just respond, but act. It creates a governance problem that clever prompts alone cannot solve.

    The Governance Gap

    When a human makes a decision in a business process, there is usually a trace. An email sent, a form submitted, a system log entry, a manager’s sign-off. When something goes wrong, you can reconstruct the sequence of events. You can ask: what were they asked to do, what did they do, and why?

    When an AI agent executes an action, that trace often does not exist. The agent receives a prompt, processes it through one or more model calls, and performs an action. If the action is wrong — if it updates the wrong record, sends a message to the wrong person, or executes a workflow it should not have — the organisation is left with a result and no explanation.

    This is not a theoretical risk. In my own infrastructure, I have built agentic workflows that interact with live systems. The difference between a safe deployment and an unsafe one is not the quality of the prompt. It is whether the system logs enough information to reconstruct what happened after the fact.

    What Happens Without Audit Trails

    Without audit trails, three things break down.

    You cannot reconstruct events. If an agent produces an incorrect output or takes an unintended action, you need to know what input it received, which model or tool it called, what intermediate decisions it made, and what action it executed. Without this, debugging is guesswork. You are trying to diagnose a problem without access to the patient’s notes.

    You cannot establish accountability. When an automated system causes harm — a data breach, a financial error, a compliance failure — someone needs to be able to explain what happened. Under UK GDPR, the accountability principle requires organisations to demonstrate compliance, not just claim it. If your AI agent processes personal data and you cannot show what it did with that data, you are not compliant. It does not matter how good the system is in theory.

    You cannot improve the system. Agentic AI systems iterate. You adjust prompts, change tool configurations, add guardrails. Without structured logs of what each execution actually did, you are optimising in the dark.

    What a Practical Audit Trail Looks Like

    An audit trail for an agentic AI system does not need to be complex. It needs to be consistent and complete. At minimum, each agent execution should capture:

    • Input received. What was the agent asked to do? This includes the user’s request, any system context, and the prompt that was constructed.
    • Decision chain. What steps did the agent take? Which tools did it call? What intermediate outputs did it produce? For multi-step agents, this is the sequence of reasoning that led to the final action.
    • Action taken. What did the agent actually do? Which API was called, which record was updated, which message was sent.
    • Output produced. What was the final result returned to the user or passed to the next step in the workflow.
    • Timestamp and identity. When did this happen, and which agent or workflow executed it?

    This is not excessive. It is the same information you would expect from any business system that takes actions on data. The fact that the system is powered by a language model does not change the requirement — it increases it, because the system’s behaviour is less deterministic and harder to predict.

    The Regulatory Dimension

    For UK organisations, this is not optional. UK GDPR Article 5(2) establishes the accountability principle: you must be able to demonstrate that you comply with data protection principles. If an AI agent processes personal data — and most business agents do — you need to show what data it accessed, what it did with that data, and on what basis.

    Article 30 requires records of processing activities. An agent that processes client records, employee data, or customer information is conducting processing activity. If you cannot produce a log of that activity, you do not have the records the regulation requires.

    For financial services firms, the FCA’s operational resilience framework adds another layer. Important business services must withstand disruption and recover. If your AI agents are part of an important business service — processing transactions, managing client communications, monitoring risk — you need to understand how they behave, what they depend on, and what happens when they fail. Audit trails are the evidence base for your resilience assessment.

    How to Implement Audit Trails in Agent Workflows

    The good news is that the tooling exists. You do not need to build this from scratch.

    Structured logging at every node. In n8n workflows, each node can be configured to log its input and output. For agentic workflows, you should log at minimum the trigger, each decision point, and the final action. Use a consistent schema — timestamp, node name, input summary, output summary, and execution status — so that logs are searchable and comparable.

    Observability platforms. Tools like Langfuse are designed for exactly this purpose. They capture the full execution trace of an agent: prompt, model response, tool calls, and final output. When connected to your workflow engine, they give you a queryable record of every agent execution without building custom logging infrastructure.

    Immutable storage. Audit logs must be tamper-evident. If the log can be modified after the fact, it is not an audit trail — it is a diary. Store logs in append-only storage with access controls that prevent modification. This can be as simple as writing to a write-once bucket or using a logging service that enforces retention policies.

    Structured output from agents. Design your agents to return structured output, not just free text. A JSON response that includes the action taken, the target system, and the rationale is far more useful for auditing than a paragraph of prose. This also makes it easier to validate agent behaviour programmatically — you can check that the action taken is within the set of permitted actions before it executes.

    Regular review. Audit trails are only useful if someone looks at them. Build a review cadence — weekly for high-risk agents, monthly for lower-risk ones — where you sample executions and check for anomalies.

    The Bottom Line

    The organisations that will get the most value from agentic AI are not the ones with the most sophisticated prompts. They are the ones that can trust their agents to act safely, verify what those agents did, and improve them over time.

    Audit trails are the foundation of that trust. They are how you move from hoping your agents behave to knowing they do. They are how you satisfy regulators, reassure boards, and sleep at night.

    If you are deploying agentic AI in your organisation — or planning to — audit infrastructure is not a phase-two consideration. It is a prerequisite.

    If you are building agentic AI systems and need help establishing the governance, architecture, and audit infrastructure to support them, the AI & Automation Architecture service covers exactly this. For a broader conversation about where your organisation stands, get in touch.

  • Why Agentic AI Needs Audit Trails, Not Just Clever Prompts

    Why AI Workflows Need Audit Trails

    The conversation around AI has shifted. It is no longer just about drafting text or summarising meetings. More and more often, these systems are taking actions on live business processes.

    That is where the risk changes shape.

    A clever prompt can produce a good-looking result. It cannot tell you what happened after the fact if the output was wrong.

    The gap

    When a person makes a decision in a process, there is usually some trace of it. An email, a ticket, a sign-off, a log entry. With an AI system, that trace is often thin or missing.

    If the system updates the wrong record or sends the wrong message, the team is left with a result and very little explanation.

    What breaks without logs

    – You cannot reconstruct the sequence of events.
    – You cannot show who approved what.
    – You cannot improve the workflow with confidence.

    That is not just a debugging problem. It is an accountability problem.

    What a useful audit trail looks like

    At minimum, every execution should capture:

    – the input it received
    – the steps it took
    – the action it actually executed
    – the output it produced
    – the timestamp and identity of the run

    That is enough to answer the questions that matter later.

    The practical bit

    The tooling is already there. Structured logs, append-only storage, and reviewable traces are all enough to get started. The main thing is to design for visibility before the workflow is under pressure.

    If the system is allowed to act, it should also be required to explain itself.

  • The Case for Explicit Policies

    The Case for Explicit Policies

    Reliable systems do not emerge from good intentions. They emerge when the rules are explicit enough that another operator can understand what the system is supposed to do without reverse-engineering its behaviour from the wreckage.

    That sounds obvious, but it is still one of the most common gaps I see in automation and AI work. Teams build the workflow, connect the services, and get something working end to end. Then they leave the important decisions half-stated. Which provider is preferred? When should the fallback fire? What counts as a real health check? Which version of a process note is authoritative? The system may run, but the operating model is still fuzzy.

    The problem is not that people are careless. It is that policy work often looks less urgent than delivery work. Until something breaks, the invisible rule feels good enough.

    Where ambiguity shows up first

    The first place ambiguity appears is usually routing.

    A stack with multiple models, providers, queues, or execution paths always contains policy whether the team writes it down or not. If the preferred provider is too expensive for low-value tasks, that is policy. If one model is allowed for drafting but not for final output, that is policy. If a workflow should fall back only on timeout and not on quality failure, that is policy too.

    When none of that is written down clearly, people start inferring intent from whatever happened last.

    That is how teams end up with arguments that sound technical but are really operational:

    • “I thought the cheaper path was the default.”
    • “I assumed the fallback only applied during outages.”
    • “I didn’t realise this job was meant to stay on the private model.”
    • “I thought the dashboard alert meant the workflow had already rerouted.”

    None of those are bugs in isolation. They are symptoms of unstated policy.

    Why observability is part of policy

    The same issue appears in monitoring.

    A lot of dashboards tell you that a process is alive. That is not the same as proving the service is doing the right thing.

    For AI and automation systems, a truthful check usually needs to answer something more useful:

    • did the workflow complete the task it was supposed to complete?
    • did it use the intended path?
    • did it return data that looks structurally valid?
    • did the fallback stay dormant when the primary path was healthy?
    • can the operator see enough detail to explain the outcome afterward?

    If the check cannot answer those questions, the dashboard may still be visually tidy, but it is not giving the operator what they need.

    This is why I think observability should be treated as policy, not just instrumentation. The team has to decide what “working” actually means. Otherwise the monitoring layer simply reflects a vague assumption instead of a deliberate standard.

    Reuse is how policy survives handover

    The other quiet benefit of explicit policy is reuse.

    If a team has to rediscover the same routing rule, the same rollback sequence, or the same publishing checklist every time, then the policy is not really part of the system yet. It still lives in memory.

    That is expensive in a small team and dangerous in a growing one.

    Good reuse does not have to be elaborate. Often it is just a set of plain habits:

    • keep one canonical source of truth for important workflows
    • write fallback conditions near the implementation
    • keep short runbooks for the obvious failure modes
    • use the same naming and review patterns across similar jobs
    • record decisions before context evaporates

    None of that feels exciting while you are doing it. But it changes the quality of handover completely. A new operator no longer has to absorb the entire history of the stack before they can act safely.

    What explicit policy looks like in practice

    In practical terms, I look for a few simple signals.

    1. The preferred path is obvious

    The system should make it clear what happens first, what happens second, and under which conditions the fallback is allowed to take over.

    2. The checks reflect user reality

    A green dashboard should mean more than “something is listening on a port”. It should tell the operator whether the real job still works.

    3. Recovery paths exist before the incident

    If the first time a team documents the rollback sequence is during a failure, the policy work happened too late.

    4. Repeated patterns are actually reusable

    If the same kind of workflow appears three times, there should be a shared pattern instead of three slightly different tribal versions.

    Why this matters more with AI systems

    AI systems raise the cost of ambiguity because they turn a fuzzy rule into machine-speed inconsistency.

    In a manual process, unclear policy wastes time. In an automated one, it can silently change outputs, route work to the wrong provider, or create a trail that is too vague to audit later.

    That is why I think trustworthy AI is less about magic prompts and more about explicit operating rules.

    If the rules matter, write them down.

    If the outcome matters, check the real behaviour.

    If the workflow repeats, make it reusable.

    That does not make the system flashy. It makes it dependable.

    And in production, dependable usually wins.

    If you are building automation that needs to survive handover, escalation, and real operational scrutiny, the AI & Automation Architecture work is designed for exactly that. Or get in touch if you want a second pair of eyes on the operating model before the ambiguity becomes expensive.

  • Why AI Workflows Need Audit Trails

    Why AI Workflows Need Audit Trails

    The conversation around AI has shifted. It is no longer just about drafting text or summarising meetings. More and more often, these systems are taking actions on live business processes.

    That is where the risk changes shape.

    A clever prompt can produce a good-looking result. It cannot tell you what happened after the fact if the output was wrong.

    The gap

    When a person makes a decision in a process, there is usually some trace of it. An email, a ticket, a sign-off, a log entry. With an AI system, that trace is often thin or missing.

    If the system updates the wrong record or sends the wrong message, the team is left with a result and very little explanation.

    What breaks without logs

    • You cannot reconstruct the sequence of events.
    • You cannot show who approved what.
    • You cannot improve the workflow with confidence.

    That is not just a debugging problem. It is an accountability problem.

    What a useful audit trail looks like

    At minimum, every execution should capture:

    • the input it received
    • the steps it took
    • the action it actually executed
    • the output it produced
    • the timestamp and identity of the run

    That is enough to answer the questions that matter later.

    The practical bit

    The tooling is already there. Structured logs, append-only storage, and reviewable traces are all enough to get started. The main thing is to design for visibility before the workflow is under pressure.

    If the system is allowed to act, it should also be required to explain itself.

  • Weekly GitHub Activity — 22–29 June 2026

    Introduction

    Another busy week in the repositories. Between infrastructure automation, platform hardening, commercial product development, and a small mountain of documentation, the commit logs tell the story of a systemthat’s rapidly moving from “hand-crafted and held together with SSH sessions” toward something far more repeatable and governed. Let’s walk through what landed.

    What Happened

    hermes-mgmt: Hardening, Governance, and AI Spend Controls

    The hermes-mgmt repository saw 10 commits and 20 pull requests this week, almost all of them orbiting two themes: Tier 2 operational hardening and AI spend governance.

    On the hardening side, a suite of operator runbooks and governance records landed — documenting the execution of Tier 2 hardening scripts, session decisions from backlog triage, and a full operator TODO instruction document. PR #694 tackled memory infrastructure head-on, covering a Qdrant deployment, gateway shutdown procedures, and Telegram 429 rate-limit handling. PR #695 addressed bug remediation across Spotify integration, n8n authentication, vision/video backend issues, and a 429 retry storm that was causing more harm than good.

    The AI spend work is particularly interesting. PR #696 introduced budget policies, cost telemetry, context-optimisation strategies, and prompt-caching controls — essentially an AI spend optimisation control plane. As LLM-powered workloads grow, having a formal mechanism to track, cap, and optimise spend is no longer nice-to-have; it’s operational hygiene. The design docs for child components that accompanied this work sketch out a system where every agent invocation carries a cost context, and budget alarms fire before the invoice does.

    Security compliance also got attention. PR #691 documented progress against SecureScore controls SEC-002, SEC-006, RES-002, and MAT-001 — a reminder that even in a solo-operator environment, treating compliance as code keeps the audit trail honest.

    A quieter but important fix: PR #640 made the Mem0→Letta memory backfill idempotent, preventing duplicate records on re-runs. Small change, big impact on data integrity.

    hamnet: The Control Plane Takes Shape

    hamnet was the most active repository this week — 10 commits, 18 pull requests, and 40 open issues. The headline is the emergence of a single-pane control-plane dashboard (PR #183), backed by a growing collection of infrastructure collectors.

    The control-plane work is being built incrementally and deliberately:

    • PR #177 established the identity model, a static inventory MVP, and the first batch of workload collectors.
    • PR #179 added a second batch — AgentRadar, Ollama status, repository state, and NetBox integration.
    • PR #176 fixed Grafana drift authentication and a tvheadend timeout issue that was causing monitoring gaps.

    On the infrastructure-as-code front, PR #178 delivered a comprehensive IaC roadmap with schemas, runbooks, and a system-classification model. PR #194 proposed enforcing unattended-upgrades policies on external cloud hosts through Ansible — a defensive measure that ensures security patches land even when nobody’s watching.

    Several fixes made day-to-day operations smoother: a script to tag missing VMs with ansible:managed in NetBox (commit 33d2fd9), a UTF-8 locale fix for ansible-inventory (commit 198d9b8), and adding Gitea to the Docker container detection logic (commit a950c67) — a gap that meant the monitoring stack was blind to a self-hosted service.

    A Docker image drift management runbook (commit 81fa372) and operator runbooks for 11 infrastructure issues (commit 8f09076) round out the picture. hamnet is building the operational backbone: not just running services, but knowing what’s running, whether it should be, and what to do when it isn’t.

    The 40 open issues deserve a mention — they’re a candid inventory of what needs attention, from an Oracle scanner that’s out of sync and a PVE exporter reporting zero metrics to Loki missing log-shipping and stale NetBox entries. Filing them is half the battle.

    ms365-agentic-ai: Collectors, Cost Controls, and Production Readiness

    The ms365-agentic-ai repository is accelerating. Five commits this week laid significant groundwork:

    • A deployable Azure Functions host with a run-history sink and live-path tests (commit 5465364).
    • Live read-only fetch wiring for all collectors (commit 416c537).
    • A cost-management component combining an estimator with an Azure cost collector (commit 86ab202).
    • Production-readiness runbook with dry-run and preflight tooling (commit 5da09c4).
    • A simple start guide, docs index, and FAQ/troubleshooting section (commit 212729a).

    The 10 open issues (#6–#15) are essentially the product roadmap: approval-gated remediation models, audit logging, Teams health digests, Azure resource health and Defender XDR collectors, Graph permissions matrices, and licensing/cost matrices. What’s notable is the breadth — this isn’t just a tool, it’s an operational platform being designed with governance from the start.

    richardham-co-uk-ConsultancyOS: Commercial Foundations

    ConsultancyOS — the commercial operating-system template — got its infrastructure in order this week. PR #12 added CI, licensing, a contributing guide, and a docs handbook. A SessionStart hook for web sessions (commit 1a54d22) shows the product thinking extending into the user experience.

    The 9 open issues map out the productisation journey: operational dashboards, a GitHub delivery-repo template, LLM integration, n8n workflow scaffolding, accountant review processes, company identity, and integrations with Dolibarr CRM and Invoice Ninja. This is a project that’s explicitly moving from “personal tooling” toward “repeatable commercial offering.”

    on-maintenance-ai-roadmap: Client-Facing Delivery

    The on-maintenance-ai-roadmap repository focused on communication and deployment model this week. A client-facing project plan (PR #23), an Architecture Decision Record for production hosting in client-owned Azure (PR #21), and a deploy-from-GitHub infrastructure model (PR #24) all point to a delivery pattern where the client retains ownership and control of their environment while the project provides the automation and intelligence layer.

    ai-cloud-credits-grant: Cloud Funding Applications

    Six commits across the ai-cloud-credits-grant repository added usage plans for NVIDIA Inception, Cloudflare, Oracle OCI, AWS, and Google Cloud. These are structured applications for cloud-provider credit programmes — the kind of unglamorous but essential work that unlocks compute capacity for projects that need it. The 6 open issues track the remaining work: a master company profile, applications to the major providers, and an evidence pack.

    Key Takeaways

    1. The control plane is real. hamnet’s dashboard, collectors, and identity model aren’t prototypes — they’re an incremental build toward a system that provides visibility and governance across the entire infrastructure. When you can see everything from one pane, you can fix things before they break.

    2. AI spend is now a first-class concern. The hermes-mgmt AI spend control plane — with budget policies, cost telemetry, and prompt caching — reflects a mature approach to LLM operations. The era of “just call the API and hope the bill is reasonable” is over, at least in this stack.

    3. Documentation as a delivery artifact. Across every repository this week, documentation wasn’t an afterthought — it was a first-class deliverable. Operator runbooks, governance records, ADRs, client-facing plans, and onboarding guides shipped alongside the code. This is the behaviour that separates sustainable projects from weekend hacks.

    4. Compliance isn’t optional, even for small teams. The SecureScore work in hermes-mgmt and the approval-gated remediation model in ms365-agentic-ai show that compliance and auditability are being designed in, not bolted on. That’s cheaper and more effective at any scale.

    5. Open issues are a feature, not a bug. hamnet’s 40 open issues and ms365-agentic-ai’s 10 open issues aren’t backlogs — they’re transparent roadmaps. Filing what needs doing, and making it visible, is how solo operators avoid the “I’ll remember that later” trap.

    This post is part of a weekly series summarising GitHub activity across the project portfolio. Previous entries are available in the blog archive.

Powered by CookieAdmin