RichardHam.co.uk Blog

Tag: security

  • The Case for Explicit Policies

    Reliable systems do not emerge from good intentions. They emerge when the rules are explicit enough that another operator can understand what the system is supposed to do without reverse-engineering its behaviour from the wreckage.

    That sounds obvious, but it is still one of the most common gaps I see in automation and AI work. Teams build the workflow, connect the services, and get something working end to end. Then they leave the important decisions half-stated. Which provider is preferred? When should the fallback fire? What counts as a real health check? Which version of a process note is authoritative? The system may run, but the operating model is still fuzzy.

    The problem is not that people are careless. It is that policy work often looks less urgent than delivery work. Until something breaks, the invisible rule feels good enough.

    Where ambiguity shows up first

    The first place ambiguity appears is usually routing.

    A stack with multiple models, providers, queues, or execution paths always contains policy whether the team writes it down or not. If the preferred provider is too expensive for low-value tasks, that is policy. If one model is allowed for drafting but not for final output, that is policy. If a workflow should fall back only on timeout and not on quality failure, that is policy too.

    When none of that is written down clearly, people start inferring intent from whatever happened last.

    That is how teams end up with arguments that sound technical but are really operational:

    • “I thought the cheaper path was the default.”
    • “I assumed the fallback only applied during outages.”
    • “I didn’t realise this job was meant to stay on the private model.”
    • “I thought the dashboard alert meant the workflow had already rerouted.”

    None of those are bugs in isolation. They are symptoms of unstated policy.

    Why observability is part of policy

    The same issue appears in monitoring.

    A lot of dashboards tell you that a process is alive. That is not the same as proving the service is doing the right thing.

    For AI and automation systems, a truthful check usually needs to answer something more useful:

    • did the workflow complete the task it was supposed to complete?
    • did it use the intended path?
    • did it return data that looks structurally valid?
    • did the fallback stay dormant when the primary path was healthy?
    • can the operator see enough detail to explain the outcome afterward?

    If the check cannot answer those questions, the dashboard may still be visually tidy, but it is not giving the operator what they need.

    This is why I think observability should be treated as policy, not just instrumentation. The team has to decide what “working” actually means. Otherwise the monitoring layer simply reflects a vague assumption instead of a deliberate standard.

    Reuse is how policy survives handover

    The other quiet benefit of explicit policy is reuse.

    If a team has to rediscover the same routing rule, the same rollback sequence, or the same publishing checklist every time, then the policy is not really part of the system yet. It still lives in memory.

    That is expensive in a small team and dangerous in a growing one.

    Good reuse does not have to be elaborate. Often it is just a set of plain habits:

    • keep one canonical source of truth for important workflows
    • write fallback conditions near the implementation
    • keep short runbooks for the obvious failure modes
    • use the same naming and review patterns across similar jobs
    • record decisions before context evaporates

    None of that feels exciting while you are doing it. But it changes the quality of handover completely. A new operator no longer has to absorb the entire history of the stack before they can act safely.

    What explicit policy looks like in practice

    In practical terms, I look for a few simple signals.

    1. The preferred path is obvious

    The system should make it clear what happens first, what happens second, and under which conditions the fallback is allowed to take over.

    2. The checks reflect user reality

    A green dashboard should mean more than “something is listening on a port”. It should tell the operator whether the real job still works.

    3. Recovery paths exist before the incident

    If the first time a team documents the rollback sequence is during a failure, the policy work happened too late.

    4. Repeated patterns are actually reusable

    If the same kind of workflow appears three times, there should be a shared pattern instead of three slightly different tribal versions.

    Why this matters more with AI systems

    AI systems raise the cost of ambiguity because they turn a fuzzy rule into machine-speed inconsistency.

    In a manual process, unclear policy wastes time. In an automated one, it can silently change outputs, route work to the wrong provider, or create a trail that is too vague to audit later.

    That is why I think trustworthy AI is less about magic prompts and more about explicit operating rules.

    If the rules matter, write them down.

    If the outcome matters, check the real behaviour.

    If the workflow repeats, make it reusable.

    That does not make the system flashy. It makes it dependable.

    And in production, dependable usually wins.

    If you are building automation that needs to survive handover, escalation, and real operational scrutiny, the AI & Automation Architecture work is designed for exactly that. Or get in touch if you want a second pair of eyes on the operating model before the ambiguity becomes expensive.

  • GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    Introduction

    Some weeks are about shipping something obvious. Other weeks are about making the systems around the work easier to trust.

    This week looked like the second kind of week. Across the repos I reviewed, the change was direction, not just volume. Model routing became more deliberate. Monitoring became more accurate. Reusable delivery patterns became more explicit. Creative systems got clearer guardrails. And the small operational details that usually sit in the background were pulled into the foreground where they belong.

    That matters because the hardest part of running AI-assisted or automation-heavy systems is rarely the first prototype. The hard part is the handoff from “this works on my machine” to “this can be operated, explained, and improved without guesswork.” This week’s activity was a good reminder that trustworthy systems are built from a lot of very ordinary decisions made consistently.

    What happened

    Model routing stopped being an assumption and became a policy

    The clearest example came from hermes-mgmt. There was a real shift in how model choice is being handled: routing policy updates were merged, a stable default was pinned down, and provider-diverse fallback and outage behaviour were documented more explicitly. On top of that, role-based routing for research, coding, and review was defined instead of leaving model selection to whatever happened to be convenient in the moment.

    A lot of teams start with a simple belief: pick the biggest or newest model and let it do everything. That works until it doesn’t. Once you begin relying on models for different kinds of work, the distinction between “best model” and “best model for this job” becomes the difference between a healthy control plane and an expensive guessing game.

    I like this change because it turns model selection into an operating decision. It acknowledges that research work, coding work, and review work are not the same thing. It also gives the system a way to recover when a provider has issues, rather than forcing every incident into a manual exception.

    The surrounding work supports that direction too. A stale current-state snapshot was refreshed, the wider documentation was merged, and an earlier activation runbook for mixture-of-agents work shows that the system is being treated as something that needs operational memory, not just clever prompts.

    Observability was corrected, not merely expanded

    The biggest lesson from hamnet was the same one I have seen many times in production environments: a dashboard is only useful if it is telling the truth.

    This week’s work there focused on dashboard reliability rather than cosmetic additions. Grafana datasource timing was corrected, time series queries were given explicit intervals, default time windows were adjusted, and Pushgateway queries were wrapped so they render consistently. There was also a practical backlog item to track services that still need HTTPS equivalents, which is the kind of operational debt that gets forgotten unless somebody writes it down.

    I think this is a good example of how observability should be approached. It is tempting to treat monitoring as a collection of panels: add some graphs, colour some thresholds, and call it “visibility.” But real observability is more demanding. It means the query window is right, the step size is right, the defaults are right, and the chart is actually showing the thing you think it is showing.

    That is not a minor distinction. If your dashboard is technically live but operationally misleading, it can create more risk than it removes.

    There was also a useful signal in the issues around connecting a second Hermes instance as a remote subagent. That kind of work says the control plane is no longer just about one local setup. It is beginning to think in terms of distributed operation, where trust, connectivity, and monitoring all need to survive outside a single happy path.

    Creative systems became more governable

    HamMediaLabs showed a similar pattern, but from a different angle. The work there focused on creative-quality instrumentation, creative-ops playbooks, a per-brand visual identity layer, and a short-form voice system.

    That sounds very different from routing policies and dashboard fixes, but the underlying principle is the same: if you want something to scale, you need a repeatable operating model.

    In creative work, people often assume the value lives in taste alone. Taste matters, of course, but once you are managing multiple brands or content streams, taste has to be supported by structure. Otherwise every output becomes a one-off argument.

    The interesting thing here is that the repo is not just producing creative assets. It is creating a system for how those assets should be judged and delivered. That is the right order. First you define the identity. Then you define the checks. Then you can move quickly without drifting away from what the work is supposed to feel like.

    Delivery got more reusable, not just more active

    richardham-co-uk-ConsultancyOS was another good example of this week’s theme. CI was added, licensing and contributing guidance were written, a docs handbook appeared, and a SessionStart hook was introduced. In parallel, there was work on reusable delivery-repo patterns, operational dashboards, and prompt-pack integration.

    That is the sort of repo work that pays off over time in a way that is easy to underestimate.

    Most teams can write a one-off project. Fewer teams can turn that project into a template that makes the next one easier. The difference is not just technical polish. It is whether the project has been made reusable. When the CI, contribution rules, documentation, and bootstrap hooks are all captured explicitly, the next engagement starts with less friction and fewer assumptions.

    I see the same thing in the planning and monitoring work happening in lk-ai-roadmap. The hosting and runtime map was clarified, with a clear “no standing servers” direction. Token usage and cost-versus-budget monitoring were documented. Network and security monitoring agent work moved forward as a pilot spec. And the baseline capture kit and endorsement tracking suggest the project is being built with traceability in mind rather than as an ad hoc collection of tasks.

    That is what mature delivery looks like: fewer hidden decisions, more explicit rules, and less dependence on individual memory.

    Key takeaways

    The pattern across all of this work was remarkably consistent.

    • Policies are better than assumptions. Model routing, fallback behaviour, and role-based usage are clearer when they are written down.
    • Monitoring is only useful when it is accurate. A dashboard with the wrong time window or query behaviour is not a shortcut to clarity.
    • Reuse is a force multiplier. CI, docs, hooks, and templates make the next delivery easier than the last one.
    • Creative work benefits from the same discipline as infrastructure. Identity and quality checks are what make scale possible without drift.
    • Governance works best when it becomes ordinary. The more decisions are captured in the system itself, the less they depend on memory or heroics.

    What I take from this week is simple: the work is moving from experimentation to governable systems. That is a good sign. It means the platform is getting more predictable without losing momentum, and the people operating it are making fewer decisions in the dark.

    That is the kind of progress that compounds.

    If you are building AI systems, automation, or operational dashboards and want help turning the moving parts into something easier to trust, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to run.

  • What Real Delivery Discipline Looks Like

    What Real Delivery Discipline Looks Like

    A lot of teams say they care about security and reliability. Fewer teams build delivery habits that prove it.

    That difference matters more than most strategy decks admit. Hardening work is easy to describe in principle. Everyone agrees that authentication should be stronger, fallbacks should be safer, and monitoring should be more truthful. The real question is whether those concerns are allowed to shape the delivery path itself.

    If they are not, the system ends up with good intentions and brittle behaviour.

    Hardening is not a final polish step

    One of the most persistent delivery mistakes is treating hardening as something that happens after the “real” build is done.

    In practice, the opposite is true. Security-sensitive choices usually need to be made while the implementation is still taking shape:

    • what credentials are allowed to reach which component
    • what action is taken when verification fails
    • whether a degraded path is acceptable or whether the job should stop
    • what evidence an operator needs before calling something healthy
    • how much trust the system gives to generated output by default

    Those are not edge details. They shape the behaviour of the whole stack.

    If the hardening conversation starts only once the workflow is already live, the team ends up retrofitting controls around assumptions that were never designed for scrutiny.

    Monitoring has to be able to disagree with the diagram

    A good architecture diagram can still hide a weak operating model.

    I see this most often in monitoring. Teams build checks that confirm whether the process is running, but not whether the real task is succeeding in a meaningful way. That produces the worst kind of comfort: the dashboard looks clean while the system is quietly failing at the thing that matters.

    Useful monitoring needs to be willing to contradict appearances.

    For example, I would rather know that:

    • the workflow completed but returned structurally bad output
    • the service stayed up but fell onto the wrong provider path
    • the API responded but the downstream data plane was empty
    • the fallback recovered execution but created a lower-trust result

    That kind of signal is less glamorous than a green uptime chart, but it is far more useful when somebody has to operate the platform at speed.

    Delivery discipline is mostly about reducing hidden surprises

    The phrase “delivery discipline” can sound heavier than it needs to. I do not mean process theatre. I mean building a path to change that does not rely on luck.

    In practical terms, the teams that do this well tend to share a few habits:

    They make verification part of the job

    A change is not finished because the command returned zero. It is finished when the operator can show that the intended behaviour is visible from the outside.

    They do not improvise every rollback

    If a deployment path matters, the recovery path should already exist before the incident.

    They keep release logic legible

    The more a release process depends on one person remembering unwritten exceptions, the less mature it is.

    They distinguish activity from progress

    A lot of work can happen in a sprint without improving the trustworthiness of the system. Good delivery discipline asks whether the change made the operating model clearer, safer, or easier to verify.

    Why this matters in AI and automation work

    Automation makes weak delivery habits more expensive because the system can now repeat them at scale.

    If a manual operator makes one bad judgement call, the damage is limited. If the workflow itself contains vague trust boundaries, shallow health checks, or unclear fallback rules, the same weakness can repeat every minute.

    That is why I treat discipline as a feature.

    Not in the moral sense. In the architectural sense.

    A disciplined delivery path gives the team:

    • safer defaults
    • clearer escalation points
    • more credible monitoring
    • easier handover between operators
    • fewer silent regressions

    That is not bureaucracy. It is what makes the rest of the system believable.

    What I would check first

    If I am assessing whether a delivery process is genuinely disciplined, I usually start with a small checklist:

    • are the trust boundaries obvious?
    • does the monitoring test real behaviour or just process existence?
    • can somebody explain the fallback path without guessing?
    • is there a documented way to verify the live result after a change?
    • would another operator know what “healthy” means from the artefacts alone?

    If the answer to those questions is mostly “not yet”, then the team probably does not have a tooling problem. It has a delivery-discipline problem.

    The good news is that this is fixable. Most of the gains come from clearer defaults, sharper verification, and a willingness to treat operational trust as part of the design instead of a layer you add afterward.

    That work is rarely flashy. It just means the system behaves like something serious.

    If your stack needs stronger delivery discipline around hardening, observability, or AI workflows, the AI & Automation Architecture and Security Strategy work is built around exactly those problems. Or get in touch if you want a practical review of where the delivery path is still carrying too much guesswork.

  • GitHub Weekly — Security Hardening, Monitoring, and Delivery Discipline

    GitHub Weekly — Security Hardening, Monitoring, and Delivery Discipline

    Introduction

    Some weeks are about visible progress: a new feature, a fresh integration, a launch announcement. Other weeks are about the work that keeps those things safe to run. This was one of those weeks.

    Across the repos I reviewed, the pattern was consistent. Security controls were tightened, monitoring got corrected rather than just expanded, operational runbooks were written down instead of left in people’s heads, and delivery work kept moving in small, deliberate steps. That is not glamorous work, but it is the kind of work that stops a platform from becoming fragile.

    The theme that emerged was simple: maintenance is not separate from delivery. It is part of delivery.

    What happened

    Security work moved from incident response to operating discipline

    In hermes-mgmt, the work was clearly shaped by hard lessons. Prompt injection concerns were captured in the changelog and backlog, webhook authentication was hardened, Qdrant backups were automated, and CVE verification was folded into the wider hardening effort.

    That combination matters. A lot of teams treat security as a single event — a scan, a policy review, a one-off fix. In practice, it is a chain of habits. You need authentication on the edges, backups in the middle, and verification at the end. If any one of those is missing, the system may still look healthy right up until the day it is not.

    What I liked most about this pattern was that the repo was not just reacting to a risk. It was converting the risk into repeatable operations: operator scripts, backlog triage, and the documentation needed to make the next decision easier than the last one.

    Monitoring got corrected, not just decorated

    In hamnet, the week was dominated by practical observability fixes. Grafana datasource timing was corrected, dashboard defaults were adjusted to now-6h, pushgateway metrics were refined, and time series queries were updated so they would render reliably instead of misleadingly.

    That sounds minor until you have lived through a dashboard that lies to you.

    Monitoring failures are often subtle. The data is present, the panels load, and the colours look reassuring — but the time interval is wrong, the step size is off, or the query is hiding the very behaviour you wanted to see. In that state, dashboards become theatre. They exist, but they do not help.

    There was also a useful governance signal in the tracking of HTTP-only services that still need HTTPS evaluation. That is exactly the sort of operational backlog item that gets forgotten unless someone records it explicitly. Good monitoring is not only about more charts. It is about acknowledging what is still unfinished and making that visible.

    AI automation was tightened with guardrails and auditability

    ms365-agentic-ai was another strong example of maturity through constraint. Prompt-injection and AI-security guards were hardened, a hub-and-spoke team operating model was merged in, a plan-only remote command action was introduced, and cost-first routing became part of the model strategy.

    This is the right direction for agentic systems. The mistake many teams make is to optimise for autonomy too early. They want the agent to do more before they have decided how to constrain it, audit it, or roll it back. That is how you end up with a clever demo and an unsafe production system.

    The useful pattern here is the opposite: make autonomy conditional. Limit the command surface. Make some actions plan-only. Keep an audit trail. Route routine work to cheaper models where appropriate. When a system can explain what it intended to do before it does it, you are much closer to something governable.

    Delivery work stayed concrete and decomposed

    ricambio-ai-roadmap looked like a good example of steady execution rather than big-bang progress. Provider configuration UI work landed, pagination and infrastructure fixes followed, LaunchDaemon plists and boot startup scripts were added, and a run-sheet for an onsite email MVP visit was written down.

    There was also a practical focus on PII redaction and provider bake-off work, which tells me the team is thinking about both usability and safety. That combination is important. A delivery track is strongest when it can handle real-world constraints without losing momentum.

    The interesting part here is not any one commit. It is the sequence. Research, configuration, bootstrapping, runbooks, redaction, delivery. That is what healthy delivery looks like when it is being treated as a system rather than a one-off project.

    Governance became a daily rhythm

    The control-tower repo reinforced the same message from a different angle. Daily “Decision Desk” issues kept appearing, which tells me governance is not being treated as a monthly review or a backlog afterthought. It is being made into a rhythm.

    That matters because the absence of rhythm is what creates drift. If you only review decisions occasionally, then the rationale behind them disappears into chat threads and memory. If you capture them daily, you create a trail that can be checked, challenged, and reused.

    Why this matters

    The common thread across all of this work is that automation only becomes trustworthy when it is surrounded by discipline.

    Security controls matter because agents, webhooks, and operational tools all expand the number of ways a system can be influenced.

    Monitoring matters because a dashboard that is technically live but operationally wrong can be worse than no dashboard at all.

    Runbooks matter because the first time something breaks, people rarely have time to invent the procedure from scratch.

    And delivery discipline matters because every project eventually reaches a point where execution is more about coordination, sequencing, and proof than raw feature count.

    I think that is why this week felt coherent even though the repos were diverse. The activity was not random. It was converging on a single operating idea: if you want automation to scale, you have to make the surrounding system easier to trust.

    That means:

    • tightening the edge of the system with authentication and verification
    • making dashboards accurate enough to act on
    • writing down the operational steps before they are needed in anger
    • splitting risky work into smaller, reviewable pieces
    • keeping audit trails and decision logs close to the work itself

    Those are not just engineering habits. They are management habits.

    Key takeaways

    • Security hardening is strongest when it becomes a repeatable operating pattern, not a one-off response.
    • Monitoring is only useful if the time windows, query intervals, and defaults actually match the system you are trying to observe.
    • Agentic systems need guardrails, auditability, and plan-only steps before they need more autonomy.
    • Delivery improves when work is decomposed into concrete, sequenced steps with runbooks and release notes alongside the code.
    • Governance becomes useful when it is captured as a daily rhythm rather than a periodic ceremony.

    This week did not hinge on a single dramatic release. It showed something more important: the system is getting better at protecting itself while it moves.

    That is the work that keeps the next launch calm.

    If you are building automation, AI systems, or operational dashboards and want help turning the hidden work into a reliable operating model, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to trust.

  • What Real Delivery Discipline Looks Like

    A lot of teams say they care about security and reliability. Fewer teams build delivery habits that prove it.

    That difference matters more than most strategy decks admit. Hardening work is easy to describe in principle. Everyone agrees that authentication should be stronger, fallbacks should be safer, and monitoring should be more truthful. The real question is whether those concerns are allowed to shape the delivery path itself.

    If they are not, the system ends up with good intentions and brittle behaviour.

    Hardening is not a final polish step

    One of the most persistent delivery mistakes is treating hardening as something that happens after the “real” build is done.

    In practice, the opposite is true. Security-sensitive choices usually need to be made while the implementation is still taking shape:

    • what credentials are allowed to reach which component
    • what action is taken when verification fails
    • whether a degraded path is acceptable or whether the job should stop
    • what evidence an operator needs before calling something healthy
    • how much trust the system gives to generated output by default

    Those are not edge details. They shape the behaviour of the whole stack.

    If the hardening conversation starts only once the workflow is already live, the team ends up retrofitting controls around assumptions that were never designed for scrutiny.

    Monitoring has to be able to disagree with the diagram

    A good architecture diagram can still hide a weak operating model.

    I see this most often in monitoring. Teams build checks that confirm whether the process is running, but not whether the real task is succeeding in a meaningful way. That produces the worst kind of comfort: the dashboard looks clean while the system is quietly failing at the thing that matters.

    Useful monitoring needs to be willing to contradict appearances.

    For example, I would rather know that:

    • the workflow completed but returned structurally bad output
    • the service stayed up but fell onto the wrong provider path
    • the API responded but the downstream data plane was empty
    • the fallback recovered execution but created a lower-trust result

    That kind of signal is less glamorous than a green uptime chart, but it is far more useful when somebody has to operate the platform at speed.

    Delivery discipline is mostly about reducing hidden surprises

    The phrase “delivery discipline” can sound heavier than it needs to. I do not mean process theatre. I mean building a path to change that does not rely on luck.

    In practical terms, the teams that do this well tend to share a few habits:

    They make verification part of the job

    A change is not finished because the command returned zero. It is finished when the operator can show that the intended behaviour is visible from the outside.

    They do not improvise every rollback

    If a deployment path matters, the recovery path should already exist before the incident.

    They keep release logic legible

    The more a release process depends on one person remembering unwritten exceptions, the less mature it is.

    They distinguish activity from progress

    A lot of work can happen in a sprint without improving the trustworthiness of the system. Good delivery discipline asks whether the change made the operating model clearer, safer, or easier to verify.

    Why this matters in AI and automation work

    Automation makes weak delivery habits more expensive because the system can now repeat them at scale.

    If a manual operator makes one bad judgement call, the damage is limited. If the workflow itself contains vague trust boundaries, shallow health checks, or unclear fallback rules, the same weakness can repeat every minute.

    That is why I treat discipline as a feature.

    Not in the moral sense. In the architectural sense.

    A disciplined delivery path gives the team:

    • safer defaults
    • clearer escalation points
    • more credible monitoring
    • easier handover between operators
    • fewer silent regressions

    That is not bureaucracy. It is what makes the rest of the system believable.

    What I would check first

    If I am assessing whether a delivery process is genuinely disciplined, I usually start with a small checklist:

    • are the trust boundaries obvious?
    • does the monitoring test real behaviour or just process existence?
    • can somebody explain the fallback path without guessing?
    • is there a documented way to verify the live result after a change?
    • would another operator know what “healthy” means from the artefacts alone?

    If the answer to those questions is mostly “not yet”, then the team probably does not have a tooling problem. It has a delivery-discipline problem.

    The good news is that this is fixable. Most of the gains come from clearer defaults, sharper verification, and a willingness to treat operational trust as part of the design instead of a layer you add afterward.

    That work is rarely flashy. It just means the system behaves like something serious.

    If your stack needs stronger delivery discipline around hardening, observability, or AI workflows, the AI & Automation Architecture and Security Strategy work is built around exactly those problems. Or get in touch if you want a practical review of where the delivery path is still carrying too much guesswork.

  • GitHub Weekly — Security Hardening, Monitoring, and Delivery Discipline

    GitHub Weekly — Security Hardening, Monitoring, and Delivery Discipline

    Introduction

    Some weeks are about visible progress: a new feature, a fresh integration, a launch announcement. Other weeks are about the work that keeps those things safe to run. This was one of those weeks.

    Across the repos I reviewed, the pattern was consistent. Security controls were tightened, monitoring got corrected rather than just expanded, operational runbooks were written down instead of left in people’s heads, and delivery work kept moving in small, deliberate steps. That is not glamorous work, but it is the kind of work that stops a platform from becoming fragile.

    The theme that emerged was simple: maintenance is not separate from delivery. It is part of delivery.

    What happened

    Security work moved from incident response to operating discipline

    In hermes-mgmt, the work was clearly shaped by hard lessons. Prompt injection concerns were captured in the changelog and backlog, webhook authentication was hardened, Qdrant backups were automated, and CVE verification was folded into the wider hardening effort.

    That combination matters. A lot of teams treat security as a single event — a scan, a policy review, a one-off fix. In practice, it is a chain of habits. You need authentication on the edges, backups in the middle, and verification at the end. If any one of those is missing, the system may still look healthy right up until the day it is not.

    What I liked most about this pattern was that the repo was not just reacting to a risk. It was converting the risk into repeatable operations: operator scripts, backlog triage, and the documentation needed to make the next decision easier than the last one.

    Monitoring got corrected, not just decorated

    In hamnet, the week was dominated by practical observability fixes. Grafana datasource timing was corrected, dashboard defaults were adjusted to now-6h, pushgateway metrics were refined, and time series queries were updated so they would render reliably instead of misleadingly.

    That sounds minor until you have lived through a dashboard that lies to you.

    Monitoring failures are often subtle. The data is present, the panels load, and the colours look reassuring — but the time interval is wrong, the step size is off, or the query is hiding the very behaviour you wanted to see. In that state, dashboards become theatre. They exist, but they do not help.

    There was also a useful governance signal in the tracking of HTTP-only services that still need HTTPS evaluation. That is exactly the sort of operational backlog item that gets forgotten unless someone records it explicitly. Good monitoring is not only about more charts. It is about acknowledging what is still unfinished and making that visible.

    AI automation was tightened with guardrails and auditability

    ms365-agentic-ai was another strong example of maturity through constraint. Prompt-injection and AI-security guards were hardened, a hub-and-spoke team operating model was merged in, a plan-only remote command action was introduced, and cost-first routing became part of the model strategy.

    This is the right direction for agentic systems. The mistake many teams make is to optimise for autonomy too early. They want the agent to do more before they have decided how to constrain it, audit it, or roll it back. That is how you end up with a clever demo and an unsafe production system.

    The useful pattern here is the opposite: make autonomy conditional. Limit the command surface. Make some actions plan-only. Keep an audit trail. Route routine work to cheaper models where appropriate. When a system can explain what it intended to do before it does it, you are much closer to something governable.

    Delivery work stayed concrete and decomposed

    ricambio-ai-roadmap looked like a good example of steady execution rather than big-bang progress. Provider configuration UI work landed, pagination and infrastructure fixes followed, LaunchDaemon plists and boot startup scripts were added, and a run-sheet for an onsite email MVP visit was written down.

    There was also a practical focus on PII redaction and provider bake-off work, which tells me the team is thinking about both usability and safety. That combination is important. A delivery track is strongest when it can handle real-world constraints without losing momentum.

    The interesting part here is not any one commit. It is the sequence. Research, configuration, bootstrapping, runbooks, redaction, delivery. That is what healthy delivery looks like when it is being treated as a system rather than a one-off project.

    Governance became a daily rhythm

    The control-tower repo reinforced the same message from a different angle. Daily “Decision Desk” issues kept appearing, which tells me governance is not being treated as a monthly review or a backlog afterthought. It is being made into a rhythm.

    That matters because the absence of rhythm is what creates drift. If you only review decisions occasionally, then the rationale behind them disappears into chat threads and memory. If you capture them daily, you create a trail that can be checked, challenged, and reused.

    Why this matters

    The common thread across all of this work is that automation only becomes trustworthy when it is surrounded by discipline.

    Security controls matter because agents, webhooks, and operational tools all expand the number of ways a system can be influenced.

    Monitoring matters because a dashboard that is technically live but operationally wrong can be worse than no dashboard at all.

    Runbooks matter because the first time something breaks, people rarely have time to invent the procedure from scratch.

    And delivery discipline matters because every project eventually reaches a point where execution is more about coordination, sequencing, and proof than raw feature count.

    I think that is why this week felt coherent even though the repos were diverse. The activity was not random. It was converging on a single operating idea: if you want automation to scale, you have to make the surrounding system easier to trust.

    That means:

    • tightening the edge of the system with authentication and verification
    • making dashboards accurate enough to act on
    • writing down the operational steps before they are needed in anger
    • splitting risky work into smaller, reviewable pieces
    • keeping audit trails and decision logs close to the work itself

    Those are not just engineering habits. They are management habits.

    Key takeaways

    • Security hardening is strongest when it becomes a repeatable operating pattern, not a one-off response.
    • Monitoring is only useful if the time windows, query intervals, and defaults actually match the system you are trying to observe.
    • Agentic systems need guardrails, auditability, and plan-only steps before they need more autonomy.
    • Delivery improves when work is decomposed into concrete, sequenced steps with runbooks and release notes alongside the code.
    • Governance becomes useful when it is captured as a daily rhythm rather than a periodic ceremony.

    This week did not hinge on a single dramatic release. It showed something more important: the system is getting better at protecting itself while it moves.

    That is the work that keeps the next launch calm.

    If you are building automation, AI systems, or operational dashboards and want help turning the hidden work into a reliable operating model, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to trust.

  • The Secrets Management Mistakes I See in AI Infrastructure

    The Secrets Management Mistakes I See in AI Infrastructure

    I have reviewed dozens of AI infrastructure stacks over the past year. The pattern is depressingly consistent: a team spins up Langfuse for LLM observability, adds n8n for workflow automation, plugs in Hermes or another AI agent — and within a month, there are database passwords committed to git and API keys baked directly into Docker Compose files.

    It is not that these teams do not care about security. It is that the tooling is easy to deploy and terrifyingly easy to deploy wrong. The secrets management layer is an afterthought, and by the time anyone notices, the rot is already baked into the repository history — often with several stale copies scattered across forks, CI logs, and deployment scripts.

    The tools are not the problem. The defaults are survivable if you change them immediately and manage them properly. The problem is that almost nobody does.

    Why this matters now

    AI infrastructure is different from a typical web app in one important respect: it holds the keys to your models, your data pipelines, and increasingly, your customer data. When a secrets leak happens in an AI stack, it is not just a credential rotation exercise. It can mean exposing vector stores full of proprietary documents, handing over API keys with uncapped billing, or losing control of an agent that has been given broad access to your internal systems.

    UK SMEs are adopting these tools faster than their security practices can keep up. Boards are asking for AI capability. Technical founders want to move fast. The result is that Langfuse and n8n instances go live with the same enthusiasm and the same rigour as a weekend side project.

    The NCSC has been clear that the shared responsibility model applies here. The platform provides the controls. You have to configure them. And right now, most teams deploying AI infrastructure are not.

    A war story from the field

    I recently reviewed a deployment where the Langfuse Docker Compose file had been committed to a version-controlled repository with production database credentials hardcoded directly into it. Not in an environment variable reference — the actual username and password, sitting in plaintext in a file that had been committed, reviewed, merged, and deployed.

    That was not the only issue. The same deployment had a Claude configuration that did not properly handle credential rotation after operator restarts. Every time the container restarted — patching, scaling, node migration — the service would silently fall back to insecure defaults. Nobody noticed for weeks because the service appeared to be running. It was running. It was just doing so with credentials that had long since been rotated and should no longer have been valid.

    I also found an n8n instance where the encryption key had not been persisted. The team had recreated their n8n container as part of a routine update, and the platform generated a new encryption key on startup. Every credential stored in n8n’s database became undecryptable. Every workflow that depended on stored API keys, database connections, or OAuth tokens broke simultaneously.

    The error message is admirably specific: “A different encryptionKey was used to encrypt the data.” But by the time you see it, all of your workflows are failing in production — and unless you have the original encryption key backed up somewhere outside the container, those credentials are gone.

    These are not edge cases. They are the most common findings in every AI infrastructure review I have done in the last twelve months.

    The five mistakes I see most often

    1. Hardcoded secrets in Docker Compose and environment files

    This is the big one. Teams copy a docker-compose.yml from a project README, fill in their passwords in plaintext, and commit it. Sometimes they remember to add .env to .gitignore but leave the Compose file itself exposed. Sometimes they move the secrets to an environment file but commit that too, because the .gitignore was only added after the first commit.

    2. Using default credentials past the first five minutes of setup

    Default credentials exist so you can get started quickly. They should exist in production for approximately zero seconds after the health check passes. I regularly find admin/admin or changeme on instances that have been running for months, sometimes years. If a tool ships with a default password, changing it should be the very first action in your runbook — not something you plan to do later.

    3. Losing the encryption key when containers are recreated

    This one catches people out constantly with n8n, but it applies to any platform that encrypts stored credentials. When you recreate a container without persisting the encryption keys, a new one is generated and the old data is orphaned. The platform cannot protect you from this. It is a configuration decision. You need to persist encryption keys outside the container lifecycle — in your secrets manager, in your CI/CD pipeline configuration, in a mounted volume that survives container recreation.

    4. Treating the AI stack as lower risk than the rest of the infrastructure

    There is a pernicious perception that the “AI tools” are supplementary, experimental, not worth the same rigour as the production database or the payment gateway. This is backwards. Your AI stack touches your most sensitive data — the documents you embed, the conversations you log, the internal APIs your agents call. It has the broadest external API surface in your architecture. It deserves more scrutiny, not less.

    5. Relying on git history alone to “remove” secrets

    Committing a secret, then removing it in a follow-up commit, does not remove it from git history. It just adds another commit on top. The secret is still there, reachable via git log -p, via GitHub’s commit history, via any clone or fork. Once a secret has been committed, the only safe response is rotation — treat it as compromised and issue new credentials.

    What to do next

    You do not need an enterprise secrets vault to get the basics right. You need a checklist and the discipline to follow it.

    1. Audit your repositories now. Search for common patterns: PASSWORD=, SECRET=, API_KEY=, base64-encoded blobs in environment files. Use tools like gitleaks or trufflehog to scan both current state and history. If anything turns up, rotate the credentials immediately — do not just remove them from the latest commit.

    2. Externalise every secret. In Docker Compose, reference ${VAR} values and use a .env file that is .gitignored at the repository root level. If you are on a managed platform, use its native secrets manager. No exceptions, no “just for now”, no “it is only a development environment.”

    3. Persist your n8n encryption key. Set the N8N_ENCRYPTION_KEY environment variable explicitly — do not let n8n auto-generate it. Store it in your secrets manager and make it part of your container orchestration configuration, not the container itself. Back it up. Test that you can restore it.

    4. Rotate after every environment change. When a container is recreated, when a team member leaves, when you are not even sure something was exposed — rotate. Treat the cost of rotation as negligible compared to the cost of a breach. Automate it wherever you can.

    5. Apply the same standard to AI tools as everything else. Your Langfuse instance, your n8n deployment, your AI agent platform — these hold credentials and data that would interest an attacker. Give them the same security treatment you would give a production database. No concessions for “it is internal” or “it is just a prototype.”

    Where Richard can help

    If you are deploying AI infrastructure and want an honest assessment of how your secrets are managed — before an incident forces the conversation — I offer infrastructure security reviews and fractional CISO engagements tailored to UK SMEs running containers, workflows, and AI tooling.

    The mistakes above are among the most common findings in the engagements I do. They are also almost always fixable in a day.

    Get in touch or review the available services to arrange a review.

  • GitHub Weekly — SecureScore Goes Live, Agentic Ops Mature, and the Atlas Foundation Takes Shape

    Introduction

    Seventy-four repos, one week, and a surprising amount of shipped work. This week’s activity spans three themes: production security tooling crossing from plan into live operation, agentic infrastructure maturing with better observability and failover, and a new initiative — the Atlas Foundation — taking shape around public-good AI proposals. Here’s what happened, what shipped, and what it signals.

    What Happened

    SecureScore: From Dry-Run to Live

    The Hermes SecureScore project crossed a meaningful threshold this week. After install dry-runs and collector CLI fixes over the previous days, the project recorded its live activation on June 19. The Docker collector for SecureScore evidence shipped, a config collector bug for string provider entries was fixed, and the evidence pipeline moved from testing to production. On the dashboard side, the main Hermes Agent repo merged PR #1 adding a SecureScore view, confirming this isn’t a standalone experiment but an integrated part of operations.

    Why it matters: security评分 that runs locally, on your own infra, without sending data externally — that’s a pattern more teams will need as AI agent deployments multiply and audit requirements tighten.

    Hermes Mgmt: Dashboard v0.17 and the Work Behind It

    The management dashboard saw a methodical series of doc-and-script updates. Highlights:

    • v0.17.0 migration guide published, including a process restart requirement — a small detail that saves a lot of “why isn’t this working?” confusion.
    • Langfuse cost analytics landed as a cross-provider spend report with a weekly digest cron. Teams running multi-model setups can now track spend across providers without spreadsheet gymnastics.
    • Agent Tool Audit Report 2026-06-24 opened as a PR, systematically cataloguing what the agent toolkit actually contains and where the gaps are.
    • Telegram flood protection (Layers 3+4+5) merged, covering restart hygiene, notification deduplication, and rate-limit handling.
    • Security vulnerability remediation — 16 CVEs patched across nltk and starlette dependencies.

    The pattern here isn’t glamorous features; it’s the unglamorous maintainability work that keeps agent infrastructure from rotting.

    Agentic Coordination: Failover, Cost Awareness, and Escalation

    Three PRs from the Hermes Mgmt repo tell a story about agentic ops moving past the demo phase:

    1. Local-primary triage and tiered escalation (PR #624) — an EPIC outlining how a local model (Hermes3:8b) handles first-tier requests and escalates to stronger models when responses are weak or empty. This is the failover tier the project has been building toward.
    2. Credit-awareness fix plan for OpenRouter — because nothing kills an agent workflow faster than hitting a model credit limit mid-task without warning.
    3. Agent Radar got its initial commit with a roadmap of MVP tasks: detector rules for common agent frameworks, repo status badges, and schema examples. The idea is to build a tool that scans repos and identifies agent frameworks automatically.

    Infrastructure: Hardening and Monitoring

    On the infrastructure side:

    • Hamnet shipped SSH hardening for the VPS with a dynamic-IP allowlist failsafe — the kind of defensive depth that matters when IPs change and locks you out. Vhost routing was also hardened after an incident where a domain served the wrong site.
    • Ollama dashboard and metrics pipeline (Hamnet PR #134) — because if you’re running models locally, you need to know the server is alive without checking manually.
    • A WordPress blog import runbook was added, documenting the pipeline that keeps the blog publishing workflow reproducible.

    Atlas Foundation: Public-Good Agent Proposals

    The project-atlas-foundation repo saw significant activity: candidate proposals scored, shaping docs created, and a safety checklist added. The current proposal slate includes:

    • An open-source issue triage assistant
    • A digital-access assistant for elderly users
    • A small charity automation kit
    • A public-good agent template library

    A Claude Code handover document was added for the team, covering setup, sync, and triage workflows — suggesting the project is moving from concept to collaborative execution.

    HamMediaLabs: Governance Deepening

    HamMediaLabs shipped a wave of internal governance: onboarding guides, a development guide, risk register, branch hygiene policy, and a PR review dashboard with dependency health reporting. For a small team, this is the scaffolding that prevents chaos as contributors scale up.

    Key Takeaways

    1. Production security is becoming operational, not aspirational. SecureScore’s live activation is a signal that local-first security scoring is viable for small teams without enterprise tooling budgets.

    2. The agent ops story is shifting from “can we build it?” to “can we keep it running?” Flood protection, cost dashboards, failover tiers, and tool audits — this is the maintenance phase of the agentic infrastructure lifecycle.

    3. Multi-model cost visibility is now a first-class concern. Langfuse cost analytics with cross-provider reporting, OpenRouter credit-awareness, and tiered escalation — these all reflect the reality that running multiple AI models costs real money and needs real monitoring.

    4. Governance-as-code is emerging in smaller projects. HamMediaLabs and Atlas Foundation both shipped policy documents as code: branch hygiene, risk registers, safety checklists. This is where the industry is heading — compliance documentation that lives in the repo, not in a SharePoint graveyard.

    Code Snippet: Dynamic SSH Allowlist

    From the Hamnet infrastructure work, a pattern worth showing — dynamically resolving an ISP’s current IP for an SSH allowlist, with a failsafe that doesn’t lock you out:

    #!/usr/bin/env bash
# update-ssh-allowlist.sh — refresh dynamic IP in allowlist
set -euo pipefail

CURRENT_IP=$(curl -s --max-time 5 https://ifconfig.me)
KNOWN_FALLBACK="203.0.113.0/24"  # static backup range

if [[ "$CURRENT_IP" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    iptables -D INPUT -p tcp --dport 22 -j DROP 2>/dev/null || true
    iptables -A INPUT -s "$CURRENT_IP/32" -p tcp --dport 22 -j ACCEPT
    echo "Updated: $CURRENT_IP/32"
else
    iptables -A INPUT -s "$KNOWN_FALLBACK" -p tcp --dport 22 -j ACCEPT
    echo "Fallback applied: $KNOWN_FALLBACK"
fi

    The key insight: always have a fallback. Dynamic DNS is reliable until the day your ISP changes your IP at 2 AM and you can’t SSH in to fix it.

    Data source: commits, issues, and PRs from 49 repositories over the past 7 days. Collected via the GitHub CLI on 2026-06-25.

  • Hardening Without Drama

    Hardening Without Drama

    This was one of those weeks where the work that mattered most was the work nobody would notice if it went right. That usually means it is important.

    What changed

    • The security posture got tighter at the edges. A system that handles real work needs controls that survive when the environment changes.

    • Observability was also pushed a little closer to the truth. Good metrics do not flatter the operator; they tell you what is actually happening.

    • Several changes pointed to the same idea: if a process can fail, it should have a clear fallback and a clear owner.

    Closing thought

    The useful version of hardening is calm. No theatre, no chest-beating, just fewer weak spots and better recovery when something does break.

  • GitHub Weekly — SecureScore Goes Live, Agentic Ops Mature, and the Atlas Foundation Takes Shape

    Introduction

    Seventy-four repos, one week, and a surprising amount of shipped work. This week’s activity spans three themes: production security tooling crossing from plan into live operation, agentic infrastructure maturing with better observability and failover, and a new initiative — the Atlas Foundation — taking shape around public-good AI proposals. Here’s what happened, what shipped, and what it signals.

    What Happened

    SecureScore: From Dry-Run to Live

    The Hermes SecureScore project crossed a meaningful threshold this week. After install dry-runs and collector CLI fixes over the previous days, the project recorded its live activation on June 19. The Docker collector for SecureScore evidence shipped, a config collector bug for string provider entries was fixed, and the evidence pipeline moved from testing to production. On the dashboard side, the main Hermes Agent repo merged PR #1 adding a SecureScore view, confirming this isn’t a standalone experiment but an integrated part of operations.

    Why it matters: security评分 that runs locally, on your own infra, without sending data externally — that’s a pattern more teams will need as AI agent deployments multiply and audit requirements tighten.

    Hermes Mgmt: Dashboard v0.17 and the Work Behind It

    The management dashboard saw a methodical series of doc-and-script updates. Highlights:

    • v0.17.0 migration guide published, including a process restart requirement — a small detail that saves a lot of “why isn’t this working?” confusion.
    • Langfuse cost analytics landed as a cross-provider spend report with a weekly digest cron. Teams running multi-model setups can now track spend across providers without spreadsheet gymnastics.
    • Agent Tool Audit Report 2026-06-24 opened as a PR, systematically cataloguing what the agent toolkit actually contains and where the gaps are.
    • Telegram flood protection (Layers 3+4+5) merged, covering restart hygiene, notification deduplication, and rate-limit handling.
    • Security vulnerability remediation — 16 CVEs patched across nltk and starlette dependencies.

    The pattern here isn’t glamorous features; it’s the unglamorous maintainability work that keeps agent infrastructure from rotting.

    Agentic Coordination: Failover, Cost Awareness, and Escalation

    Three PRs from the Hermes Mgmt repo tell a story about agentic ops moving past the demo phase:

    1. Local-primary triage and tiered escalation (PR #624) — an EPIC outlining how a local model (Hermes3:8b) handles first-tier requests and escalates to stronger models when responses are weak or empty. This is the failover tier the project has been building toward.
    2. Credit-awareness fix plan for OpenRouter — because nothing kills an agent workflow faster than hitting a model credit limit mid-task without warning.
    3. Agent Radar got its initial commit with a roadmap of MVP tasks: detector rules for common agent frameworks, repo status badges, and schema examples. The idea is to build a tool that scans repos and identifies agent frameworks automatically.

    Infrastructure: Hardening and Monitoring

    On the infrastructure side:

    • Hamnet shipped SSH hardening for the VPS with a dynamic-IP allowlist failsafe — the kind of defensive depth that matters when IPs change and locks you out. Vhost routing was also hardened after an incident where a domain served the wrong site.
    • Ollama dashboard and metrics pipeline (Hamnet PR #134) — because if you’re running models locally, you need to know the server is alive without checking manually.
    • A WordPress blog import runbook was added, documenting the pipeline that keeps the blog publishing workflow reproducible.

    Atlas Foundation: Public-Good Agent Proposals

    The project-atlas-foundation repo saw significant activity: candidate proposals scored, shaping docs created, and a safety checklist added. The current proposal slate includes:

    • An open-source issue triage assistant
    • A digital-access assistant for elderly users
    • A small charity automation kit
    • A public-good agent template library

    A Claude Code handover document was added for the team, covering setup, sync, and triage workflows — suggesting the project is moving from concept to collaborative execution.

    HamMediaLabs: Governance Deepening

    HamMediaLabs shipped a wave of internal governance: onboarding guides, a development guide, risk register, branch hygiene policy, and a PR review dashboard with dependency health reporting. For a small team, this is the scaffolding that prevents chaos as contributors scale up.

    Key Takeaways

    1. Production security is becoming operational, not aspirational. SecureScore’s live activation is a signal that local-first security scoring is viable for small teams without enterprise tooling budgets.

    2. The agent ops story is shifting from “can we build it?” to “can we keep it running?” Flood protection, cost dashboards, failover tiers, and tool audits — this is the maintenance phase of the agentic infrastructure lifecycle.

    3. Multi-model cost visibility is now a first-class concern. Langfuse cost analytics with cross-provider reporting, OpenRouter credit-awareness, and tiered escalation — these all reflect the reality that running multiple AI models costs real money and needs real monitoring.

    4. Governance-as-code is emerging in smaller projects. HamMediaLabs and Atlas Foundation both shipped policy documents as code: branch hygiene, risk registers, safety checklists. This is where the industry is heading — compliance documentation that lives in the repo, not in a SharePoint graveyard.

    Code Snippet: Dynamic SSH Allowlist

    From the Hamnet infrastructure work, a pattern worth showing — dynamically resolving an ISP’s current IP for an SSH allowlist, with a failsafe that doesn’t lock you out:

    #!/usr/bin/env bash
# update-ssh-allowlist.sh — refresh dynamic IP in allowlist
set -euo pipefail

CURRENT_IP=$(curl -s --max-time 5 https://ifconfig.me)
KNOWN_FALLBACK="203.0.113.0/24"  # static backup range

if [[ "$CURRENT_IP" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    iptables -D INPUT -p tcp --dport 22 -j DROP 2>/dev/null || true
    iptables -A INPUT -s "$CURRENT_IP/32" -p tcp --dport 22 -j ACCEPT
    echo "Updated: $CURRENT_IP/32"
else
    iptables -A INPUT -s "$KNOWN_FALLBACK" -p tcp --dport 22 -j ACCEPT
    echo "Fallback applied: $KNOWN_FALLBACK"
fi

    The key insight: always have a fallback. Dynamic DNS is reliable until the day your ISP changes your IP at 2 AM and you can’t SSH in to fix it.

    Data source: commits, issues, and PRs from 49 repositories over the past 7 days. Collected via the GitHub CLI on 2026-06-25.

Powered by CookieAdmin