RichardHam.co.uk Blog

Tag: DevOps

  • GitHub Weekly — Security Hardening, Governance Cleanup, and Website Delivery

    GitHub Weekly — Security Hardening, Governance Cleanup, and Website Delivery

    This week’s GitHub activity had a very familiar shape: a lot of small changes, but a very clear direction.

    When I pulled the last seven days together, I ended up with 250 events across 64 repositories. The detail varied, but the pattern was hard to miss. The most useful work clustered around three themes: making systems safer to operate, making decisions easier to trace, and moving real work out of drafts and into something usable.

    That combination matters. It is easy to celebrate a feature release and overlook the quieter work that makes the next release easier, safer, and less dependent on memory. This week was mostly about exactly that.

    What happened

    1. The hardening work kept moving

    The busiest thread was around the Hermes management side of the stack. There were issues and pull requests covering gateway restart behaviour, profile-scoped tools and memory surfaces, local Vault health, OAuth token expiry, dependency CVEs, and a few routing and performance concerns.

    That is the kind of activity I like to see in a living system, even when it is uncomfortable. The work was not cosmetic. It was focused on failure modes:

    • what happens when the secrets backend is sealed
    • what happens when a token expires unexpectedly
    • what happens when dependency drift creates exposure in the runtime
    • what happens when routing logic gets too expensive to keep running blindly
    • what happens when multiple operators need clean boundaries around tools and state

    In other words, the week was spent asking the right questions before the answers became incidents.

    There was also a useful operational thread around keeping the system honest: adding safety nets, reviewing carry-forward behaviour, and tightening the control plane around restarts and state. That kind of work rarely looks dramatic in a changelog, but it is often where the real reliability gains come from.

    2. Governance became more concrete

    A second pattern showed up in the roadmap and control repositories. There were commits and pull requests for provisioning scripts, README links, alert-rule tuning, and monitoring documentation.

    That matters because strategy is only useful when it can be executed repeatedly.

    A plan is just a plan until it has:

    • a provisioning path
    • a documented handover
    • a clear monitoring expectation
    • a change history that another person can follow

    The activity this week pushed in that direction. Roadmap work became more operational. Monitoring guidance became more explicit. Decision-making became easier to track. Even the recurring “decision desk” style updates are useful in that sense: they turn vague progress into a traceable record.

    That is particularly important in AI and automation work, where teams can move quickly but still leave behind unclear assumptions. The more complex the stack gets, the more valuable it becomes to treat governance as part of the delivery process rather than as a separate admin task.

    3. Website work moved from intention to delivery

    The third thread was more visible: website work.

    There was a full redesign path on one service site, including layout work, deploy tooling, and a clear move from mockups to implemented pages. On the content side, the blog workflow itself also kept moving, with a new weekly roundup drafted and the editorial queue updated.

    I think this is an underrated signal. People often talk about code, but delivery includes the path around the code too:

    • the build steps
    • the draft content
    • the publication workflow
    • the editorial queue
    • the handoff between “done locally” and “live somewhere useful”

    If those pieces are weak, the site may look finished while still being awkward to maintain. If they are strong, the site becomes easier to update, easier to trust, and easier to keep current.

    That same lesson shows up in content systems, operations tooling, and AI workflows. The system is only as strong as the path from intent to output.

    Key takeaways

    A week like this leaves a few practical lessons.

    • Security debt is easiest to fix before it becomes visible. The moment a sealed Vault, an expired token, or a dependency CVE shows up in a weekly review is the moment to deal with it.
    • Operational boundaries matter more as systems grow. Profile-scoped tools, explicit memory surfaces, and predictable restart behaviour are all examples of the same idea: reduce ambiguity.
    • Roadmaps need executable steps, not just aspirations. Provisioning scripts, alert-rule updates, and readable docs make a roadmap real.
    • Delivery includes the publishing path. A draft that never reaches the right place is only half a result.
    • The best week-over-week improvement is often cumulative, not flashy. Small fixes across security, governance, and publishing add up to a stronger operating model.

    A practical standard

    If I had to compress the week into one rule, it would be this:

    • if it can fail, define the failure mode
    • if it repeats, make it traceable
    • if it matters, write it down
    • if it ships, make the path to shipping reliable

    That is a good standard for AI work, but it is just as useful for infrastructure, websites, and internal operations. The details change. The principle does not.

    The goal is not to eliminate all uncertainty. The goal is to make the important parts of the system clear enough that they can be operated without guesswork.

    Closing thought

    The most useful work this week was not a single large feature. It was the accumulation of smaller changes that make a system easier to run: better failure handling, clearer governance, more repeatable delivery, and a stronger publishing path.

    That is usually where durable progress lives.

    If you are working through a similar mix of security, automation, and delivery problems, the AI & Automation Architecture service is a good starting point. Or get in touch if you want to talk through the shape of the system before it becomes the problem.

  • The Hidden Cost Structure of Multi-Model AI

    The Hidden Cost Structure of Multi-Model AI

    You set a budget. You picked a model. You shipped the feature. Then the bill arrived.

    If you are running AI at scale across a product or platform, you have probably noticed that costs do not behave the way you expected. The per-token price looked reasonable in the dashboard. The monthly total did not.

    The problem is not that AI is expensive. The problem is that the cost structure is hidden.

    Why this matters now

    The shift from single-model to multi-model architectures is well underway. Most serious AI-powered products now route between several providers: a frontier model for complex reasoning, a cheaper one for classification and extraction, maybe a local model for high-volume, low-judgement work.

    That architecture is the right move. But it introduces a cost surface that almost nobody tracks properly. Different models have wildly different pricing. Some tasks genuinely need GPT-4o or Claude Sonnet. Most do not. And without visibility, you end up paying frontier prices for commodity work.

    The organisations that figure this out early will have a structural advantage. The ones that do not will burn budget and wonder why.

    What is really going on

    There are four hidden cost drivers in most multi-model setups. They compound each other.

    1. Token waste

    Every call to an LLM carries context: system prompts, conversation history, tool definitions, retrieved documents. In a multi-agent or multi-step workflow, that context grows at each hop. It is common to see workflows where 70 to 80 percent of the tokens sent are redundant or repeated across steps.

    At scale, that waste is not trivial. A workflow running 10,000 times a day with an average of 4,000 input tokens per call, where half of those tokens are unnecessary, can easily cost hundreds of pounds per month in tokens that contributed nothing to the output.

    2. Model mismatch

    This is the single biggest lever. Using Claude Sonnet for a task that Llama 3.2 handles just as well is a 10x cost difference for no quality gain. Using GPT-4o to classify support tickets is not just expensive, it is architecturally unnecessary.

    The pattern I see repeatedly: teams default to the most capable model available, then never revisit that decision once the feature is in production. The model that was right for prototyping is not necessarily right for day 10,000 of production.

    3. Lack of visibility

    Most AI billing is provider-level. You can see what you spent on OpenAI versus Anthropic versus a smaller provider. What you cannot see, without deliberate instrumentation, is the cost per workflow, per agent, per feature, or per team.

    That means nobody can answer basic questions. Which feature is the most expensive to run? Which team’s AI usage is growing fastest? Which workflow has the worst token-to-value ratio? Without per-workflow cost tracking, you are managing AI spend with a monthly credit card statement.

    4. API sprawl

    When multiple teams adopt AI independently, you end up with overlapping credentials, redundant integrations, and no centralised billing. One team uses OpenAI directly. Another wraps it in a different SDK. A third spins up a separate provider for a narrow use case. Each team optimises locally. Nobody optimises globally.

    This is not a technology problem. It is a governance gap, and it is expensive.

    What to do next

    The good news is that the solution pattern is well understood and does not require ripping out what you have already built.

    1. Route cheap tasks to free or local models.

    If a task does not require frontier reasoning, run it locally. Tools like Ollama make this practical for classification, extraction, summarisation, and other high-volume, low-judgement work. The cost difference is not marginal. It is the difference between paying per token and paying nothing beyond your compute.

    2. Reserve expensive models for high-judgement work.

    Define clear criteria for when a task warrants a frontier model. Complex reasoning, nuanced language generation, multi-step planning, and safety-critical decisions qualify. Everything else should default to the cheapest model that meets your quality bar.

    3. Track costs per workflow, not just per provider.

    Instrument your routing layer to log token usage and cost at the workflow or agent level. This does not require a complex platform. Even a lightweight middleware that tags each request with a workflow identifier and records input/output token counts will give you the visibility you need.

    4. Set budget alerts before you need them.

    Per-workflow budgets with hard and soft alerts let you catch runaway costs before they become line items in a board report. A soft alert at 70 percent of budget gives you time to investigate. A hard cap at 100 percent protects you from the unexpected.

    5. Audit your context.

    Review the prompts and context you are sending at each step of every workflow. Remove redundant system prompts, collapse repeated context, and set maximum token limits. This is the fastest way to reduce waste without changing any model choices.

    6. Centralise provider management.

    One set of credentials. One billing view. One team responsible for provider relationships and rate negotiations. This is not about slowing teams down. It is about making sure the organisation can see and manage what it is spending.

    Where Richard can help

    I work with technical leaders who are building AI-powered products and need the cost structure to be as well-designed as the architecture. That means routing strategies, cost tracking, and practical governance that does not get in the way of delivery.

    If multi-model cost management is on your roadmap, get in touch or review the relevant services.

  • What Real Delivery Discipline Looks Like

    What Real Delivery Discipline Looks Like

    A lot of teams say they care about security and reliability. Fewer teams build delivery habits that prove it.

    That difference matters more than most strategy decks admit. Hardening work is easy to describe in principle. Everyone agrees that authentication should be stronger, fallbacks should be safer, and monitoring should be more truthful. The real question is whether those concerns are allowed to shape the delivery path itself.

    If they are not, the system ends up with good intentions and brittle behaviour.

    Hardening is not a final polish step

    One of the most persistent delivery mistakes is treating hardening as something that happens after the “real” build is done.

    In practice, the opposite is true. Security-sensitive choices usually need to be made while the implementation is still taking shape:

    • what credentials are allowed to reach which component
    • what action is taken when verification fails
    • whether a degraded path is acceptable or whether the job should stop
    • what evidence an operator needs before calling something healthy
    • how much trust the system gives to generated output by default

    Those are not edge details. They shape the behaviour of the whole stack.

    If the hardening conversation starts only once the workflow is already live, the team ends up retrofitting controls around assumptions that were never designed for scrutiny.

    Monitoring has to be able to disagree with the diagram

    A good architecture diagram can still hide a weak operating model.

    I see this most often in monitoring. Teams build checks that confirm whether the process is running, but not whether the real task is succeeding in a meaningful way. That produces the worst kind of comfort: the dashboard looks clean while the system is quietly failing at the thing that matters.

    Useful monitoring needs to be willing to contradict appearances.

    For example, I would rather know that:

    • the workflow completed but returned structurally bad output
    • the service stayed up but fell onto the wrong provider path
    • the API responded but the downstream data plane was empty
    • the fallback recovered execution but created a lower-trust result

    That kind of signal is less glamorous than a green uptime chart, but it is far more useful when somebody has to operate the platform at speed.

    Delivery discipline is mostly about reducing hidden surprises

    The phrase “delivery discipline” can sound heavier than it needs to. I do not mean process theatre. I mean building a path to change that does not rely on luck.

    In practical terms, the teams that do this well tend to share a few habits:

    They make verification part of the job

    A change is not finished because the command returned zero. It is finished when the operator can show that the intended behaviour is visible from the outside.

    They do not improvise every rollback

    If a deployment path matters, the recovery path should already exist before the incident.

    They keep release logic legible

    The more a release process depends on one person remembering unwritten exceptions, the less mature it is.

    They distinguish activity from progress

    A lot of work can happen in a sprint without improving the trustworthiness of the system. Good delivery discipline asks whether the change made the operating model clearer, safer, or easier to verify.

    Why this matters in AI and automation work

    Automation makes weak delivery habits more expensive because the system can now repeat them at scale.

    If a manual operator makes one bad judgement call, the damage is limited. If the workflow itself contains vague trust boundaries, shallow health checks, or unclear fallback rules, the same weakness can repeat every minute.

    That is why I treat discipline as a feature.

    Not in the moral sense. In the architectural sense.

    A disciplined delivery path gives the team:

    • safer defaults
    • clearer escalation points
    • more credible monitoring
    • easier handover between operators
    • fewer silent regressions

    That is not bureaucracy. It is what makes the rest of the system believable.

    What I would check first

    If I am assessing whether a delivery process is genuinely disciplined, I usually start with a small checklist:

    • are the trust boundaries obvious?
    • does the monitoring test real behaviour or just process existence?
    • can somebody explain the fallback path without guessing?
    • is there a documented way to verify the live result after a change?
    • would another operator know what “healthy” means from the artefacts alone?

    If the answer to those questions is mostly “not yet”, then the team probably does not have a tooling problem. It has a delivery-discipline problem.

    The good news is that this is fixable. Most of the gains come from clearer defaults, sharper verification, and a willingness to treat operational trust as part of the design instead of a layer you add afterward.

    That work is rarely flashy. It just means the system behaves like something serious.

    If your stack needs stronger delivery discipline around hardening, observability, or AI workflows, the AI & Automation Architecture and Security Strategy work is built around exactly those problems. Or get in touch if you want a practical review of where the delivery path is still carrying too much guesswork.

  • GitHub Weekly — Security Hardening, Governance Cleanup, and Website Delivery

    GitHub Weekly — Security Hardening, Governance Cleanup, and Website Delivery

    This week’s GitHub activity had a very familiar shape: a lot of small changes, but a very clear direction.

    When I pulled the last seven days together, I ended up with 250 events across 64 repositories. The detail varied, but the pattern was hard to miss. The most useful work clustered around three themes: making systems safer to operate, making decisions easier to trace, and moving real work out of drafts and into something usable.

    That combination matters. It is easy to celebrate a feature release and overlook the quieter work that makes the next release easier, safer, and less dependent on memory. This week was mostly about exactly that.

    What happened

    1. The hardening work kept moving

    The busiest thread was around the Hermes management side of the stack. There were issues and pull requests covering gateway restart behaviour, profile-scoped tools and memory surfaces, local Vault health, OAuth token expiry, dependency CVEs, and a few routing and performance concerns.

    That is the kind of activity I like to see in a living system, even when it is uncomfortable. The work was not cosmetic. It was focused on failure modes:

    • what happens when the secrets backend is sealed
    • what happens when a token expires unexpectedly
    • what happens when dependency drift creates exposure in the runtime
    • what happens when routing logic gets too expensive to keep running blindly
    • what happens when multiple operators need clean boundaries around tools and state

    In other words, the week was spent asking the right questions before the answers became incidents.

    There was also a useful operational thread around keeping the system honest: adding safety nets, reviewing carry-forward behaviour, and tightening the control plane around restarts and state. That kind of work rarely looks dramatic in a changelog, but it is often where the real reliability gains come from.

    2. Governance became more concrete

    A second pattern showed up in the roadmap and control repositories. There were commits and pull requests for provisioning scripts, README links, alert-rule tuning, and monitoring documentation.

    That matters because strategy is only useful when it can be executed repeatedly.

    A plan is just a plan until it has:

    • a provisioning path
    • a documented handover
    • a clear monitoring expectation
    • a change history that another person can follow

    The activity this week pushed in that direction. Roadmap work became more operational. Monitoring guidance became more explicit. Decision-making became easier to track. Even the recurring “decision desk” style updates are useful in that sense: they turn vague progress into a traceable record.

    That is particularly important in AI and automation work, where teams can move quickly but still leave behind unclear assumptions. The more complex the stack gets, the more valuable it becomes to treat governance as part of the delivery process rather than as a separate admin task.

    3. Website work moved from intention to delivery

    The third thread was more visible: website work.

    There was a full redesign path on one service site, including layout work, deploy tooling, and a clear move from mockups to implemented pages. On the content side, the blog workflow itself also kept moving, with a new weekly roundup drafted and the editorial queue updated.

    I think this is an underrated signal. People often talk about code, but delivery includes the path around the code too:

    • the build steps
    • the draft content
    • the publication workflow
    • the editorial queue
    • the handoff between “done locally” and “live somewhere useful”

    If those pieces are weak, the site may look finished while still being awkward to maintain. If they are strong, the site becomes easier to update, easier to trust, and easier to keep current.

    That same lesson shows up in content systems, operations tooling, and AI workflows. The system is only as strong as the path from intent to output.

    Key takeaways

    A week like this leaves a few practical lessons.

    • Security debt is easiest to fix before it becomes visible. The moment a sealed Vault, an expired token, or a dependency CVE shows up in a weekly review is the moment to deal with it.
    • Operational boundaries matter more as systems grow. Profile-scoped tools, explicit memory surfaces, and predictable restart behaviour are all examples of the same idea: reduce ambiguity.
    • Roadmaps need executable steps, not just aspirations. Provisioning scripts, alert-rule updates, and readable docs make a roadmap real.
    • Delivery includes the publishing path. A draft that never reaches the right place is only half a result.
    • The best week-over-week improvement is often cumulative, not flashy. Small fixes across security, governance, and publishing add up to a stronger operating model.

    A practical standard

    If I had to compress the week into one rule, it would be this:

    • if it can fail, define the failure mode
    • if it repeats, make it traceable
    • if it matters, write it down
    • if it ships, make the path to shipping reliable

    That is a good standard for AI work, but it is just as useful for infrastructure, websites, and internal operations. The details change. The principle does not.

    The goal is not to eliminate all uncertainty. The goal is to make the important parts of the system clear enough that they can be operated without guesswork.

    Closing thought

    The most useful work this week was not a single large feature. It was the accumulation of smaller changes that make a system easier to run: better failure handling, clearer governance, more repeatable delivery, and a stronger publishing path.

    That is usually where durable progress lives.

    If you are working through a similar mix of security, automation, and delivery problems, the AI & Automation Architecture service is a good starting point. Or get in touch if you want to talk through the shape of the system before it becomes the problem.

  • Release Readiness Starts Small

    Release readiness is one of those phrases people tend to associate with major milestones. A big launch. A freeze window. A final pre-production review. In reality, most release readiness work starts much earlier and looks much less dramatic.

    It usually begins with the boring choices teams make while nothing appears to be on fire.

    That is when dependency drift gets handled before it turns into surprise breakage. That is when decision records are written before the rationale disappears. That is when logs, checklists, and ownership lines are tightened while there is still time to do it calmly.

    By the time a release feels stressful, the underlying maintenance decisions have usually already been made.

    Drift is rarely dangerous all at once

    Dependency drift does not normally announce itself as a crisis. It accumulates.

    One package lags a little behind. A toolchain bump gets deferred because it is inconvenient. A warning sits in CI because it is noisy rather than urgent. A config difference between environments becomes accepted because nobody wants to touch it this week.

    Individually, each decision can feel reasonable.

    Collectively, they change the character of the next release.

    Suddenly the team is not just shipping a feature. It is shipping a feature while also discovering which old assumptions have quietly expired.

    That is why I think release readiness starts with maintenance discipline rather than ceremony. If the stack is allowed to drift without a conscious limit, the release process becomes a gamble disguised as a plan.

    Governance makes releases cheaper

    People sometimes hear the word “governance” and assume it means slowing things down. Bad governance does. Useful governance does the opposite.

    Useful governance reduces rediscovery.

    A lightweight decision record, a clear runbook note, or a short changelog entry can save a team from re-litigating the same question under deadline pressure. It is much easier to release calmly when the important context is already captured.

    That does not mean documenting everything. It means documenting the parts that would otherwise have to be guessed later:

    • why a dependency was pinned or deferred
    • which path is considered the supported one
    • what the fallback looks like if the preferred route fails
    • which checks count as real release verification
    • who owns the decision when the evidence is mixed

    Without those anchors, “release readiness” often becomes a frantic search for institutional memory.

    Verification needs to reflect live behaviour

    Another trap is treating pre-release verification as a box-ticking exercise.

    A test passing in isolation is useful, but it does not prove that the live behaviour matches the intent of the release. A service can be technically up while still being operationally wrong. A workflow can complete while routing work through the wrong path. A published change can deploy cleanly while still leaving the user-facing result inconsistent.

    That is why the best release checks are usually the least theatrical ones. They ask plain questions:

    • does the thing work from the outside?
    • did it use the expected route?
    • do the logs and outputs make sense together?
    • would another operator understand what happened from the artefacts alone?
    • if rollback is needed, is the sequence already known?

    Those checks do not make the process glamorous, but they make it credible.

    Small maintenance work changes the feel of a release

    You can often tell how ready a team really is by how the release conversation sounds.

    When the groundwork has been done, the language is calm. People are checking, confirming, and verifying.

    When it has not, the language becomes speculative:

    • “I think this should still be compatible.”
    • “We can probably fix that after deploy.”
    • “I’m not sure which version is on the live path.”
    • “That alert is usually harmless.”
    • “Let’s ship it and see.”

    That is not a release strategy. It is unresolved maintenance debt surfacing at the worst moment.

    The teams that avoid that pattern are not necessarily the most resourced. They are usually the ones that kept chipping away at the small corrections before the release window forced urgency onto everything.

    What I would tighten first

    If I wanted to improve release readiness without adding unnecessary process, I would start with a short list:

    1. Reduce unmanaged drift

    Know which dependencies, config deltas, and environment differences are tolerated and which ones are not.

    2. Write down the decisions that matter

    Especially the ones that affect supported paths, rollback logic, or verification expectations.

    3. Test the live outcome, not just the local command

    A clean build is not the same thing as a trustworthy release.

    4. Keep the rollback path boring

    If recovery depends on improvisation, the release is not actually ready.

    5. Make ownership obvious

    If something looks ambiguous during release, someone should know who decides.

    Readiness is a maintenance habit

    That is the real point. Release readiness is less a milestone than a maintenance habit.

    It comes from keeping the stack current enough to trust, the decisions visible enough to follow, and the verification honest enough to mean something. Teams that do that consistently make releases feel uneventful in the best possible way.

    And that is usually the goal. Not excitement. Predictability.

    If you are trying to make releases calmer by improving the operating model underneath them, the AI & Automation Architecture work covers exactly that kind of practical governance and delivery design. Or get in touch if you want help identifying where drift, ambiguity, or weak verification is making your next release harder than it needs to be.

  • What Hardening a Production Server Actually Looks Like

    What Hardening a Production Server Actually Looks Like

    There’s a version of “server hardening” that exists in compliance documents: a tidy checklist, a one-time audit, a box ticked. Then there’s what it actually looks like in production — messy, iterative, and never quite finished.

    I’ve spent the last several weeks hardening infrastructure across multiple environments, and the pattern is always the same. What was secure at deployment drifts. Defaults get forgotten. Services get bolted on. And suddenly the thing you trusted is quietly doing something you never intended.

    Here’s what the real work looks like.

    The Problem Nobody Warns You About: Configuration Drift

    You deploy a server with a locked-down SSH config, a clean firewall, and sensible defaults. Three months later, someone adds a web server, opens a port for debugging, and forgets to close it. Or an Nginx vhost gets added without an explicit server_name, and the default catch-all starts serving the wrong site to the wrong visitors.

    This isn’t hypothetical. I recently found a production edge server where the Nginx default_server block was silently intercepting requests meant for a different virtual host. The site was working — but it was serving the wrong content to a subset of visitors. No errors in the logs. No alerts. Just quiet, invisible misconfiguration that had been running who knows how long.

    The fix was straightforward: explicit server_name directives on every vhost, and removing the catch-all entirely. But finding it required actually looking, which is the part that doesn’t happen often enough.

    SSH Access With a Dynamic IP: The Failsafe Pattern

    Here’s another one that comes up constantly. You lock SSH down to specific source IPs — best practice, absolutely. But your home IP is assigned by your ISP via DHCP, and it changes. Now you’re locked out of your own server, or you’re tempted to leave the firewall wide open “just until I update it.”

    The solution I implemented on a production VPS was an automated failsafe script. It runs periodically, detects the current public IP, and updates the firewall allowlist if the IP has changed. The key design principles:

    • The script authenticates outbound — it calls a known endpoint to discover the current IP, then pushes the update.
    • It only modifies the specific allowlist rule — it doesn’t touch any other firewall configuration.
    • It logs every change — so there’s an audit trail of when and why the IP was updated.
    • It fails closed — if the script can’t determine the current IP, it doesn’t open anything up.

    This pattern means you get the security of IP-restricted SSH without the operational risk of locking yourself out when your ISP rotates your address. It’s not fancy. It’s just honest about the reality that infrastructure has to be operable by humans.

    Multi-Hop SSH: Key Management Across Trust Boundaries

    The more complex the infrastructure, the more carefully you need to think about SSH key propagation. In one environment, the access path runs through multiple hops: local machine to management host, management host to container runtime, container runtime to VPS.

    Each hop is a trust boundary. The question at each one is: what keys exist here, who can use them, and what happens if this host is compromised?

    The hardening approach:

    • Separate keys per hop — no single key traverses the entire chain. If one host is compromised, the blast radius is limited to the next hop, not the entire path.
    • Keys are never copied manually — they’re provisioned through automation, with expiry and rotation baked in.
    • Agent forwarding is scoped, not blanket — it’s enabled only for specific connections and disabled by default.
    • Every key has a known owner and purpose — if you can’t explain why a key exists, it gets removed.

    This is the kind of thing that feels excessive until the day it prevents a lateral move during an incident. Then it feels like the most important work you did.

    The Hardening Checklist You Can Actually Use

    If you’re responsible for production infrastructure — whether you’re a CTO, a technical founder, or the person who just ended up owning the servers — here’s a practical checklist drawn from real hardening work:

    1. Audit your firewall rules quarterly. Every rule should have a comment explaining why it exists. If you can’t explain it, remove it.
    2. Remove default_server catch-alls from web servers. Every vhost should have an explicit server_name. If a request doesn’t match a known vhost, it should get a 444 or a meaningful error — not silently served by the wrong site.
    3. Automate your SSH access management. If your source IP can change, build the failsafe before you get locked out, not after.
    4. Segment your SSH keys by trust boundary. One key per hop. No exceptions for convenience.
    5. Review listening services monthly. Run ss -tlnp and verify every open port is intentional. You will find surprises.
    6. Check for configuration drift after every change. The best time to catch a misconfiguration is right after someone made a different change nearby.
    7. Log access and changes. If you can’t tell who connected, when, and what they changed, you don’t have a hardened server — you have a shared secret.

    The Uncomfortable Truth

    Hardening isn’t a project with a finish line. It’s a discipline. The servers that get compromised aren’t usually the ones that were never hardened — they’re the ones that were hardened once and then left to drift.

    The work is unglamorous. It’s reviewing firewall rules on a Friday afternoon. It’s removing a default_server block that “seems to be working.” It’s writing a small script so your SSH access doesn’t break when your ISP does its thing.

    But this is the work that keeps production infrastructure trustworthy. And it’s the work that separates infrastructure that looks secure from infrastructure that is secure.

    If this is on your roadmap — or if you’d rather someone else owned it — get in touch or review the relevant services.

  • GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    Introduction

    Some weeks are about shipping something obvious. Other weeks are about making the systems around the work easier to trust.

    This week looked like the second kind of week. Across the repos I reviewed, the change was direction, not just volume. Model routing became more deliberate. Monitoring became more accurate. Reusable delivery patterns became more explicit. Creative systems got clearer guardrails. And the small operational details that usually sit in the background were pulled into the foreground where they belong.

    That matters because the hardest part of running AI-assisted or automation-heavy systems is rarely the first prototype. The hard part is the handoff from “this works on my machine” to “this can be operated, explained, and improved without guesswork.” This week’s activity was a good reminder that trustworthy systems are built from a lot of very ordinary decisions made consistently.

    What happened

    Model routing stopped being an assumption and became a policy

    The clearest example came from hermes-mgmt. There was a real shift in how model choice is being handled: routing policy updates were merged, a stable default was pinned down, and provider-diverse fallback and outage behaviour were documented more explicitly. On top of that, role-based routing for research, coding, and review was defined instead of leaving model selection to whatever happened to be convenient in the moment.

    A lot of teams start with a simple belief: pick the biggest or newest model and let it do everything. That works until it doesn’t. Once you begin relying on models for different kinds of work, the distinction between “best model” and “best model for this job” becomes the difference between a healthy control plane and an expensive guessing game.

    I like this change because it turns model selection into an operating decision. It acknowledges that research work, coding work, and review work are not the same thing. It also gives the system a way to recover when a provider has issues, rather than forcing every incident into a manual exception.

    The surrounding work supports that direction too. A stale current-state snapshot was refreshed, the wider documentation was merged, and an earlier activation runbook for mixture-of-agents work shows that the system is being treated as something that needs operational memory, not just clever prompts.

    Observability was corrected, not merely expanded

    The biggest lesson from hamnet was the same one I have seen many times in production environments: a dashboard is only useful if it is telling the truth.

    This week’s work there focused on dashboard reliability rather than cosmetic additions. Grafana datasource timing was corrected, time series queries were given explicit intervals, default time windows were adjusted, and Pushgateway queries were wrapped so they render consistently. There was also a practical backlog item to track services that still need HTTPS equivalents, which is the kind of operational debt that gets forgotten unless somebody writes it down.

    I think this is a good example of how observability should be approached. It is tempting to treat monitoring as a collection of panels: add some graphs, colour some thresholds, and call it “visibility.” But real observability is more demanding. It means the query window is right, the step size is right, the defaults are right, and the chart is actually showing the thing you think it is showing.

    That is not a minor distinction. If your dashboard is technically live but operationally misleading, it can create more risk than it removes.

    There was also a useful signal in the issues around connecting a second Hermes instance as a remote subagent. That kind of work says the control plane is no longer just about one local setup. It is beginning to think in terms of distributed operation, where trust, connectivity, and monitoring all need to survive outside a single happy path.

    Creative systems became more governable

    HamMediaLabs showed a similar pattern, but from a different angle. The work there focused on creative-quality instrumentation, creative-ops playbooks, a per-brand visual identity layer, and a short-form voice system.

    That sounds very different from routing policies and dashboard fixes, but the underlying principle is the same: if you want something to scale, you need a repeatable operating model.

    In creative work, people often assume the value lives in taste alone. Taste matters, of course, but once you are managing multiple brands or content streams, taste has to be supported by structure. Otherwise every output becomes a one-off argument.

    The interesting thing here is that the repo is not just producing creative assets. It is creating a system for how those assets should be judged and delivered. That is the right order. First you define the identity. Then you define the checks. Then you can move quickly without drifting away from what the work is supposed to feel like.

    Delivery got more reusable, not just more active

    richardham-co-uk-ConsultancyOS was another good example of this week’s theme. CI was added, licensing and contributing guidance were written, a docs handbook appeared, and a SessionStart hook was introduced. In parallel, there was work on reusable delivery-repo patterns, operational dashboards, and prompt-pack integration.

    That is the sort of repo work that pays off over time in a way that is easy to underestimate.

    Most teams can write a one-off project. Fewer teams can turn that project into a template that makes the next one easier. The difference is not just technical polish. It is whether the project has been made reusable. When the CI, contribution rules, documentation, and bootstrap hooks are all captured explicitly, the next engagement starts with less friction and fewer assumptions.

    I see the same thing in the planning and monitoring work happening in lk-ai-roadmap. The hosting and runtime map was clarified, with a clear “no standing servers” direction. Token usage and cost-versus-budget monitoring were documented. Network and security monitoring agent work moved forward as a pilot spec. And the baseline capture kit and endorsement tracking suggest the project is being built with traceability in mind rather than as an ad hoc collection of tasks.

    That is what mature delivery looks like: fewer hidden decisions, more explicit rules, and less dependence on individual memory.

    Key takeaways

    The pattern across all of this work was remarkably consistent.

    • Policies are better than assumptions. Model routing, fallback behaviour, and role-based usage are clearer when they are written down.
    • Monitoring is only useful when it is accurate. A dashboard with the wrong time window or query behaviour is not a shortcut to clarity.
    • Reuse is a force multiplier. CI, docs, hooks, and templates make the next delivery easier than the last one.
    • Creative work benefits from the same discipline as infrastructure. Identity and quality checks are what make scale possible without drift.
    • Governance works best when it becomes ordinary. The more decisions are captured in the system itself, the less they depend on memory or heroics.

    What I take from this week is simple: the work is moving from experimentation to governable systems. That is a good sign. It means the platform is getting more predictable without losing momentum, and the people operating it are making fewer decisions in the dark.

    That is the kind of progress that compounds.

    If you are building AI systems, automation, or operational dashboards and want help turning the moving parts into something easier to trust, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to run.

  • GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    GitHub Weekly — Trustworthy Systems Through Routing, Observability, and Reuse

    Introduction

    Some weeks are about shipping something obvious. Other weeks are about making the systems around the work easier to trust.

    This week looked like the second kind of week. Across the repos I reviewed, the change was direction, not just volume. Model routing became more deliberate. Monitoring became more accurate. Reusable delivery patterns became more explicit. Creative systems got clearer guardrails. And the small operational details that usually sit in the background were pulled into the foreground where they belong.

    That matters because the hardest part of running AI-assisted or automation-heavy systems is rarely the first prototype. The hard part is the handoff from “this works on my machine” to “this can be operated, explained, and improved without guesswork.” This week’s activity was a good reminder that trustworthy systems are built from a lot of very ordinary decisions made consistently.

    What happened

    Model routing stopped being an assumption and became a policy

    The clearest example came from hermes-mgmt. There was a real shift in how model choice is being handled: routing policy updates were merged, a stable default was pinned down, and provider-diverse fallback and outage behaviour were documented more explicitly. On top of that, role-based routing for research, coding, and review was defined instead of leaving model selection to whatever happened to be convenient in the moment.

    A lot of teams start with a simple belief: pick the biggest or newest model and let it do everything. That works until it doesn’t. Once you begin relying on models for different kinds of work, the distinction between “best model” and “best model for this job” becomes the difference between a healthy control plane and an expensive guessing game.

    I like this change because it turns model selection into an operating decision. It acknowledges that research work, coding work, and review work are not the same thing. It also gives the system a way to recover when a provider has issues, rather than forcing every incident into a manual exception.

    The surrounding work supports that direction too. A stale current-state snapshot was refreshed, the wider documentation was merged, and an earlier activation runbook for mixture-of-agents work shows that the system is being treated as something that needs operational memory, not just clever prompts.

    Observability was corrected, not merely expanded

    The biggest lesson from hamnet was the same one I have seen many times in production environments: a dashboard is only useful if it is telling the truth.

    This week’s work there focused on dashboard reliability rather than cosmetic additions. Grafana datasource timing was corrected, time series queries were given explicit intervals, default time windows were adjusted, and Pushgateway queries were wrapped so they render consistently. There was also a practical backlog item to track services that still need HTTPS equivalents, which is the kind of operational debt that gets forgotten unless somebody writes it down.

    I think this is a good example of how observability should be approached. It is tempting to treat monitoring as a collection of panels: add some graphs, colour some thresholds, and call it “visibility.” But real observability is more demanding. It means the query window is right, the step size is right, the defaults are right, and the chart is actually showing the thing you think it is showing.

    That is not a minor distinction. If your dashboard is technically live but operationally misleading, it can create more risk than it removes.

    There was also a useful signal in the issues around connecting a second Hermes instance as a remote subagent. That kind of work says the control plane is no longer just about one local setup. It is beginning to think in terms of distributed operation, where trust, connectivity, and monitoring all need to survive outside a single happy path.

    Creative systems became more governable

    HamMediaLabs showed a similar pattern, but from a different angle. The work there focused on creative-quality instrumentation, creative-ops playbooks, a per-brand visual identity layer, and a short-form voice system.

    That sounds very different from routing policies and dashboard fixes, but the underlying principle is the same: if you want something to scale, you need a repeatable operating model.

    In creative work, people often assume the value lives in taste alone. Taste matters, of course, but once you are managing multiple brands or content streams, taste has to be supported by structure. Otherwise every output becomes a one-off argument.

    The interesting thing here is that the repo is not just producing creative assets. It is creating a system for how those assets should be judged and delivered. That is the right order. First you define the identity. Then you define the checks. Then you can move quickly without drifting away from what the work is supposed to feel like.

    Delivery got more reusable, not just more active

    richardham-co-uk-ConsultancyOS was another good example of this week’s theme. CI was added, licensing and contributing guidance were written, a docs handbook appeared, and a SessionStart hook was introduced. In parallel, there was work on reusable delivery-repo patterns, operational dashboards, and prompt-pack integration.

    That is the sort of repo work that pays off over time in a way that is easy to underestimate.

    Most teams can write a one-off project. Fewer teams can turn that project into a template that makes the next one easier. The difference is not just technical polish. It is whether the project has been made reusable. When the CI, contribution rules, documentation, and bootstrap hooks are all captured explicitly, the next engagement starts with less friction and fewer assumptions.

    I see the same thing in the planning and monitoring work happening in lk-ai-roadmap. The hosting and runtime map was clarified, with a clear “no standing servers” direction. Token usage and cost-versus-budget monitoring were documented. Network and security monitoring agent work moved forward as a pilot spec. And the baseline capture kit and endorsement tracking suggest the project is being built with traceability in mind rather than as an ad hoc collection of tasks.

    That is what mature delivery looks like: fewer hidden decisions, more explicit rules, and less dependence on individual memory.

    Key takeaways

    The pattern across all of this work was remarkably consistent.

    • Policies are better than assumptions. Model routing, fallback behaviour, and role-based usage are clearer when they are written down.
    • Monitoring is only useful when it is accurate. A dashboard with the wrong time window or query behaviour is not a shortcut to clarity.
    • Reuse is a force multiplier. CI, docs, hooks, and templates make the next delivery easier than the last one.
    • Creative work benefits from the same discipline as infrastructure. Identity and quality checks are what make scale possible without drift.
    • Governance works best when it becomes ordinary. The more decisions are captured in the system itself, the less they depend on memory or heroics.

    What I take from this week is simple: the work is moving from experimentation to governable systems. That is a good sign. It means the platform is getting more predictable without losing momentum, and the people operating it are making fewer decisions in the dark.

    That is the kind of progress that compounds.

    If you are building AI systems, automation, or operational dashboards and want help turning the moving parts into something easier to trust, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to run.

  • The Case for Explicit Policies

    Reliable systems do not emerge from good intentions. They emerge when the rules are explicit enough that another operator can understand what the system is supposed to do without reverse-engineering its behaviour from the wreckage.

    That sounds obvious, but it is still one of the most common gaps I see in automation and AI work. Teams build the workflow, connect the services, and get something working end to end. Then they leave the important decisions half-stated. Which provider is preferred? When should the fallback fire? What counts as a real health check? Which version of a process note is authoritative? The system may run, but the operating model is still fuzzy.

    The problem is not that people are careless. It is that policy work often looks less urgent than delivery work. Until something breaks, the invisible rule feels good enough.

    Where ambiguity shows up first

    The first place ambiguity appears is usually routing.

    A stack with multiple models, providers, queues, or execution paths always contains policy whether the team writes it down or not. If the preferred provider is too expensive for low-value tasks, that is policy. If one model is allowed for drafting but not for final output, that is policy. If a workflow should fall back only on timeout and not on quality failure, that is policy too.

    When none of that is written down clearly, people start inferring intent from whatever happened last.

    That is how teams end up with arguments that sound technical but are really operational:

    • “I thought the cheaper path was the default.”
    • “I assumed the fallback only applied during outages.”
    • “I didn’t realise this job was meant to stay on the private model.”
    • “I thought the dashboard alert meant the workflow had already rerouted.”

    None of those are bugs in isolation. They are symptoms of unstated policy.

    Why observability is part of policy

    The same issue appears in monitoring.

    A lot of dashboards tell you that a process is alive. That is not the same as proving the service is doing the right thing.

    For AI and automation systems, a truthful check usually needs to answer something more useful:

    • did the workflow complete the task it was supposed to complete?
    • did it use the intended path?
    • did it return data that looks structurally valid?
    • did the fallback stay dormant when the primary path was healthy?
    • can the operator see enough detail to explain the outcome afterward?

    If the check cannot answer those questions, the dashboard may still be visually tidy, but it is not giving the operator what they need.

    This is why I think observability should be treated as policy, not just instrumentation. The team has to decide what “working” actually means. Otherwise the monitoring layer simply reflects a vague assumption instead of a deliberate standard.

    Reuse is how policy survives handover

    The other quiet benefit of explicit policy is reuse.

    If a team has to rediscover the same routing rule, the same rollback sequence, or the same publishing checklist every time, then the policy is not really part of the system yet. It still lives in memory.

    That is expensive in a small team and dangerous in a growing one.

    Good reuse does not have to be elaborate. Often it is just a set of plain habits:

    • keep one canonical source of truth for important workflows
    • write fallback conditions near the implementation
    • keep short runbooks for the obvious failure modes
    • use the same naming and review patterns across similar jobs
    • record decisions before context evaporates

    None of that feels exciting while you are doing it. But it changes the quality of handover completely. A new operator no longer has to absorb the entire history of the stack before they can act safely.

    What explicit policy looks like in practice

    In practical terms, I look for a few simple signals.

    1. The preferred path is obvious

    The system should make it clear what happens first, what happens second, and under which conditions the fallback is allowed to take over.

    2. The checks reflect user reality

    A green dashboard should mean more than “something is listening on a port”. It should tell the operator whether the real job still works.

    3. Recovery paths exist before the incident

    If the first time a team documents the rollback sequence is during a failure, the policy work happened too late.

    4. Repeated patterns are actually reusable

    If the same kind of workflow appears three times, there should be a shared pattern instead of three slightly different tribal versions.

    Why this matters more with AI systems

    AI systems raise the cost of ambiguity because they turn a fuzzy rule into machine-speed inconsistency.

    In a manual process, unclear policy wastes time. In an automated one, it can silently change outputs, route work to the wrong provider, or create a trail that is too vague to audit later.

    That is why I think trustworthy AI is less about magic prompts and more about explicit operating rules.

    If the rules matter, write them down.

    If the outcome matters, check the real behaviour.

    If the workflow repeats, make it reusable.

    That does not make the system flashy. It makes it dependable.

    And in production, dependable usually wins.

    If you are building automation that needs to survive handover, escalation, and real operational scrutiny, the AI & Automation Architecture work is designed for exactly that. Or get in touch if you want a second pair of eyes on the operating model before the ambiguity becomes expensive.

  • GitHub Weekly — Security Hardening, Monitoring, and Delivery Discipline

    GitHub Weekly — Security Hardening, Monitoring, and Delivery Discipline

    Introduction

    Some weeks are about visible progress: a new feature, a fresh integration, a launch announcement. Other weeks are about the work that keeps those things safe to run. This was one of those weeks.

    Across the repos I reviewed, the pattern was consistent. Security controls were tightened, monitoring got corrected rather than just expanded, operational runbooks were written down instead of left in people’s heads, and delivery work kept moving in small, deliberate steps. That is not glamorous work, but it is the kind of work that stops a platform from becoming fragile.

    The theme that emerged was simple: maintenance is not separate from delivery. It is part of delivery.

    What happened

    Security work moved from incident response to operating discipline

    In hermes-mgmt, the work was clearly shaped by hard lessons. Prompt injection concerns were captured in the changelog and backlog, webhook authentication was hardened, Qdrant backups were automated, and CVE verification was folded into the wider hardening effort.

    That combination matters. A lot of teams treat security as a single event — a scan, a policy review, a one-off fix. In practice, it is a chain of habits. You need authentication on the edges, backups in the middle, and verification at the end. If any one of those is missing, the system may still look healthy right up until the day it is not.

    What I liked most about this pattern was that the repo was not just reacting to a risk. It was converting the risk into repeatable operations: operator scripts, backlog triage, and the documentation needed to make the next decision easier than the last one.

    Monitoring got corrected, not just decorated

    In hamnet, the week was dominated by practical observability fixes. Grafana datasource timing was corrected, dashboard defaults were adjusted to now-6h, pushgateway metrics were refined, and time series queries were updated so they would render reliably instead of misleadingly.

    That sounds minor until you have lived through a dashboard that lies to you.

    Monitoring failures are often subtle. The data is present, the panels load, and the colours look reassuring — but the time interval is wrong, the step size is off, or the query is hiding the very behaviour you wanted to see. In that state, dashboards become theatre. They exist, but they do not help.

    There was also a useful governance signal in the tracking of HTTP-only services that still need HTTPS evaluation. That is exactly the sort of operational backlog item that gets forgotten unless someone records it explicitly. Good monitoring is not only about more charts. It is about acknowledging what is still unfinished and making that visible.

    AI automation was tightened with guardrails and auditability

    ms365-agentic-ai was another strong example of maturity through constraint. Prompt-injection and AI-security guards were hardened, a hub-and-spoke team operating model was merged in, a plan-only remote command action was introduced, and cost-first routing became part of the model strategy.

    This is the right direction for agentic systems. The mistake many teams make is to optimise for autonomy too early. They want the agent to do more before they have decided how to constrain it, audit it, or roll it back. That is how you end up with a clever demo and an unsafe production system.

    The useful pattern here is the opposite: make autonomy conditional. Limit the command surface. Make some actions plan-only. Keep an audit trail. Route routine work to cheaper models where appropriate. When a system can explain what it intended to do before it does it, you are much closer to something governable.

    Delivery work stayed concrete and decomposed

    ricambio-ai-roadmap looked like a good example of steady execution rather than big-bang progress. Provider configuration UI work landed, pagination and infrastructure fixes followed, LaunchDaemon plists and boot startup scripts were added, and a run-sheet for an onsite email MVP visit was written down.

    There was also a practical focus on PII redaction and provider bake-off work, which tells me the team is thinking about both usability and safety. That combination is important. A delivery track is strongest when it can handle real-world constraints without losing momentum.

    The interesting part here is not any one commit. It is the sequence. Research, configuration, bootstrapping, runbooks, redaction, delivery. That is what healthy delivery looks like when it is being treated as a system rather than a one-off project.

    Governance became a daily rhythm

    The control-tower repo reinforced the same message from a different angle. Daily “Decision Desk” issues kept appearing, which tells me governance is not being treated as a monthly review or a backlog afterthought. It is being made into a rhythm.

    That matters because the absence of rhythm is what creates drift. If you only review decisions occasionally, then the rationale behind them disappears into chat threads and memory. If you capture them daily, you create a trail that can be checked, challenged, and reused.

    Why this matters

    The common thread across all of this work is that automation only becomes trustworthy when it is surrounded by discipline.

    Security controls matter because agents, webhooks, and operational tools all expand the number of ways a system can be influenced.

    Monitoring matters because a dashboard that is technically live but operationally wrong can be worse than no dashboard at all.

    Runbooks matter because the first time something breaks, people rarely have time to invent the procedure from scratch.

    And delivery discipline matters because every project eventually reaches a point where execution is more about coordination, sequencing, and proof than raw feature count.

    I think that is why this week felt coherent even though the repos were diverse. The activity was not random. It was converging on a single operating idea: if you want automation to scale, you have to make the surrounding system easier to trust.

    That means:

    • tightening the edge of the system with authentication and verification
    • making dashboards accurate enough to act on
    • writing down the operational steps before they are needed in anger
    • splitting risky work into smaller, reviewable pieces
    • keeping audit trails and decision logs close to the work itself

    Those are not just engineering habits. They are management habits.

    Key takeaways

    • Security hardening is strongest when it becomes a repeatable operating pattern, not a one-off response.
    • Monitoring is only useful if the time windows, query intervals, and defaults actually match the system you are trying to observe.
    • Agentic systems need guardrails, auditability, and plan-only steps before they need more autonomy.
    • Delivery improves when work is decomposed into concrete, sequenced steps with runbooks and release notes alongside the code.
    • Governance becomes useful when it is captured as a daily rhythm rather than a periodic ceremony.

    This week did not hinge on a single dramatic release. It showed something more important: the system is getting better at protecting itself while it moves.

    That is the work that keeps the next launch calm.

    If you are building automation, AI systems, or operational dashboards and want help turning the hidden work into a reliable operating model, the AI & Automation Architecture service covers exactly this. Or get in touch if you want a practical conversation about making the system easier to trust.

Powered by