RichardHam.co.uk Blog

Tag: engineering

  • When Systems Stop Relying on Guesswork

    A lot of avoidable operational pain comes from one simple source: somebody had to guess.

    They had to guess which model the workflow would choose. They had to guess whether a green dashboard meant the service was actually working. They had to guess how to recover something because the only real runbook lived in one person’s head. They had to guess which version of a draft was the right one to publish.

    That is a small sentence with expensive consequences.

    In my experience, many reliability problems are not caused by one dramatic bug. They come from layers of ambiguity that were tolerated because the system appeared to work often enough. The workflow usually picked the expected path. The dashboard was usually telling the truth. The operator usually remembered the recovery sequence. The content pipeline usually published the right version.

    Usually is not a safe operating model.

    Where the friction really sits

    The obvious failure is the outage. The more common failure is ambiguity that keeps making the whole stack harder to trust.

    Routing decisions

    If a system can choose between providers, models, or execution paths, then the decision logic needs to be durable and visible. Otherwise behaviour changes by accident.

    That is how teams end up with a platform that feels inconsistent even though each individual component is technically healthy. One person thinks the cheap path is the default. Another thinks the reliable path is. A third assumes the fallback only applies in a timeout case. None of that is a software problem at first. It is a policy problem.

    The fix is rarely glamorous. Write the rule. Keep it near the implementation. Make it obvious what happens first, what happens second, and what happens when the preferred path fails.

    Monitoring that tells the truth

    There is a big difference between “the process is running” and “the service is working”.

    A healthy-looking process table can still hide a broken dependency, a dead API path, a stale token, or an empty data plane. If the check only proves that something is listening on a port, it is not telling the operator what they actually need to know.

    Useful monitoring is usually the boring kind. Does the real workflow respond, with the right output, in the way the operator expects? Can someone tell whether a fallback fired? Can they see whether the service remained functionally correct rather than merely alive?

    If not, the dashboard is decoration.

    Recovery and repeatability

    A lot of infrastructure looks solid right up until someone else has to operate it.

    That is where repeatable recovery paths matter. Clear defaults. Known fallbacks. A documented sequence for fixing the obvious failure modes. The point is not to eliminate human judgement. The point is to stop every incident from turning into archaeology.

    Runbooks, checklists, and decision notes are not admin overhead for serious systems. They are part of the product if the product has to survive handover.

    Publishing discipline

    This applies to content as much as infrastructure.

    If the publishing path is fuzzy, you get duplicate posts, mismatched versions, archive drift, and copy that sounds assembled rather than written. That is not only a branding issue. It is the same operational problem wearing different clothes.

    The draft, the review step, the canonical source file, and the final publish target all need to be explicit. Otherwise the system eventually ships the wrong thing.

    Why AI multiplies the cost of ambiguity

    AI and automation raise the stakes because they convert vague process into machine-speed inconsistency.

    When a human operator is doing everything manually, weak process often shows up as wasted time. When workflows start routing, deciding, and acting on their own, the same vagueness becomes a production risk. The system can now repeat the wrong assumption quickly, quietly, and at scale.

    That is why I keep coming back to the same standard:

    • if it matters, write it down
    • if it repeats, make it reusable
    • if it can fail, define the fallback
    • if it is monitored, test the real behaviour
    • if it is published, keep one clear source of truth

    None of that is flashy. All of it matters.

    The useful kind of progress

    The best weeks are not always the dramatic ones.

    Sometimes the real progress is a tighter routing rule. A dashboard that finally reflects reality. A recovery path that no longer depends on memory. A publishing workflow that stops sending the wrong version into the public archive.

    That kind of work rarely looks exciting from the outside. It just makes the system calmer, safer, and easier to trust.

    And in practice, that is often the difference between something that demos well and something that survives production.

    If you want help turning AI or automation from a pile of moving parts into an operating model people can actually trust, start with AI & Automation Architecture or get in touch.

  • When Systems Stop Relying on Guesswork

    When Systems Stop Relying on Guesswork

    A system usually becomes more trustworthy for one simple reason: the people running it stop having to guess.

    That sounds obvious, but it is often the difference between something that demos well and something that survives real use. Early on, teams tend to depend on memory, informal habits, and whoever happens to know how the pieces fit together. The more a platform grows, the more fragile that becomes. Hidden choices turn into operational risk. Unwritten expectations become inconsistent behaviour. Small misunderstandings start to show up as outages, wasted time, or avoidable rework.

    The most useful progress is often not glamorous. It looks like defining defaults, documenting recovery paths, tightening feedback loops, and making the important steps repeatable. In other words: less mystery, more system.

    What changed

    Across a typical week of work, the strongest improvements usually fall into a few categories.

    1. Decisions become explicit

    A lot of problems come from the same place: a critical choice was never written down.

    That might be a model selection rule, a deployment expectation, a fallback path, or a review step. If the team needs the same answer more than once, it should probably live in a policy, not in somebody’s head.

    Explicit decisions are easier to audit, easier to improve, and easier to hand over. They also reduce the chance that the system behaves differently depending on who touched it last.

    2. Observability becomes truthful

    Dashboards are useful only when they reflect reality.

    It is very easy to build something that looks informative while quietly hiding the thing you actually need to know. Wrong time windows, weak queries, misleading defaults, and over-optimistic thresholds can all create the illusion of control. The result is a lot of visual noise and very little operational value.

    Good observability is boring in the best possible way. It tells you what happened, when it happened, and whether the current state matches the story the interface is telling.

    3. Reuse reduces friction

    Reusable process is one of the highest-leverage things a team can build.

    Runbooks, checklists, CI steps, templates, decision logs, and bootstrap scripts all do the same job: they reduce the amount of context that has to be remembered manually. That makes the next delivery faster, but more importantly it makes the next delivery less dependent on luck.

    A mature team does not just ship features. It also ships the scaffolding that makes future work safer.

    4. Governance becomes routine

    Governance only helps when it is part of the rhythm of work.

    If decisions are captured sporadically, the rationale gets lost. If they are recorded regularly, they start to form a usable memory for the organisation. That is especially important in AI and automation work, where the consequences of a shortcut can show up much later than the moment it was taken.

    Routine does not have to mean bureaucracy. It can simply mean that important questions are answered in the same place, the same way, every time.

    Why this matters for AI and automation

    AI systems are often judged by how clever they look in isolation. That is the wrong benchmark.

    The real test is whether the system can be operated reliably by other people. Can it recover when something fails? Can it explain what it is doing? Can the team change it without fear? Can the output be trusted enough to act on?

    Those questions are answered by architecture, process, and discipline more than by novelty.

    If a workflow depends on a model, the model choice should be deliberate. If a chart drives decisions, it should be accurate. If a process gets used repeatedly, it should be documented. If a system matters, its operation should not depend on tribal knowledge.

    A practical standard

    A useful rule of thumb is this:

    • if it matters, write it down
    • if it repeats, make it reusable
    • if it can fail, define the fallback
    • if it is monitored, make sure the monitoring is honest
    • if it is operationally important, keep the reasoning close to the work

    That standard is not flashy, but it works.

    It makes AI systems easier to run.
    It makes automation easier to trust.
    It makes teams less dependent on memory.
    And it turns a collection of clever individual actions into something more durable.

    Closing thought

    The best systems are not the ones that never need attention. They are the ones that make attention easier to apply.

    When the defaults are clear, the checks are real, and the process is reusable, the whole stack becomes calmer. That is the kind of progress that matters most: not dramatic, but lasting.

    If you are trying to make an AI or automation workflow more reliable, the first step is usually not adding more complexity. It is removing guesswork.

    If you want help turning a messy operational process into something clearer and easier to trust, the AI & Automation Architecture service is a good place to start. Or get in touch for a practical conversation.

  • GitHub Weekly — Resilience Engineering and the Work That Prevents Fires

    Introduction

    There is a pattern that shows up in mature engineering systems: the most important work is not the work that adds features. It is the work that keeps the existing features from falling apart.

    This week’s GitHub activity fits that pattern closely. Across the repos I watched, the dominant theme was resilience — making systems that handle failure gracefully, that do not require heroic intervention when something unexpected happens, and that can be operated by a small team without constant firefighting.

    That resilience showed up in several forms: monitoring pipelines that no longer depend on a single service being alive, deployment processes that treat infrastructure as code, project scoping that starts with a genuinely achievable first step, and dependency maintenance that keeps the whole stack patchable. None of it is glamorous. All of it is what separates a system that works in a demo from one that works in production.

    What happened

    Monitoring learned to survive its own dependencies

    The infrastructure monitoring stack received a set of practical hardening changes. The daily report was made resilient to a scenario where the local AI backend it depends on is unavailable — previously, an outage there would block the report entirely. The Grafana drift-check gained support for API token authentication, removing a dependency on a specific auth path that had been causing unauthorised-access failures. Inventory reconciliation and upgrade catalog hosts were aligned, closing a gap where some hosts were being silently missed.

    These are not feature releases. They are the kind of changes that only matter when something goes wrong — which is exactly when you want them to be there. The pattern is consistent: identify a single point of failure, remove it, document the fix, move on.

    A deployment pipeline became properly declarative

    One of the roadmap repos took a meaningful step forward with a deploy-from-GitHub workflow. Instead of manual steps or ad-hoc scripts, the system can now be managed as code and triggered directly from the repository. Alongside this, the team added a client-facing project plan for the first contained step, rescoping the initial delivery to something that can actually be built and tested: an intake flow and a secure home base.

    This kind of scoping discipline is one of the highest-leverage decisions a project can make. It is tempting to plan the whole system at once, but a plan that starts with a genuinely shippable slice is worth far more than a grand architecture that never lands. The team also documented a production hosting approach in an Architecture Decision Record, making the reasoning explicit and reviewable.

    Cloud credit applications moved from discussion to evidence

    The cloud credits grant repo filled out usage plans across six providers — AWS, Google Cloud, Azure, Oracle OCI, Cloudflare, and NVIDIA Inception — and added three anonymised case studies covering different industry verticals. Each plan is specific enough to be actionable rather than aspirational.

    This is the unglamorous side of grant writing: turning “we should apply everywhere” into a structured set of provider-specific steps with realistic timelines. The case studies ground the application in concrete scenarios rather than abstract potential.

    Content operations kept getting more predictable

    The content and media side of the house continued to professionalise its operations. Branch hygiene was tightened with a policy for cleaning up merged branches. A PR review dashboard and dependency health report were added so the team can see at a glance where attention is needed. Onboarding documentation and a risk register were brought up to date, reducing the bus factor that comes from having processes that only exist in one person’s head.

    On the SEO and brand side, the team mapped out a topical authority plan — a structured approach to building discoverability over time rather than chasing one-off traffic wins. That kind of systematic thinking about content is what separates sustainable growth from random spikes.

    Proposal scoring and launch preparation

    The Atlas Foundation project continued its methodical approach to selecting and shaping proposals. Candidate proposals were scored using an impact model, shaping documents were created for the top contenders, and the launch checklist was updated. A safety checklist, research workflow, and promotion process were added to make the path from proposal to production more explicit.

    The Claude Code handover documentation was also updated, covering team setup, repo sync, and issue triage — the practical details that make it possible for a new contributor to become productive quickly.

    Dependency maintenance kept the stack healthy

    The AI cost tracker repo received a set of dependency bumps — scipy, openai, pytest, coverage, numpy — keeping the Python stack on current supported versions. It is the kind of work that is easy to defer until it becomes urgent, but staying current means security patches apply cleanly and new features are available when needed.

    Key takeaways

    Resilience is built in small increments. None of the changes described above are dramatic on their own. Together, they make the difference between a system that needs constant attention and one that mostly takes care of itself.

    Scoping is a superpower. The decision to rescope a project to a genuinely deliverable first step is one of the most valuable things a team can do. It reduces risk, builds momentum, and creates a foundation for everything that follows.

    Documentation is a force multiplier. Whether it is an Architecture Decision Record for hosting, a risk register for a media team, or a handover doc for new contributors, written context makes every future decision cheaper.

    Maintenance is not failure. Dependency bumps, branch hygiene, and monitoring fixes are not signs that something is wrong. They are signs that the system is being looked after well enough to prevent problems before they happen.

    The best engineering is often invisible. When things work reliably, nobody notices. This week’s activity was a good example of that principle in action.

  • GitHub Weekly — Resilience Engineering and the Work That Prevents Fires

    Introduction

    There is a pattern that shows up in mature engineering systems: the most important work is not the work that adds features. It is the work that keeps the existing features from falling apart.

    This week’s GitHub activity fits that pattern closely. Across the repos I watched, the dominant theme was resilience — making systems that handle failure gracefully, that do not require heroic intervention when something unexpected happens, and that can be operated by a small team without constant firefighting.

    That resilience showed up in several forms: monitoring pipelines that no longer depend on a single service being alive, deployment processes that treat infrastructure as code, project scoping that starts with a genuinely achievable first step, and dependency maintenance that keeps the whole stack patchable. None of it is glamorous. All of it is what separates a system that works in a demo from one that works in production.

    What happened

    Monitoring learned to survive its own dependencies

    The infrastructure monitoring stack received a set of practical hardening changes. The daily report was made resilient to a scenario where the local AI backend it depends on is unavailable — previously, an outage there would block the report entirely. The Grafana drift-check gained support for API token authentication, removing a dependency on a specific auth path that had been causing unauthorised-access failures. Inventory reconciliation and upgrade catalog hosts were aligned, closing a gap where some hosts were being silently missed.

    These are not feature releases. They are the kind of changes that only matter when something goes wrong — which is exactly when you want them to be there. The pattern is consistent: identify a single point of failure, remove it, document the fix, move on.

    A deployment pipeline became properly declarative

    One of the roadmap repos took a meaningful step forward with a deploy-from-GitHub workflow. Instead of manual steps or ad-hoc scripts, the system can now be managed as code and triggered directly from the repository. Alongside this, the team added a client-facing project plan for the first contained step, rescoping the initial delivery to something that can actually be built and tested: an intake flow and a secure home base.

    This kind of scoping discipline is one of the highest-leverage decisions a project can make. It is tempting to plan the whole system at once, but a plan that starts with a genuinely shippable slice is worth far more than a grand architecture that never lands. The team also documented a production hosting approach in an Architecture Decision Record, making the reasoning explicit and reviewable.

    Cloud credit applications moved from discussion to evidence

    The cloud credits grant repo filled out usage plans across six providers — AWS, Google Cloud, Azure, Oracle OCI, Cloudflare, and NVIDIA Inception — and added three anonymised case studies covering different industry verticals. Each plan is specific enough to be actionable rather than aspirational.

    This is the unglamorous side of grant writing: turning “we should apply everywhere” into a structured set of provider-specific steps with realistic timelines. The case studies ground the application in concrete scenarios rather than abstract potential.

    Content operations kept getting more predictable

    The content and media side of the house continued to professionalise its operations. Branch hygiene was tightened with a policy for cleaning up merged branches. A PR review dashboard and dependency health report were added so the team can see at a glance where attention is needed. Onboarding documentation and a risk register were brought up to date, reducing the bus factor that comes from having processes that only exist in one person’s head.

    On the SEO and brand side, the team mapped out a topical authority plan — a structured approach to building discoverability over time rather than chasing one-off traffic wins. That kind of systematic thinking about content is what separates sustainable growth from random spikes.

    Proposal scoring and launch preparation

    The Atlas Foundation project continued its methodical approach to selecting and shaping proposals. Candidate proposals were scored using an impact model, shaping documents were created for the top contenders, and the launch checklist was updated. A safety checklist, research workflow, and promotion process were added to make the path from proposal to production more explicit.

    The Claude Code handover documentation was also updated, covering team setup, repo sync, and issue triage — the practical details that make it possible for a new contributor to become productive quickly.

    Dependency maintenance kept the stack healthy

    The AI cost tracker repo received a set of dependency bumps — scipy, openai, pytest, coverage, numpy — keeping the Python stack on current supported versions. It is the kind of work that is easy to defer until it becomes urgent, but staying current means security patches apply cleanly and new features are available when needed.

    Key takeaways

    Resilience is built in small increments. None of the changes described above are dramatic on their own. Together, they make the difference between a system that needs constant attention and one that mostly takes care of itself.

    Scoping is a superpower. The decision to rescope a project to a genuinely deliverable first step is one of the most valuable things a team can do. It reduces risk, builds momentum, and creates a foundation for everything that follows.

    Documentation is a force multiplier. Whether it is an Architecture Decision Record for hosting, a risk register for a media team, or a handover doc for new contributors, written context makes every future decision cheaper.

    Maintenance is not failure. Dependency bumps, branch hygiene, and monitoring fixes are not signs that something is wrong. They are signs that the system is being looked after well enough to prevent problems before they happen.

    The best engineering is often invisible. When things work reliably, nobody notices. This week’s activity was a good example of that principle in action.

Powered by