RichardHam.co.uk Blog

Blog

  • GitHub Weekly — Resilience Engineering and the Work That Prevents Fires

    Introduction

    There is a pattern that shows up in mature engineering systems: the most important work is not the work that adds features. It is the work that keeps the existing features from falling apart.

    This week’s GitHub activity fits that pattern closely. Across the repos I watched, the dominant theme was resilience — making systems that handle failure gracefully, that do not require heroic intervention when something unexpected happens, and that can be operated by a small team without constant firefighting.

    That resilience showed up in several forms: monitoring pipelines that no longer depend on a single service being alive, deployment processes that treat infrastructure as code, project scoping that starts with a genuinely achievable first step, and dependency maintenance that keeps the whole stack patchable. None of it is glamorous. All of it is what separates a system that works in a demo from one that works in production.

    What happened

    Monitoring learned to survive its own dependencies

    The infrastructure monitoring stack received a set of practical hardening changes. The daily report was made resilient to a scenario where the local AI backend it depends on is unavailable — previously, an outage there would block the report entirely. The Grafana drift-check gained support for API token authentication, removing a dependency on a specific auth path that had been causing unauthorised-access failures. Inventory reconciliation and upgrade catalog hosts were aligned, closing a gap where some hosts were being silently missed.

    These are not feature releases. They are the kind of changes that only matter when something goes wrong — which is exactly when you want them to be there. The pattern is consistent: identify a single point of failure, remove it, document the fix, move on.

    A deployment pipeline became properly declarative

    One of the roadmap repos took a meaningful step forward with a deploy-from-GitHub workflow. Instead of manual steps or ad-hoc scripts, the system can now be managed as code and triggered directly from the repository. Alongside this, the team added a client-facing project plan for the first contained step, rescoping the initial delivery to something that can actually be built and tested: an intake flow and a secure home base.

    This kind of scoping discipline is one of the highest-leverage decisions a project can make. It is tempting to plan the whole system at once, but a plan that starts with a genuinely shippable slice is worth far more than a grand architecture that never lands. The team also documented a production hosting approach in an Architecture Decision Record, making the reasoning explicit and reviewable.

    Cloud credit applications moved from discussion to evidence

    The cloud credits grant repo filled out usage plans across six providers — AWS, Google Cloud, Azure, Oracle OCI, Cloudflare, and NVIDIA Inception — and added three anonymised case studies covering different industry verticals. Each plan is specific enough to be actionable rather than aspirational.

    This is the unglamorous side of grant writing: turning “we should apply everywhere” into a structured set of provider-specific steps with realistic timelines. The case studies ground the application in concrete scenarios rather than abstract potential.

    Content operations kept getting more predictable

    The content and media side of the house continued to professionalise its operations. Branch hygiene was tightened with a policy for cleaning up merged branches. A PR review dashboard and dependency health report were added so the team can see at a glance where attention is needed. Onboarding documentation and a risk register were brought up to date, reducing the bus factor that comes from having processes that only exist in one person’s head.

    On the SEO and brand side, the team mapped out a topical authority plan — a structured approach to building discoverability over time rather than chasing one-off traffic wins. That kind of systematic thinking about content is what separates sustainable growth from random spikes.

    Proposal scoring and launch preparation

    The Atlas Foundation project continued its methodical approach to selecting and shaping proposals. Candidate proposals were scored using an impact model, shaping documents were created for the top contenders, and the launch checklist was updated. A safety checklist, research workflow, and promotion process were added to make the path from proposal to production more explicit.

    The Claude Code handover documentation was also updated, covering team setup, repo sync, and issue triage — the practical details that make it possible for a new contributor to become productive quickly.

    Dependency maintenance kept the stack healthy

    The AI cost tracker repo received a set of dependency bumps — scipy, openai, pytest, coverage, numpy — keeping the Python stack on current supported versions. It is the kind of work that is easy to defer until it becomes urgent, but staying current means security patches apply cleanly and new features are available when needed.

    Key takeaways

    Resilience is built in small increments. None of the changes described above are dramatic on their own. Together, they make the difference between a system that needs constant attention and one that mostly takes care of itself.

    Scoping is a superpower. The decision to rescope a project to a genuinely deliverable first step is one of the most valuable things a team can do. It reduces risk, builds momentum, and creates a foundation for everything that follows.

    Documentation is a force multiplier. Whether it is an Architecture Decision Record for hosting, a risk register for a media team, or a handover doc for new contributors, written context makes every future decision cheaper.

    Maintenance is not failure. Dependency bumps, branch hygiene, and monitoring fixes are not signs that something is wrong. They are signs that the system is being looked after well enough to prevent problems before they happen.

    The best engineering is often invisible. When things work reliably, nobody notices. This week’s activity was a good example of that principle in action.

  • Resilience Is Mostly Boring

    Resilience Is Mostly Boring

    People tend to talk about resilience as if it were dramatic. In practice, it is usually just careful repetition of the right things.

    What changed

    • The valuable work here was about understanding failure before it bites. Backup, restore, and recovery steps are only useful if they are written down and tested.

    • A system that looks healthy during the day but collapses under pressure is not resilient. That means monitoring has to be paired with recovery thinking.

    • The other hidden win was consistency. Small maintenance improvements compound quickly when they reduce the number of odd edge cases operators need to remember.

    Closing thought

    Resilience is rarely a single move. It is a collection of unglamorous habits that keep the floor from falling away.

  • GitHub Weekly — Resilience Engineering and the Work That Prevents Fires

    Introduction

    There is a pattern that shows up in mature engineering systems: the most important work is not the work that adds features. It is the work that keeps the existing features from falling apart.

    This week’s GitHub activity fits that pattern closely. Across the repos I watched, the dominant theme was resilience — making systems that handle failure gracefully, that do not require heroic intervention when something unexpected happens, and that can be operated by a small team without constant firefighting.

    That resilience showed up in several forms: monitoring pipelines that no longer depend on a single service being alive, deployment processes that treat infrastructure as code, project scoping that starts with a genuinely achievable first step, and dependency maintenance that keeps the whole stack patchable. None of it is glamorous. All of it is what separates a system that works in a demo from one that works in production.

    What happened

    Monitoring learned to survive its own dependencies

    The infrastructure monitoring stack received a set of practical hardening changes. The daily report was made resilient to a scenario where the local AI backend it depends on is unavailable — previously, an outage there would block the report entirely. The Grafana drift-check gained support for API token authentication, removing a dependency on a specific auth path that had been causing unauthorised-access failures. Inventory reconciliation and upgrade catalog hosts were aligned, closing a gap where some hosts were being silently missed.

    These are not feature releases. They are the kind of changes that only matter when something goes wrong — which is exactly when you want them to be there. The pattern is consistent: identify a single point of failure, remove it, document the fix, move on.

    A deployment pipeline became properly declarative

    One of the roadmap repos took a meaningful step forward with a deploy-from-GitHub workflow. Instead of manual steps or ad-hoc scripts, the system can now be managed as code and triggered directly from the repository. Alongside this, the team added a client-facing project plan for the first contained step, rescoping the initial delivery to something that can actually be built and tested: an intake flow and a secure home base.

    This kind of scoping discipline is one of the highest-leverage decisions a project can make. It is tempting to plan the whole system at once, but a plan that starts with a genuinely shippable slice is worth far more than a grand architecture that never lands. The team also documented a production hosting approach in an Architecture Decision Record, making the reasoning explicit and reviewable.

    Cloud credit applications moved from discussion to evidence

    The cloud credits grant repo filled out usage plans across six providers — AWS, Google Cloud, Azure, Oracle OCI, Cloudflare, and NVIDIA Inception — and added three anonymised case studies covering different industry verticals. Each plan is specific enough to be actionable rather than aspirational.

    This is the unglamorous side of grant writing: turning “we should apply everywhere” into a structured set of provider-specific steps with realistic timelines. The case studies ground the application in concrete scenarios rather than abstract potential.

    Content operations kept getting more predictable

    The content and media side of the house continued to professionalise its operations. Branch hygiene was tightened with a policy for cleaning up merged branches. A PR review dashboard and dependency health report were added so the team can see at a glance where attention is needed. Onboarding documentation and a risk register were brought up to date, reducing the bus factor that comes from having processes that only exist in one person’s head.

    On the SEO and brand side, the team mapped out a topical authority plan — a structured approach to building discoverability over time rather than chasing one-off traffic wins. That kind of systematic thinking about content is what separates sustainable growth from random spikes.

    Proposal scoring and launch preparation

    The Atlas Foundation project continued its methodical approach to selecting and shaping proposals. Candidate proposals were scored using an impact model, shaping documents were created for the top contenders, and the launch checklist was updated. A safety checklist, research workflow, and promotion process were added to make the path from proposal to production more explicit.

    The Claude Code handover documentation was also updated, covering team setup, repo sync, and issue triage — the practical details that make it possible for a new contributor to become productive quickly.

    Dependency maintenance kept the stack healthy

    The AI cost tracker repo received a set of dependency bumps — scipy, openai, pytest, coverage, numpy — keeping the Python stack on current supported versions. It is the kind of work that is easy to defer until it becomes urgent, but staying current means security patches apply cleanly and new features are available when needed.

    Key takeaways

    Resilience is built in small increments. None of the changes described above are dramatic on their own. Together, they make the difference between a system that needs constant attention and one that mostly takes care of itself.

    Scoping is a superpower. The decision to rescope a project to a genuinely deliverable first step is one of the most valuable things a team can do. It reduces risk, builds momentum, and creates a foundation for everything that follows.

    Documentation is a force multiplier. Whether it is an Architecture Decision Record for hosting, a risk register for a media team, or a handover doc for new contributors, written context makes every future decision cheaper.

    Maintenance is not failure. Dependency bumps, branch hygiene, and monitoring fixes are not signs that something is wrong. They are signs that the system is being looked after well enough to prevent problems before they happen.

    The best engineering is often invisible. When things work reliably, nobody notices. This week’s activity was a good example of that principle in action.

  • GitHub Weekly — Maintenance, Roadmaps, and Cloud Credits

    Introduction

    This week’s GitHub activity had a very clear theme: the useful work was mostly the boring work. There was documentation being tightened, backup and retention notes being clarified, deployment steps being made more explicit, and a set of project plans that turned vague ideas into something a lot more concrete.

    That is not a bad week. In fact, it is often exactly what a mature system looks like. The more stable the stack becomes, the more time it spends on maintenance, documentation, and operational discipline. That is the work that keeps everything else trustworthy.

    Across the repos I watched, the activity clustered around five practical themes: infrastructure hygiene, roadmap definition, content workflow polish, grant application prep, and dependency maintenance. None of that is flashy. All of it matters.

    What happened

    Infrastructure work kept the system honest

    The busiest activity sat in the infrastructure and operations side of the house. There were updates around backup retention cleanup, dashboard changes, hardening notes, and general documentation around how the system should be run.

    That kind of activity is easy to overlook because it does not look like a feature release. But it is exactly what separates a system that merely works from one that can be maintained under pressure. Backups only help if the policy is clear. Dashboards only help if the operator knows what changed. Hardening only matters if the steps are written down well enough to repeat.

    The pattern was familiar: make the system less surprising, reduce ambiguity, and leave behind enough context that the next change is cheaper than the last one.

    A roadmap moved from broad intent to a first step

    A separate repo focused on a client roadmap and quote process kept moving toward something more actionable. The work there was less about code and more about deciding how to proceed: clarifying the first contained step, tightening the project plan, and setting out a production hosting approach that was simple enough to explain and safe enough to defend.

    That sort of progress does not always get the same attention as shipping a new feature, but it usually has more leverage. Once a project has a clearly defined first step, the rest of the sequence becomes easier to estimate, easier to sell, and easier to deliver.

    A lot of project risk disappears the moment the team stops arguing with the abstract and starts working on a bounded slice of reality.

    The content pipeline itself kept improving

    The blog and content workflow also saw useful activity. Drafts moved forward, the content queue was updated, and the publishing path was kept in view rather than left to drift. That may sound like internal plumbing, but it is one of the most important parts of a content operation.

    A post only becomes useful once it can move cleanly from draft to review to publish. If the publishing path is fragile, every article inherits that fragility. If the queue is out of date, the editorial process gets messy. If the import workflow is not reproducible, the whole system becomes harder to trust.

    The good news is that this is exactly the kind of thing you can improve incrementally. A small queue update here, a clean restore step there, a better import path later. Content systems get stronger the same way software systems do: by being made more boring and more predictable.

    Grant application work got real

    There was also a nice burst of grant- and credits-related activity: company profile work, anonymised case studies, and usage plans for different providers.

    This is one of those areas where the unglamorous work is actually the valuable work. A grant application or credits submission is only as strong as the clarity of the evidence behind it. A half-finished profile does not help. A generic case study does not help. A clearly structured, anonymised, provider-specific plan does.

    The week’s pattern suggested exactly that kind of progress: turning a broad opportunity into a set of artifacts that someone else can review without needing a follow-up call to decode them.

    Dependency maintenance continued in the background

    There was also the familiar drip of dependency updates and maintenance churn in the tracking side of the stack. These are the changes that are easiest to mentally file under “later,” which is exactly why they matter.

    Staying current is cheaper than catching up. That applies to packages, documentation, and operational assumptions alike. If you let the baseline drift for too long, every future update costs more and carries more risk.

    The boring work often has the highest long-term return.

    Key takeaways

    1. Maintenance is product work

    The old model says maintenance is overhead and features are the real output. This week looked more like the opposite. The maintenance work was the output — because it made the rest of the system more reliable, easier to explain, and cheaper to change.

    Backup cleanup, hardening docs, queue management, and dependency updates are not distractions from the product. They are what make the product durable.

    2. Documentation is part of the control surface

    Clear notes around deployment, retention, and roadmap decisions reduce the amount of guesswork in the system. That matters because guesswork is expensive. It slows down decisions, creates avoidable errors, and makes recovery harder when something does go wrong.

    Good documentation is not a passive record. It is an active control surface.

    3. Good projects reduce ambiguity early

    The roadmap and quote work showed the value of narrowing scope early. Once you have a contained first step, the whole conversation becomes easier. You can estimate better, communicate better, and avoid the drift that turns a straightforward project into a long-running negotiation.

    The same is true for grant applications and content publishing. The quicker you turn a vague idea into a structured deliverable, the faster the work starts compounding.

    4. Content operations are infrastructure too

    The blog workflow improvements were a reminder that editorial systems need the same sort of discipline as software systems. Drafts, queues, restore steps, and import tooling are part of the infrastructure of communication.

    If that plumbing is reliable, content can move. If it is not, the whole operation gets slower and more brittle.

    5. The best weeks often look mundane

    There was no single dramatic launch this week. No big announcement. No flashy pivot.

    Instead, there was a pattern of steady, practical work that keeps several different systems moving in the right direction. That is often what real progress looks like when a project is maturing: less novelty, more discipline.

    Closing thought

    If you only skim the headlines of a week like this, you might miss the story. But the story is there: the system is getting easier to trust.

    That usually happens in quiet ways — through better notes, clearer boundaries, more deliberate planning, and a publishing workflow that is stable enough to rely on. It is not glamorous, but it is the kind of work that compounds.

    And in the long run, compounding is what you want.

    If you are building something similar and want help making the operational side less fragile — whether that is infrastructure, content workflows, or AI-enabled delivery — that is exactly the kind of work I spend time on. Explore the services or get in touch.

  • GitHub Weekly — Maintenance, Roadmaps, and Cloud Credits

    Introduction

    This week’s GitHub activity had a very clear theme: the useful work was mostly the boring work. There was documentation being tightened, backup and retention notes being clarified, deployment steps being made more explicit, and a set of project plans that turned vague ideas into something a lot more concrete.

    That is not a bad week. In fact, it is often exactly what a mature system looks like. The more stable the stack becomes, the more time it spends on maintenance, documentation, and operational discipline. That is the work that keeps everything else trustworthy.

    Across the repos I watched, the activity clustered around five practical themes: infrastructure hygiene, roadmap definition, content workflow polish, grant application prep, and dependency maintenance. None of that is flashy. All of it matters.

    What happened

    Infrastructure work kept the system honest

    The busiest activity sat in the infrastructure and operations side of the house. There were updates around backup retention cleanup, dashboard changes, hardening notes, and general documentation around how the system should be run.

    That kind of activity is easy to overlook because it does not look like a feature release. But it is exactly what separates a system that merely works from one that can be maintained under pressure. Backups only help if the policy is clear. Dashboards only help if the operator knows what changed. Hardening only matters if the steps are written down well enough to repeat.

    The pattern was familiar: make the system less surprising, reduce ambiguity, and leave behind enough context that the next change is cheaper than the last one.

    A roadmap moved from broad intent to a first step

    A separate repo focused on a client roadmap and quote process kept moving toward something more actionable. The work there was less about code and more about deciding how to proceed: clarifying the first contained step, tightening the project plan, and setting out a production hosting approach that was simple enough to explain and safe enough to defend.

    That sort of progress does not always get the same attention as shipping a new feature, but it usually has more leverage. Once a project has a clearly defined first step, the rest of the sequence becomes easier to estimate, easier to sell, and easier to deliver.

    A lot of project risk disappears the moment the team stops arguing with the abstract and starts working on a bounded slice of reality.

    The content pipeline itself kept improving

    The blog and content workflow also saw useful activity. Drafts moved forward, the content queue was updated, and the publishing path was kept in view rather than left to drift. That may sound like internal plumbing, but it is one of the most important parts of a content operation.

    A post only becomes useful once it can move cleanly from draft to review to publish. If the publishing path is fragile, every article inherits that fragility. If the queue is out of date, the editorial process gets messy. If the import workflow is not reproducible, the whole system becomes harder to trust.

    The good news is that this is exactly the kind of thing you can improve incrementally. A small queue update here, a clean restore step there, a better import path later. Content systems get stronger the same way software systems do: by being made more boring and more predictable.

    Grant application work got real

    There was also a nice burst of grant- and credits-related activity: company profile work, anonymised case studies, and usage plans for different providers.

    This is one of those areas where the unglamorous work is actually the valuable work. A grant application or credits submission is only as strong as the clarity of the evidence behind it. A half-finished profile does not help. A generic case study does not help. A clearly structured, anonymised, provider-specific plan does.

    The week’s pattern suggested exactly that kind of progress: turning a broad opportunity into a set of artifacts that someone else can review without needing a follow-up call to decode them.

    Dependency maintenance continued in the background

    There was also the familiar drip of dependency updates and maintenance churn in the tracking side of the stack. These are the changes that are easiest to mentally file under “later,” which is exactly why they matter.

    Staying current is cheaper than catching up. That applies to packages, documentation, and operational assumptions alike. If you let the baseline drift for too long, every future update costs more and carries more risk.

    The boring work often has the highest long-term return.

    Key takeaways

    1. Maintenance is product work

    The old model says maintenance is overhead and features are the real output. This week looked more like the opposite. The maintenance work was the output — because it made the rest of the system more reliable, easier to explain, and cheaper to change.

    Backup cleanup, hardening docs, queue management, and dependency updates are not distractions from the product. They are what make the product durable.

    2. Documentation is part of the control surface

    Clear notes around deployment, retention, and roadmap decisions reduce the amount of guesswork in the system. That matters because guesswork is expensive. It slows down decisions, creates avoidable errors, and makes recovery harder when something does go wrong.

    Good documentation is not a passive record. It is an active control surface.

    3. Good projects reduce ambiguity early

    The roadmap and quote work showed the value of narrowing scope early. Once you have a contained first step, the whole conversation becomes easier. You can estimate better, communicate better, and avoid the drift that turns a straightforward project into a long-running negotiation.

    The same is true for grant applications and content publishing. The quicker you turn a vague idea into a structured deliverable, the faster the work starts compounding.

    4. Content operations are infrastructure too

    The blog workflow improvements were a reminder that editorial systems need the same sort of discipline as software systems. Drafts, queues, restore steps, and import tooling are part of the infrastructure of communication.

    If that plumbing is reliable, content can move. If it is not, the whole operation gets slower and more brittle.

    5. The best weeks often look mundane

    There was no single dramatic launch this week. No big announcement. No flashy pivot.

    Instead, there was a pattern of steady, practical work that keeps several different systems moving in the right direction. That is often what real progress looks like when a project is maturing: less novelty, more discipline.

    Closing thought

    If you only skim the headlines of a week like this, you might miss the story. But the story is there: the system is getting easier to trust.

    That usually happens in quiet ways — through better notes, clearer boundaries, more deliberate planning, and a publishing workflow that is stable enough to rely on. It is not glamorous, but it is the kind of work that compounds.

    And in the long run, compounding is what you want.

    If you are building something similar and want help making the operational side less fragile — whether that is infrastructure, content workflows, or AI-enabled delivery — that is exactly the kind of work I spend time on. Explore the services or get in touch.

  • The Secrets Management Mistakes I See in AI Infrastructure

    The Secrets Management Mistakes I See in AI Infrastructure

    I have reviewed dozens of AI infrastructure stacks over the past year. The pattern is depressingly consistent: a team spins up Langfuse for LLM observability, adds n8n for workflow automation, plugs in Hermes or another AI agent — and within a month, there are database passwords committed to git and API keys baked directly into Docker Compose files.

    It is not that these teams do not care about security. It is that the tooling is easy to deploy and terrifyingly easy to deploy wrong. The secrets management layer is an afterthought, and by the time anyone notices, the rot is already baked into the repository history — often with several stale copies scattered across forks, CI logs, and deployment scripts.

    The tools are not the problem. The defaults are survivable if you change them immediately and manage them properly. The problem is that almost nobody does.

    Why this matters now

    AI infrastructure is different from a typical web app in one important respect: it holds the keys to your models, your data pipelines, and increasingly, your customer data. When a secrets leak happens in an AI stack, it is not just a credential rotation exercise. It can mean exposing vector stores full of proprietary documents, handing over API keys with uncapped billing, or losing control of an agent that has been given broad access to your internal systems.

    UK SMEs are adopting these tools faster than their security practices can keep up. Boards are asking for AI capability. Technical founders want to move fast. The result is that Langfuse and n8n instances go live with the same enthusiasm and the same rigour as a weekend side project.

    The NCSC has been clear that the shared responsibility model applies here. The platform provides the controls. You have to configure them. And right now, most teams deploying AI infrastructure are not.

    A war story from the field

    I recently reviewed a deployment where the Langfuse Docker Compose file had been committed to a version-controlled repository with production database credentials hardcoded directly into it. Not in an environment variable reference — the actual username and password, sitting in plaintext in a file that had been committed, reviewed, merged, and deployed.

    That was not the only issue. The same deployment had a Claude configuration that did not properly handle credential rotation after operator restarts. Every time the container restarted — patching, scaling, node migration — the service would silently fall back to insecure defaults. Nobody noticed for weeks because the service appeared to be running. It was running. It was just doing so with credentials that had long since been rotated and should no longer have been valid.

    I also found an n8n instance where the encryption key had not been persisted. The team had recreated their n8n container as part of a routine update, and the platform generated a new encryption key on startup. Every credential stored in n8n’s database became undecryptable. Every workflow that depended on stored API keys, database connections, or OAuth tokens broke simultaneously.

    The error message is admirably specific: “A different encryptionKey was used to encrypt the data.” But by the time you see it, all of your workflows are failing in production — and unless you have the original encryption key backed up somewhere outside the container, those credentials are gone.

    These are not edge cases. They are the most common findings in every AI infrastructure review I have done in the last twelve months.

    The five mistakes I see most often

    1. Hardcoded secrets in Docker Compose and environment files

    This is the big one. Teams copy a docker-compose.yml from a project README, fill in their passwords in plaintext, and commit it. Sometimes they remember to add .env to .gitignore but leave the Compose file itself exposed. Sometimes they move the secrets to an environment file but commit that too, because the .gitignore was only added after the first commit.

    2. Using default credentials past the first five minutes of setup

    Default credentials exist so you can get started quickly. They should exist in production for approximately zero seconds after the health check passes. I regularly find admin/admin or changeme on instances that have been running for months, sometimes years. If a tool ships with a default password, changing it should be the very first action in your runbook — not something you plan to do later.

    3. Losing the encryption key when containers are recreated

    This one catches people out constantly with n8n, but it applies to any platform that encrypts stored credentials. When you recreate a container without persisting the encryption keys, a new one is generated and the old data is orphaned. The platform cannot protect you from this. It is a configuration decision. You need to persist encryption keys outside the container lifecycle — in your secrets manager, in your CI/CD pipeline configuration, in a mounted volume that survives container recreation.

    4. Treating the AI stack as lower risk than the rest of the infrastructure

    There is a pernicious perception that the “AI tools” are supplementary, experimental, not worth the same rigour as the production database or the payment gateway. This is backwards. Your AI stack touches your most sensitive data — the documents you embed, the conversations you log, the internal APIs your agents call. It has the broadest external API surface in your architecture. It deserves more scrutiny, not less.

    5. Relying on git history alone to “remove” secrets

    Committing a secret, then removing it in a follow-up commit, does not remove it from git history. It just adds another commit on top. The secret is still there, reachable via git log -p, via GitHub’s commit history, via any clone or fork. Once a secret has been committed, the only safe response is rotation — treat it as compromised and issue new credentials.

    What to do next

    You do not need an enterprise secrets vault to get the basics right. You need a checklist and the discipline to follow it.

    1. Audit your repositories now. Search for common patterns: PASSWORD=, SECRET=, API_KEY=, base64-encoded blobs in environment files. Use tools like gitleaks or trufflehog to scan both current state and history. If anything turns up, rotate the credentials immediately — do not just remove them from the latest commit.

    2. Externalise every secret. In Docker Compose, reference ${VAR} values and use a .env file that is .gitignored at the repository root level. If you are on a managed platform, use its native secrets manager. No exceptions, no “just for now”, no “it is only a development environment.”

    3. Persist your n8n encryption key. Set the N8N_ENCRYPTION_KEY environment variable explicitly — do not let n8n auto-generate it. Store it in your secrets manager and make it part of your container orchestration configuration, not the container itself. Back it up. Test that you can restore it.

    4. Rotate after every environment change. When a container is recreated, when a team member leaves, when you are not even sure something was exposed — rotate. Treat the cost of rotation as negligible compared to the cost of a breach. Automate it wherever you can.

    5. Apply the same standard to AI tools as everything else. Your Langfuse instance, your n8n deployment, your AI agent platform — these hold credentials and data that would interest an attacker. Give them the same security treatment you would give a production database. No concessions for “it is internal” or “it is just a prototype.”

    Where Richard can help

    If you are deploying AI infrastructure and want an honest assessment of how your secrets are managed — before an incident forces the conversation — I offer infrastructure security reviews and fractional CISO engagements tailored to UK SMEs running containers, workflows, and AI tooling.

    The mistakes above are among the most common findings in the engagements I do. They are also almost always fixable in a day.

    Get in touch or review the available services to arrange a review.

  • Turning Roadmaps into Real Work

    Turning Roadmaps into Real Work

    A roadmap only counts when it leads to work somebody can actually do. This week was mostly about making that bridge clearer.

    What changed

    • Maintenance notes became less hand-wavy and more actionable. That matters because a clear maintenance task is easier to schedule, assign, and finish.

    • Planning work moved from ideas toward checklists, guardrails, and dependencies. I find that useful: it turns vague intent into a sequence of real steps.

    • The surrounding operational notes were just as important. Backups, retention, and deployment steps are the boring scaffolding that keep future work moving.

    Closing thought

    The point is not to make roadmaps prettier. It is to make them honest enough that the next person can actually use them.

  • When Maintenance Starts to Look Like the Product

    One of the clearest signs that a system is growing up is that the most important work stops looking dramatic.

    There is less fascination with launch theatre and more attention on maintenance, review discipline, dependency hygiene, documentation, recovery paths, and the quiet operational habits that make future change cheaper than past change.

    From the outside, that can look unexciting. From the inside, it is often the moment the product becomes believable.

    Maintenance is where trust becomes visible

    Early-stage work is usually easy to narrate. New feature. New workflow. New integration. New capability.

    Maintenance work is harder to sell because it rarely produces a neat headline. But it is where a team proves whether it is building something durable or simply accumulating demonstrations.

    If the documentation sharpens, the dependency stream stays current, the review process gets clearer, and the operational logs start forming a usable trail, those are not background chores. They are evidence that the system can survive repetition.

    That matters because most real-world products do not fail during the polished demo. They fail during handover, under load, during maintenance, or when somebody new has to operate them without the full story.

    Governance is not separate from delivery

    A lot of teams treat governance as a separate lane from product work, as though it begins after the useful engineering is finished.

    I think that is backwards.

    Governance is simply the part of delivery that reduces ambiguity for the next decision. It is what turns one-off effort into something that can be inspected, repeated, and improved.

    That can show up in small ways:

    • proposal scoring that makes prioritisation legible
    • review guidance that reduces inconsistent judgement
    • issue tracking that records blockers instead of burying them in chat
    • branch hygiene that makes the delivery path safer
    • run logs that explain what changed and why

    None of that steals time from the product. In mature systems, it becomes part of the product because it changes the cost and risk of every future change.

    The product is bigger than the feature set

    This is the shift many teams eventually have to make.

    The product is not only the visible interface or the raw capability. It is also the collection of operating properties that determine whether the capability can be trusted. Can another operator pick it up? Can a failure be explained? Can a dependency be updated without drama? Can the next release happen without rediscovering everything from scratch?

    Once you ask those questions seriously, maintenance stops looking secondary.

    A healthy maintenance pattern normally improves at least one of these:

    • repeatability
    • auditability
    • reversibility
    • ownership clarity
    • change safety

    If the work improves none of those things, it may be motion without much payoff. But when it does improve them, it is absolutely product work.

    The hidden cost of pretending maintenance is optional

    Teams that down-rank maintenance tend to pay for it later in awkward ways.

    A roadmap becomes harder to trust because the underlying stack drifts. Delivery gets slower because every change has to rediscover old context. Incidents become more expensive because the recovery path is still tribal knowledge. Content and documentation diverge because nobody kept a canonical source of truth.

    None of that feels catastrophic at first. It just makes every subsequent piece of work more fragile.

    That is why mature engineering groups often sound calmer, not louder. They know that the best way to speed up later is to reduce the amount of avoidable uncertainty now.

    What good maintenance work usually has in common

    When maintenance is genuinely improving the system rather than just consuming time, I usually see a few shared characteristics.

    It leaves a clearer trail

    The next operator can understand what happened without interviewing the previous one.

    It reduces future decision cost

    A known pattern, documented rule, or reusable checklist means the same problem will be cheaper next time.

    It makes failures less mysterious

    Even when something still goes wrong, the team has better evidence and a cleaner path to recovery.

    It protects momentum instead of slowing it

    Strong maintenance work makes future delivery easier because the underlying operating model is less chaotic.

    Why this matters so much in automation and AI

    Automation magnifies both good and bad maintenance habits.

    If the workflow is opaque, poor maintenance leaves you with a black box that degrades quietly. If the workflow is well-governed, maintenance turns it into something operators can trust, audit, and extend without guessing.

    That is one reason I think governance, maintenance, and documentation matter more as systems become more autonomous. The machine may be doing more of the execution, but the human still has to understand the operating model well enough to own the outcome.

    That is impossible if maintenance has been treated as optional admin.

    The real signal of maturity

    The real signal of maturity is not that a team has stopped building. It is that the team has started building in a way that leaves the environment safer for the next change.

    That often looks like better maintenance because that is what it is.

    Not glamorous. Not particularly marketable on its own. But essential.

    And once a product reaches that stage, the maintenance work does not sit behind the product. It becomes one of the reasons the product is worth trusting in the first place.

    If you are trying to make systems easier to run, safer to change, and less dependent on tribal memory, the AI & Automation Architecture work is built around exactly that operating model. Or get in touch if you want help turning maintenance, governance, and delivery discipline into an actual advantage rather than a recurring source of drag.

  • The Week Maintenance Became the Product

    Introduction

    When I reviewed this week’s GitHub activity, the obvious story was not a flashy launch or a dramatic refactor. It was something more interesting: the maintenance work became the signal.

    Across 20 public repositories, there were 126 events in the last seven days. That includes dependency churn, proposal scoring, documentation hardening, automation that keeps generating useful operational signals, and a few infrastructure notes that show where the rough edges still are. In other words, the system is doing what mature systems do: it is spending less time proving that it works and more time proving that it can be trusted.

    That matters. A lot of teams talk about shipping. Fewer teams talk about the work that makes shipping repeatable. This week was a good reminder that the second part is where the real leverage lives.

    What happened

    Project Atlas moved from ideas into structure

    The project-atlas-foundation repo was the busiest in the set, with 40 events. The pattern was clear: proposals were being scored, shaping docs were being created, and the launch checklist was being tightened up.

    A few examples stood out:

    • proposal scoring for items #3–#7
    • issue creation for shaping the next phase of work
    • launch planning documents and handover notes
    • an infrastructure blocker being logged rather than ignored

    That last point matters. Good teams do not hide blockers behind optimism. They surface them early, name them clearly, and move on with the fix.

    What I like about this kind of activity is that it shows a project moving from momentum to discipline. Ideas are important, but structured ideas are what survive contact with reality.

    HamMediaLabs built the scaffolding around the work

    The HamMediaLabs repo contributed 24 events, and the theme was governance. Onboarding material, a development guide, a risk register, a PR review dashboard, a dependency health report, and a branch hygiene policy all landed in quick succession.

    That is not glamorous work. It is, however, the work that keeps the rest of the team from drifting into inconsistency.

    I have seen enough small teams to know this pattern well: once the repository starts to matter, the undocumented habits start to cost real time. Branch hygiene prevents stale work from hanging around. A PR dashboard shows where the bottlenecks are. A risk register makes it harder to ignore known issues until they become incidents.

    This is governance that lives with the code, which is the only place it reliably gets used.

    Control Tower kept producing daily operational signals

    The control-tower repo was smaller in volume, but it was one of the most revealing. The automation continued to produce daily “Decision Desk” issues, including entries for June 24 and June 25.

    That might sound routine, and that is exactly why it matters.

    A healthy automation pipeline should stop feeling novel. It should become part of the operating rhythm. When the bot keeps producing the same class of signal every day, it means the process is stable enough to be useful and visible enough to trust.

    The lesson here is not that automation is exciting. It is that automation is only useful when it becomes boring in the right way.

    ai-cost-tracker showed the cost of staying current

    The ai-cost-tracker repo generated a series of Dependabot updates across scipy, openai, pytest, coverage, numpy, and a pip group update. That is the kind of activity people often skim past, but I think it tells an important story.

    Dependency maintenance is not just housekeeping. It is a proxy for the health of the project.

    If updates are ignored for too long, the stack gets harder to trust. If they are handled routinely, the project stays closer to current, and current is cheaper than catching up later. In a world where AI tooling and Python libraries evolve quickly, that matters even more. Every stale dependency is a future problem with interest attached.

    hermes-agent continued to harden the core toolchain

    The hermes-agent repo added a SecureScore view through PR #1, while other commits focused on gateway hardening, Windows restart reliability, and test improvements.

    That combination is exactly what I want to see from a core platform repository. A view is added because the team needs better visibility. The gateway is hardened because resilience matters. Tests are improved because confidence is not something you can fake for very long.

    This is the difference between building a tool and operating a system.

    Key takeaways

    1. Maintenance is becoming product work

    The old mental model says feature work is valuable and maintenance is overhead. This week argues for a better model: maintenance is part of the product.

    If you are scoring proposals, documenting risks, tightening branch rules, and keeping dependencies current, you are not stepping away from the product. You are building the conditions that let the product keep existing.

    2. Automation is most valuable when it is visible

    The daily Decision Desk issues in control-tower are a good example. Automation should not disappear into a black box. It should leave a trace that operators can inspect.

    That trace becomes a decision record, a trend line, and a health indicator all at once. If your automation cannot explain itself in the repository, it is probably too fragile to trust elsewhere.

    3. Governance only works when it is close to the code

    Onboarding guides, risk registers, review dashboards, and branch hygiene policies are all useful because they are embedded in the same workflow as the work they govern.

    That is the difference between documentation and practice. One gets read when a problem appears. The other shapes the problem before it appears.

    4. Dependency updates are an operational metric

    The ai-cost-tracker updates are not just noise from a bot. They are evidence that the project is being actively maintained.

    If dependency updates are arriving regularly, that means somebody is paying attention. If they are not, the project may still look healthy right up until the day it suddenly is not.

    5. Mature teams spend more time making work repeatable

    The most important shift I saw this week was not in any single repository. It was in the shape of the work overall.

    The repositories are spending time on handover notes, launch checklists, review dashboards, daily operational signals, and stability fixes. That is what maturity looks like in practice. Not less work, just better-structured work.

    Closing thought

    If you only scan the headlines of a busy week, you can miss the real story. This one was not about one big release. It was about the quiet engineering that makes releases sustainable.

    That is usually where the long-term value sits: in the boring, repeatable work that turns a collection of repos into a system.

    If you are building something similar and want help turning operational complexity into something more manageable, that is exactly the kind of work I cover in my services and contact pages.

  • The Week Maintenance Became the Product

    Introduction

    When I reviewed this week’s GitHub activity, the obvious story was not a flashy launch or a dramatic refactor. It was something more interesting: the maintenance work became the signal.

    Across 20 public repositories, there were 126 events in the last seven days. That includes dependency churn, proposal scoring, documentation hardening, automation that keeps generating useful operational signals, and a few infrastructure notes that show where the rough edges still are. In other words, the system is doing what mature systems do: it is spending less time proving that it works and more time proving that it can be trusted.

    That matters. A lot of teams talk about shipping. Fewer teams talk about the work that makes shipping repeatable. This week was a good reminder that the second part is where the real leverage lives.

    What happened

    Project Atlas moved from ideas into structure

    The project-atlas-foundation repo was the busiest in the set, with 40 events. The pattern was clear: proposals were being scored, shaping docs were being created, and the launch checklist was being tightened up.

    A few examples stood out:

    • proposal scoring for items #3–#7
    • issue creation for shaping the next phase of work
    • launch planning documents and handover notes
    • an infrastructure blocker being logged rather than ignored

    That last point matters. Good teams do not hide blockers behind optimism. They surface them early, name them clearly, and move on with the fix.

    What I like about this kind of activity is that it shows a project moving from momentum to discipline. Ideas are important, but structured ideas are what survive contact with reality.

    HamMediaLabs built the scaffolding around the work

    The HamMediaLabs repo contributed 24 events, and the theme was governance. Onboarding material, a development guide, a risk register, a PR review dashboard, a dependency health report, and a branch hygiene policy all landed in quick succession.

    That is not glamorous work. It is, however, the work that keeps the rest of the team from drifting into inconsistency.

    I have seen enough small teams to know this pattern well: once the repository starts to matter, the undocumented habits start to cost real time. Branch hygiene prevents stale work from hanging around. A PR dashboard shows where the bottlenecks are. A risk register makes it harder to ignore known issues until they become incidents.

    This is governance that lives with the code, which is the only place it reliably gets used.

    Control Tower kept producing daily operational signals

    The control-tower repo was smaller in volume, but it was one of the most revealing. The automation continued to produce daily “Decision Desk” issues, including entries for June 24 and June 25.

    That might sound routine, and that is exactly why it matters.

    A healthy automation pipeline should stop feeling novel. It should become part of the operating rhythm. When the bot keeps producing the same class of signal every day, it means the process is stable enough to be useful and visible enough to trust.

    The lesson here is not that automation is exciting. It is that automation is only useful when it becomes boring in the right way.

    ai-cost-tracker showed the cost of staying current

    The ai-cost-tracker repo generated a series of Dependabot updates across scipy, openai, pytest, coverage, numpy, and a pip group update. That is the kind of activity people often skim past, but I think it tells an important story.

    Dependency maintenance is not just housekeeping. It is a proxy for the health of the project.

    If updates are ignored for too long, the stack gets harder to trust. If they are handled routinely, the project stays closer to current, and current is cheaper than catching up later. In a world where AI tooling and Python libraries evolve quickly, that matters even more. Every stale dependency is a future problem with interest attached.

    hermes-agent continued to harden the core toolchain

    The hermes-agent repo added a SecureScore view through PR #1, while other commits focused on gateway hardening, Windows restart reliability, and test improvements.

    That combination is exactly what I want to see from a core platform repository. A view is added because the team needs better visibility. The gateway is hardened because resilience matters. Tests are improved because confidence is not something you can fake for very long.

    This is the difference between building a tool and operating a system.

    Key takeaways

    1. Maintenance is becoming product work

    The old mental model says feature work is valuable and maintenance is overhead. This week argues for a better model: maintenance is part of the product.

    If you are scoring proposals, documenting risks, tightening branch rules, and keeping dependencies current, you are not stepping away from the product. You are building the conditions that let the product keep existing.

    2. Automation is most valuable when it is visible

    The daily Decision Desk issues in control-tower are a good example. Automation should not disappear into a black box. It should leave a trace that operators can inspect.

    That trace becomes a decision record, a trend line, and a health indicator all at once. If your automation cannot explain itself in the repository, it is probably too fragile to trust elsewhere.

    3. Governance only works when it is close to the code

    Onboarding guides, risk registers, review dashboards, and branch hygiene policies are all useful because they are embedded in the same workflow as the work they govern.

    That is the difference between documentation and practice. One gets read when a problem appears. The other shapes the problem before it appears.

    4. Dependency updates are an operational metric

    The ai-cost-tracker updates are not just noise from a bot. They are evidence that the project is being actively maintained.

    If dependency updates are arriving regularly, that means somebody is paying attention. If they are not, the project may still look healthy right up until the day it suddenly is not.

    5. Mature teams spend more time making work repeatable

    The most important shift I saw this week was not in any single repository. It was in the shape of the work overall.

    The repositories are spending time on handover notes, launch checklists, review dashboards, daily operational signals, and stability fixes. That is what maturity looks like in practice. Not less work, just better-structured work.

    Closing thought

    If you only scan the headlines of a busy week, you can miss the real story. This one was not about one big release. It was about the quiet engineering that makes releases sustainable.

    That is usually where the long-term value sits: in the boring, repeatable work that turns a collection of repos into a system.

    If you are building something similar and want help turning operational complexity into something more manageable, that is exactly the kind of work I cover in my services and contact pages.

Powered by