RichardHam.co.uk Blog

Tag: infrastructure

  • Weekly GitHub Activity — 22–29 June 2026

    Introduction

    Another busy week in the repositories. Between infrastructure automation, platform hardening, commercial product development, and a small mountain of documentation, the commit logs tell the story of a systemthat’s rapidly moving from “hand-crafted and held together with SSH sessions” toward something far more repeatable and governed. Let’s walk through what landed.

    What Happened

    hermes-mgmt: Hardening, Governance, and AI Spend Controls

    The hermes-mgmt repository saw 10 commits and 20 pull requests this week, almost all of them orbiting two themes: Tier 2 operational hardening and AI spend governance.

    On the hardening side, a suite of operator runbooks and governance records landed — documenting the execution of Tier 2 hardening scripts, session decisions from backlog triage, and a full operator TODO instruction document. PR #694 tackled memory infrastructure head-on, covering a Qdrant deployment, gateway shutdown procedures, and Telegram 429 rate-limit handling. PR #695 addressed bug remediation across Spotify integration, n8n authentication, vision/video backend issues, and a 429 retry storm that was causing more harm than good.

    The AI spend work is particularly interesting. PR #696 introduced budget policies, cost telemetry, context-optimisation strategies, and prompt-caching controls — essentially an AI spend optimisation control plane. As LLM-powered workloads grow, having a formal mechanism to track, cap, and optimise spend is no longer nice-to-have; it’s operational hygiene. The design docs for child components that accompanied this work sketch out a system where every agent invocation carries a cost context, and budget alarms fire before the invoice does.

    Security compliance also got attention. PR #691 documented progress against SecureScore controls SEC-002, SEC-006, RES-002, and MAT-001 — a reminder that even in a solo-operator environment, treating compliance as code keeps the audit trail honest.

    A quieter but important fix: PR #640 made the Mem0→Letta memory backfill idempotent, preventing duplicate records on re-runs. Small change, big impact on data integrity.

    hamnet: The Control Plane Takes Shape

    hamnet was the most active repository this week — 10 commits, 18 pull requests, and 40 open issues. The headline is the emergence of a single-pane control-plane dashboard (PR #183), backed by a growing collection of infrastructure collectors.

    The control-plane work is being built incrementally and deliberately:

    • PR #177 established the identity model, a static inventory MVP, and the first batch of workload collectors.
    • PR #179 added a second batch — AgentRadar, Ollama status, repository state, and NetBox integration.
    • PR #176 fixed Grafana drift authentication and a tvheadend timeout issue that was causing monitoring gaps.

    On the infrastructure-as-code front, PR #178 delivered a comprehensive IaC roadmap with schemas, runbooks, and a system-classification model. PR #194 proposed enforcing unattended-upgrades policies on external cloud hosts through Ansible — a defensive measure that ensures security patches land even when nobody’s watching.

    Several fixes made day-to-day operations smoother: a script to tag missing VMs with ansible:managed in NetBox (commit 33d2fd9), a UTF-8 locale fix for ansible-inventory (commit 198d9b8), and adding Gitea to the Docker container detection logic (commit a950c67) — a gap that meant the monitoring stack was blind to a self-hosted service.

    A Docker image drift management runbook (commit 81fa372) and operator runbooks for 11 infrastructure issues (commit 8f09076) round out the picture. hamnet is building the operational backbone: not just running services, but knowing what’s running, whether it should be, and what to do when it isn’t.

    The 40 open issues deserve a mention — they’re a candid inventory of what needs attention, from an Oracle scanner that’s out of sync and a PVE exporter reporting zero metrics to Loki missing log-shipping and stale NetBox entries. Filing them is half the battle.

    ms365-agentic-ai: Collectors, Cost Controls, and Production Readiness

    The ms365-agentic-ai repository is accelerating. Five commits this week laid significant groundwork:

    • A deployable Azure Functions host with a run-history sink and live-path tests (commit 5465364).
    • Live read-only fetch wiring for all collectors (commit 416c537).
    • A cost-management component combining an estimator with an Azure cost collector (commit 86ab202).
    • Production-readiness runbook with dry-run and preflight tooling (commit 5da09c4).
    • A simple start guide, docs index, and FAQ/troubleshooting section (commit 212729a).

    The 10 open issues (#6–#15) are essentially the product roadmap: approval-gated remediation models, audit logging, Teams health digests, Azure resource health and Defender XDR collectors, Graph permissions matrices, and licensing/cost matrices. What’s notable is the breadth — this isn’t just a tool, it’s an operational platform being designed with governance from the start.

    richardham-co-uk-ConsultancyOS: Commercial Foundations

    ConsultancyOS — the commercial operating-system template — got its infrastructure in order this week. PR #12 added CI, licensing, a contributing guide, and a docs handbook. A SessionStart hook for web sessions (commit 1a54d22) shows the product thinking extending into the user experience.

    The 9 open issues map out the productisation journey: operational dashboards, a GitHub delivery-repo template, LLM integration, n8n workflow scaffolding, accountant review processes, company identity, and integrations with Dolibarr CRM and Invoice Ninja. This is a project that’s explicitly moving from “personal tooling” toward “repeatable commercial offering.”

    on-maintenance-ai-roadmap: Client-Facing Delivery

    The on-maintenance-ai-roadmap repository focused on communication and deployment model this week. A client-facing project plan (PR #23), an Architecture Decision Record for production hosting in client-owned Azure (PR #21), and a deploy-from-GitHub infrastructure model (PR #24) all point to a delivery pattern where the client retains ownership and control of their environment while the project provides the automation and intelligence layer.

    ai-cloud-credits-grant: Cloud Funding Applications

    Six commits across the ai-cloud-credits-grant repository added usage plans for NVIDIA Inception, Cloudflare, Oracle OCI, AWS, and Google Cloud. These are structured applications for cloud-provider credit programmes — the kind of unglamorous but essential work that unlocks compute capacity for projects that need it. The 6 open issues track the remaining work: a master company profile, applications to the major providers, and an evidence pack.

    Key Takeaways

    1. The control plane is real. hamnet’s dashboard, collectors, and identity model aren’t prototypes — they’re an incremental build toward a system that provides visibility and governance across the entire infrastructure. When you can see everything from one pane, you can fix things before they break.

    2. AI spend is now a first-class concern. The hermes-mgmt AI spend control plane — with budget policies, cost telemetry, and prompt caching — reflects a mature approach to LLM operations. The era of “just call the API and hope the bill is reasonable” is over, at least in this stack.

    3. Documentation as a delivery artifact. Across every repository this week, documentation wasn’t an afterthought — it was a first-class deliverable. Operator runbooks, governance records, ADRs, client-facing plans, and onboarding guides shipped alongside the code. This is the behaviour that separates sustainable projects from weekend hacks.

    4. Compliance isn’t optional, even for small teams. The SecureScore work in hermes-mgmt and the approval-gated remediation model in ms365-agentic-ai show that compliance and auditability are being designed in, not bolted on. That’s cheaper and more effective at any scale.

    5. Open issues are a feature, not a bug. hamnet’s 40 open issues and ms365-agentic-ai’s 10 open issues aren’t backlogs — they’re transparent roadmaps. Filing what needs doing, and making it visible, is how solo operators avoid the “I’ll remember that later” trap.

    This post is part of a weekly series summarising GitHub activity across the project portfolio. Previous entries are available in the blog archive.

  • The Secrets Management Mistakes I See in AI Infrastructure

    The Secrets Management Mistakes I See in AI Infrastructure

    I have reviewed dozens of AI infrastructure stacks over the past year. The pattern is depressingly consistent: a team spins up Langfuse for LLM observability, adds n8n for workflow automation, plugs in Hermes or another AI agent — and within a month, there are database passwords committed to git and API keys baked directly into Docker Compose files.

    It is not that these teams do not care about security. It is that the tooling is easy to deploy and terrifyingly easy to deploy wrong. The secrets management layer is an afterthought, and by the time anyone notices, the rot is already baked into the repository history — often with several stale copies scattered across forks, CI logs, and deployment scripts.

    The tools are not the problem. The defaults are survivable if you change them immediately and manage them properly. The problem is that almost nobody does.

    Why this matters now

    AI infrastructure is different from a typical web app in one important respect: it holds the keys to your models, your data pipelines, and increasingly, your customer data. When a secrets leak happens in an AI stack, it is not just a credential rotation exercise. It can mean exposing vector stores full of proprietary documents, handing over API keys with uncapped billing, or losing control of an agent that has been given broad access to your internal systems.

    UK SMEs are adopting these tools faster than their security practices can keep up. Boards are asking for AI capability. Technical founders want to move fast. The result is that Langfuse and n8n instances go live with the same enthusiasm and the same rigour as a weekend side project.

    The NCSC has been clear that the shared responsibility model applies here. The platform provides the controls. You have to configure them. And right now, most teams deploying AI infrastructure are not.

    A war story from the field

    I recently reviewed a deployment where the Langfuse Docker Compose file had been committed to a version-controlled repository with production database credentials hardcoded directly into it. Not in an environment variable reference — the actual username and password, sitting in plaintext in a file that had been committed, reviewed, merged, and deployed.

    That was not the only issue. The same deployment had a Claude configuration that did not properly handle credential rotation after operator restarts. Every time the container restarted — patching, scaling, node migration — the service would silently fall back to insecure defaults. Nobody noticed for weeks because the service appeared to be running. It was running. It was just doing so with credentials that had long since been rotated and should no longer have been valid.

    I also found an n8n instance where the encryption key had not been persisted. The team had recreated their n8n container as part of a routine update, and the platform generated a new encryption key on startup. Every credential stored in n8n’s database became undecryptable. Every workflow that depended on stored API keys, database connections, or OAuth tokens broke simultaneously.

    The error message is admirably specific: “A different encryptionKey was used to encrypt the data.” But by the time you see it, all of your workflows are failing in production — and unless you have the original encryption key backed up somewhere outside the container, those credentials are gone.

    These are not edge cases. They are the most common findings in every AI infrastructure review I have done in the last twelve months.

    The five mistakes I see most often

    1. Hardcoded secrets in Docker Compose and environment files

    This is the big one. Teams copy a docker-compose.yml from a project README, fill in their passwords in plaintext, and commit it. Sometimes they remember to add .env to .gitignore but leave the Compose file itself exposed. Sometimes they move the secrets to an environment file but commit that too, because the .gitignore was only added after the first commit.

    2. Using default credentials past the first five minutes of setup

    Default credentials exist so you can get started quickly. They should exist in production for approximately zero seconds after the health check passes. I regularly find admin/admin or changeme on instances that have been running for months, sometimes years. If a tool ships with a default password, changing it should be the very first action in your runbook — not something you plan to do later.

    3. Losing the encryption key when containers are recreated

    This one catches people out constantly with n8n, but it applies to any platform that encrypts stored credentials. When you recreate a container without persisting the encryption keys, a new one is generated and the old data is orphaned. The platform cannot protect you from this. It is a configuration decision. You need to persist encryption keys outside the container lifecycle — in your secrets manager, in your CI/CD pipeline configuration, in a mounted volume that survives container recreation.

    4. Treating the AI stack as lower risk than the rest of the infrastructure

    There is a pernicious perception that the “AI tools” are supplementary, experimental, not worth the same rigour as the production database or the payment gateway. This is backwards. Your AI stack touches your most sensitive data — the documents you embed, the conversations you log, the internal APIs your agents call. It has the broadest external API surface in your architecture. It deserves more scrutiny, not less.

    5. Relying on git history alone to “remove” secrets

    Committing a secret, then removing it in a follow-up commit, does not remove it from git history. It just adds another commit on top. The secret is still there, reachable via git log -p, via GitHub’s commit history, via any clone or fork. Once a secret has been committed, the only safe response is rotation — treat it as compromised and issue new credentials.

    What to do next

    You do not need an enterprise secrets vault to get the basics right. You need a checklist and the discipline to follow it.

    1. Audit your repositories now. Search for common patterns: PASSWORD=, SECRET=, API_KEY=, base64-encoded blobs in environment files. Use tools like gitleaks or trufflehog to scan both current state and history. If anything turns up, rotate the credentials immediately — do not just remove them from the latest commit.

    2. Externalise every secret. In Docker Compose, reference ${VAR} values and use a .env file that is .gitignored at the repository root level. If you are on a managed platform, use its native secrets manager. No exceptions, no “just for now”, no “it is only a development environment.”

    3. Persist your n8n encryption key. Set the N8N_ENCRYPTION_KEY environment variable explicitly — do not let n8n auto-generate it. Store it in your secrets manager and make it part of your container orchestration configuration, not the container itself. Back it up. Test that you can restore it.

    4. Rotate after every environment change. When a container is recreated, when a team member leaves, when you are not even sure something was exposed — rotate. Treat the cost of rotation as negligible compared to the cost of a breach. Automate it wherever you can.

    5. Apply the same standard to AI tools as everything else. Your Langfuse instance, your n8n deployment, your AI agent platform — these hold credentials and data that would interest an attacker. Give them the same security treatment you would give a production database. No concessions for “it is internal” or “it is just a prototype.”

    Where Richard can help

    If you are deploying AI infrastructure and want an honest assessment of how your secrets are managed — before an incident forces the conversation — I offer infrastructure security reviews and fractional CISO engagements tailored to UK SMEs running containers, workflows, and AI tooling.

    The mistakes above are among the most common findings in the engagements I do. They are also almost always fixable in a day.

    Get in touch or review the available services to arrange a review.

Powered by CookieAdmin