The Secrets Management Mistakes I See in AI Infrastructure

I have reviewed dozens of AI infrastructure stacks over the past year. The pattern is depressingly consistent: a team spins up Langfuse for LLM observability, adds n8n for workflow automation, plugs in Hermes or another AI agent — and within a month, there are database passwords committed to git and API keys baked directly into Docker Compose files.

It is not that these teams do not care about security. It is that the tooling is easy to deploy and terrifyingly easy to deploy wrong. The secrets management layer is an afterthought, and by the time anyone notices, the rot is already baked into the repository history — often with several stale copies scattered across forks, CI logs, and deployment scripts.

The tools are not the problem. The defaults are survivable if you change them immediately and manage them properly. The problem is that almost nobody does.

Why this matters now

AI infrastructure is different from a typical web app in one important respect: it holds the keys to your models, your data pipelines, and increasingly, your customer data. When a secrets leak happens in an AI stack, it is not just a credential rotation exercise. It can mean exposing vector stores full of proprietary documents, handing over API keys with uncapped billing, or losing control of an agent that has been given broad access to your internal systems.

UK SMEs are adopting these tools faster than their security practices can keep up. Boards are asking for AI capability. Technical founders want to move fast. The result is that Langfuse and n8n instances go live with the same enthusiasm and the same rigour as a weekend side project.

The NCSC has been clear that the shared responsibility model applies here. The platform provides the controls. You have to configure them. And right now, most teams deploying AI infrastructure are not.

A war story from the field

I recently reviewed a deployment where the Langfuse Docker Compose file had been committed to a version-controlled repository with production database credentials hardcoded directly into it. Not in an environment variable reference — the actual username and password, sitting in plaintext in a file that had been committed, reviewed, merged, and deployed.

That was not the only issue. The same deployment had a Claude configuration that did not properly handle credential rotation after operator restarts. Every time the container restarted — patching, scaling, node migration — the service would silently fall back to insecure defaults. Nobody noticed for weeks because the service appeared to be running. It was running. It was just doing so with credentials that had long since been rotated and should no longer have been valid.

I also found an n8n instance where the encryption key had not been persisted. The team had recreated their n8n container as part of a routine update, and the platform generated a new encryption key on startup. Every credential stored in n8n’s database became undecryptable. Every workflow that depended on stored API keys, database connections, or OAuth tokens broke simultaneously.

The error message is admirably specific: “A different encryptionKey was used to encrypt the data.” But by the time you see it, all of your workflows are failing in production — and unless you have the original encryption key backed up somewhere outside the container, those credentials are gone.

These are not edge cases. They are the most common findings in every AI infrastructure review I have done in the last twelve months.

The five mistakes I see most often

1. Hardcoded secrets in Docker Compose and environment files

This is the big one. Teams copy a docker-compose.yml from a project README, fill in their passwords in plaintext, and commit it. Sometimes they remember to add .env to .gitignore but leave the Compose file itself exposed. Sometimes they move the secrets to an environment file but commit that too, because the .gitignore was only added after the first commit.

2. Using default credentials past the first five minutes of setup

Default credentials exist so you can get started quickly. They should exist in production for approximately zero seconds after the health check passes. I regularly find admin/admin or changeme on instances that have been running for months, sometimes years. If a tool ships with a default password, changing it should be the very first action in your runbook — not something you plan to do later.

3. Losing the encryption key when containers are recreated

This one catches people out constantly with n8n, but it applies to any platform that encrypts stored credentials. When you recreate a container without persisting the encryption keys, a new one is generated and the old data is orphaned. The platform cannot protect you from this. It is a configuration decision. You need to persist encryption keys outside the container lifecycle — in your secrets manager, in your CI/CD pipeline configuration, in a mounted volume that survives container recreation.

4. Treating the AI stack as lower risk than the rest of the infrastructure

There is a pernicious perception that the “AI tools” are supplementary, experimental, not worth the same rigour as the production database or the payment gateway. This is backwards. Your AI stack touches your most sensitive data — the documents you embed, the conversations you log, the internal APIs your agents call. It has the broadest external API surface in your architecture. It deserves more scrutiny, not less.

5. Relying on git history alone to “remove” secrets

Committing a secret, then removing it in a follow-up commit, does not remove it from git history. It just adds another commit on top. The secret is still there, reachable via git log -p, via GitHub’s commit history, via any clone or fork. Once a secret has been committed, the only safe response is rotation — treat it as compromised and issue new credentials.

What to do next

You do not need an enterprise secrets vault to get the basics right. You need a checklist and the discipline to follow it.

1. Audit your repositories now. Search for common patterns: PASSWORD=, SECRET=, API_KEY=, base64-encoded blobs in environment files. Use tools like gitleaks or trufflehog to scan both current state and history. If anything turns up, rotate the credentials immediately — do not just remove them from the latest commit.

2. Externalise every secret. In Docker Compose, reference ${VAR} values and use a .env file that is .gitignored at the repository root level. If you are on a managed platform, use its native secrets manager. No exceptions, no “just for now”, no “it is only a development environment.”

3. Persist your n8n encryption key. Set the N8N_ENCRYPTION_KEY environment variable explicitly — do not let n8n auto-generate it. Store it in your secrets manager and make it part of your container orchestration configuration, not the container itself. Back it up. Test that you can restore it.

4. Rotate after every environment change. When a container is recreated, when a team member leaves, when you are not even sure something was exposed — rotate. Treat the cost of rotation as negligible compared to the cost of a breach. Automate it wherever you can.

5. Apply the same standard to AI tools as everything else. Your Langfuse instance, your n8n deployment, your AI agent platform — these hold credentials and data that would interest an attacker. Give them the same security treatment you would give a production database. No concessions for “it is internal” or “it is just a prototype.”

Where Richard can help

If you are deploying AI infrastructure and want an honest assessment of how your secrets are managed — before an incident forces the conversation — I offer infrastructure security reviews and fractional CISO engagements tailored to UK SMEs running containers, workflows, and AI tooling.

The mistakes above are among the most common findings in the engagements I do. They are also almost always fixable in a day.

Get in touch or review the available services to arrange a review.