SoN Vol 2, Issue 7: Silent Drift


Vol 2, Issue 7 | Feb 18 2026
Your AI Setup Is Quietly Degrading. Here's How to Check.

Dear Reader,

I renamed a skill in my AI setup last week with no issues — no error, no warning, everything looked fine. Go me 👊🏻

Then about three days later, I discovered that separate components of my workflow were silently failing, still referencing the old skill name, quietly returning nothing and skipping tasks without complaint. Silently drifting away from each other.


How AI Setups Can Fail

AI tools work from the context you gave them — not from what's currently true, but from what was true when you set things up. That context doesn't update automatically. The AI doesn't know it's stale.

So when something shifts — a file moves, a tool gets replaced, a workflow changes — the AI keeps working from what it has. It doesn't error. It doesn't warn you. It quietly uses the outdated reference, or skips the step it can't complete, or generates output from a context that no longer reflects reality.

The output still looks plausible. That's the problem.

Silent drift is what I call it when your AI setup is quietly pointing to things that have moved, changed, or disappeared — and nothing tells you. The AI works with what it has. You assume it's working as intended.

It isn't.

This Isn’t Just a Power User Problem

My setup might be unusual — 200+ custom components, all referencing each other by name and file path. But the underlying pattern shows up in much simpler configurations.

Have you ever set up a custom GPT with a system prompt that references “our current pricing” or “our product lineup” — and then never updated it when those changed? Configured an AI assistant to your role, your team, your workflow — and then changed any of those things? Used a saved workflow that was calibrated months ago to conditions that have since shifted?

The AI doesn’t know things changed — it can’t know. It works with the context you gave it when you configured it. If that context has drifted from reality, the output has too. Quietly, without error.


What I Found in My Own Setup

When I actually audited my configuration, the first run found 17 critical issues. Not “could be improved” — critical.

A voice profile for my newsletter agent was pointing to a file I’d moved months earlier. The agent was still running and generating content, just without any of the voice guidance it was supposed to follow. The output looked like mine. It wasn’t quite.

Seven components were referencing tools from a server I’d replaced with something better. They weren’t failing with an error — they were just quietly not invoking the tools correctly.

Three skill references had one-character typos — close enough that I’d never spotted them in the file names, different enough that nothing could find them. Complete failure to invoke. Completely silent.

Plus 55 configuration drift warnings where components were still using patterns from three months ago.

All of it silent. Zero error messages.


Auditing Your Own Setup

The diagnostic logic that uncovered this follows a simple principle: map what you have against what you’re referencing, and find the gaps.

For most AI users, that’s less technical than it sounds. A few questions:

When did you last review your context? Any AI tool you’ve configured with persistent context — custom GPTs, saved instructions, Claude Projects, workflow automations — carries a snapshot of your situation from whenever you set it up. How old is that snapshot?

What’s changed since you configured it? Your role, your team, your products or pricing, the tools you use, the workflows you follow. Any of these can create drift between what the AI was calibrated to and what’s actually true now.

If something was quietly failing, would you notice? The output might still look plausible. The task might still seem to complete. Drift reveals itself gradually — in output that’s a bit off, in steps that get skipped, in advice that doesn’t quite fit anymore.

What are you trusting the AI to remember that it might have gotten wrong? Memory, context documents, saved instructions — these are snapshots. They don’t update automatically.


Three Things To Try This Week

  1. Audit your most-used AI setup. Pick the workflow you use most — custom GPT, Claude Project, saved prompt, automation — and ask: when did I last review the context I gave it? What’s changed since then?
  2. Do a “what’s still true” check. Go through the instructions or context you gave your AI tool and mark anything that might no longer be accurate. Outdated role descriptions, old team structures, pricing or products that changed — these are the silent drift candidates.
  3. Ask the AI to flag its own assumptions. “What are you assuming about me or my situation that might no longer be accurate?” It won’t catch everything, but the answers are often more useful than you’d expect.

For anyone running a custom Claude Code setup with a lot of components: I built a diagnostic tool that automates this. It runs 7 checks — vault path validation, skill cross-references, MCP server health, CLI availability, configuration drift, staleness detection, orphan detection. Open source, MIT licensed. Full breakdown on the blog and on GitHub.


Signal Over Noise Is Going Paid

Quick reminder: Signal Over Noise moves to a paid newsletter on March 4th.

If you’re reading this, you’re already here — and the Founding Member rate of €49/year is still available. That’s not “first year” pricing. It’s your rate, permanently, for being early. Monthly pricing gets announced next week.

The short version of why: I’d rather be accountable to readers than to advertisers. The full explanation is in the announcement post.

Elsewhere

Claude Code Ecosystem Healthcheck

Open sourced this week - Detect drift between interconnected Claude Code components: skills, agents, MCP servers, vault paths, CLI tools, and configuration policies. GitHub.

Claude Code Ecosystem Health Check

The technical deep-dive behind today’s topic — what each of the 7 diagnostic checks actually does, and why grepping for broken references misses what structured scanning catches. For Claude Code users with complex setups. On the blog.

Second Brain Chronicles

New newsletter launched last week - Second Brain Chronicles, where I discuss my Claude Code + Obsidian system more in depth for those who want more of a technical focus. Free each week on Fridays.

Until next week,

Jim

Signal Over Noise is weekly, reader-first publication on AI "without the hype" published by Jim Christian. If you've been forwarded this issue, you can subscribe here: go.signalovernoise.at.

600 1st Ave, Ste 330 PMB 92768, Seattle, WA 98104-2246
​
Unsubscribe · Preferences​

Jim Christian - Signal Over Noise

What I care about is helping people bring technology into their lives well — where it's genuinely needed, and where it can do the most good. Not tech for its own sake, not change to look modern. Sometimes that means AI; just as often it means something far simpler.The tech world is accelerating. This is my attempt to slow things down a little: to pick apart the hype and look for the reality and the truth in the messages — primarily around AI, and how it impacts us all.

Read more from Jim Christian - Signal Over Noise
A yellow robot beside an empty desk chair holds up a long paper scroll of numbered question cards looping across the floor, while a green dog trots off with card 7; a tray tagged GROUP 1 sits under a chalkboard that reads THREE AT A TIME.

VOL 2, ISSUE 43 · NOVEMBER 2026 One of my favourite ways of working with AI is to set Claude or ChatGPT a task, or a series of them, before I go and do other jobs around the house. But often when I come back to it, it's waiting on me with a list of questions about details I don't remember discussing. I've spent the last hour on something else, and now it wants answers. I've often typed back something close to this: "These are too many questions. I'm a human. I can't deal with this much...

A boxy yellow robot at a wooden workbench assembles a small box labelled SKILL beside a large, worn book labelled SOURCE. Question cards numbered 1 and 2 sit on a spike on the bench, and a scruffy green dog walks up carrying card 3 in its mouth.

VOL 2, ISSUE 42 · NOVEMBER 2026 Frameworks are all around, and they've been in use for ages. Some of my favourites are the five whys, where you ask “why?” until you reach the cause of a problem, and the Eisenhower matrix, which sorts your tasks by how urgent and how important they are. There's no reason why you can't use these in your daily workflows to help you keep organised or to troubleshoot problems. If you don't know one for your job, ask your AI what established methods exist for it,...

A boxy yellow robot at a ticket kiosk labelled DATA HUB holds up a paper ticket marked TOKEN, with a canary in a cage and a scruffy green dog by the gate.

VOL 2, ISSUE 41 · NOVEMBER 2026 Part of my daily routine has changed in the last week, in that I'm talking more with Claude in voice mode on my school runs in the mornings. It beats the repetition of listening to the radio station, and it gives me more space to ramble and go down the odd rabbit hole as I question the morning headlines and look for topics of interest. When I get on to something I like and want to follow up on, I ask Claude to make sure that it's saved as a Google Doc that I...