AI is great... at first

6 min read
Article

AI assistants shine in the first sessions, then fall off once a project drags on. Here's why, and what to do about it.

The free AI newsletter
AI is great... at first

AI doesn't really know your files

There's something we forget fast with an AI assistant. Whether you use it in chat mode, asking questions, or in agentic mode, letting it drive your tools, it doesn't really know your files. Even when you wire up your folders, your documents and connectors to your data.

It knows exactly one thing: its context. That is, whatever it has loaded in the present moment, the text it has in front of it this very second. The rest doesn't exist for it. Your hundred files are right there, within reach, but until they land in its window at the right time, they might as well be on another planet.

That's why AI is often great... at first. Then noticeably less so.

Why it shines at kickoff

At the start of a project, the assistant has a huge edge: it has almost nothing to load. Everything that matters fits in a handkerchief. A few messages, a clear instruction, two files, and the right context is in front of it all at once.

It's a bit like an empty desk. As long as there are three folders on it, you grab the right one in a second. The problem doesn't show up there. It shows up when the desk gets covered in piles.

The bigger the corpus, the harder it gets

Here's the knot, and it's counterintuitive. The more material you feed your assistant, the harder it becomes for it to grab the right information at the right moment. The memory piling up week after week, the multiplying files, the swelling documentation: everything that was supposed to help ends up cluttering.

And in agentic mode it gets dizzying. A simple chat already grows heavier as the conversation runs. But when you let the AI drive your tools, its reference corpus has no real limit: it opens your code repo, reads pages of documentation, swallows the output of every command it runs, stacks the trace of previous steps. Chat suffers the effect. Agentic lives it tenfold.

This isn't armchair intuition, it's measured. A Chroma study published in July 2025 tested 18 of the most advanced models, including GPT-4.1, Claude 4 and Gemini 2.5. Its conclusion fits in one sentence: models don't use their context uniformly, and become increasingly unpredictable as the input grows. Even on simple tasks.

Worse, whatever sits in the middle of a long context is poorly used. Work by Liu and colleagues (Stanford, 2024) showed a U-shaped curve: a model spots information placed at the start or the end well, but its accuracy drops by more than 30% when it's buried in the middle. An instruction written early in the conversation becomes nearly invisible later.

It's not just a size problem

You might think a bigger context window fixes everything. It doesn't. A research effort from late 2025 shows that context length on its own degrades performance, even when the model retrieves the sought information perfectly. The volume itself weighs.

The result, in the field, is always the same scenes. The assistant re-asks a question already settled. It re-proposes an approach you'd ruled out. You have to re-fix the same error for the third time. The project moves forward, the corpus swells, and reliability, meanwhile, recedes.

"Just tidy up," except nobody tidies up

The natural reaction is to tell yourself you need to get organized. Sort, summarize, archive, keep your folders clean so the assistant finds the right file. On paper, it works.

In real life, no. Tidying up regularly is a discipline almost nobody keeps. The proof fits in one word: wiki. Companies have been paying for internal wikis for twenty years, and they're never up to date.

The page exists, but it dates back two reorganizations. Tidying treated as a separate chore, added on top of the workload, always loses to the urgency of the moment.

The system has to be the tidying

So the real question isn't "how do I force myself to tidy up." It's: what if tidying weren't an extra task, but the byproduct of the work itself?

One image captures it well, as long as you don't take it literally. On old Windows, you had to defragment your disk: a separate maintenance job, a chore you'd launch at night and put off for months. On Linux, the system tidies as it writes: the more you use the disk, the better organized it stays, without you thinking about it.

The technical details are more nuanced than that image, but it aims true. In one case, tidying is a separate action you forget. In the other, it's stitched to usage.

That's exactly what to aim for with AI. Not tidying after the fact, but a system where working already produces order. Where every decision made ends up written in the right place with no extra effort, ready to be reloaded when the moment comes.

What we're building on our end

That's the problem we wanted to solve for ourselves. A media outlet, a channel and a framework to run at the same time, over the long haul: relying on a session's memory just wasn't sustainable.

So we built Manence, our way of keeping the assistant reliable when a project drags on. The guiding idea fits in one sentence: the system is the tidying.

A knowledge base for stable facts. A dated log of decisions where nothing gets erased. Work folders for ongoing tasks. The whole thing fills up as you work, not in a Friday-evening archiving chore.

We'll say it plainly: it's our product. The site www.manence.ai has been live since today, with a promise that sums it all up: make your AI reliable on projects that last.

The framework is coming as open source on GitHub; in the meantime, you can sign up at manence.ai to be notified when it's available.

But the idea applies without us, starting now. Don't rely on the session's context: give your assistant an outside where information lives, and make sure that outside keeps itself current through use. An assistant that shines at kickoff proves nothing about the long run. What carries a project isn't the machine's memory, it's the tidying your system does in your place.

Topics covered:

ProductivityAnalysis

Frequently asked questions

Why is an AI assistant more reliable at the start of a project?
Early on there's little to load: a few messages and two files are enough to put the right context in front of it at once. The trouble starts when the corpus grows and the right piece of information gets buried.
What is a model's context?
It's what the model has loaded right now: the text it actually has in front of it. Your files and connectors exist, but until they land in its context window at the right moment, they don't exist for the model.
Why isn't a bigger context window enough?
Because length alone degrades performance, even when the model does retrieve the target information. Work from 2024-2025 shows accuracy drops for information buried in the middle of a long context, and that sheer volume weighs on the model.
How do you keep an AI assistant reliable over time?
Don't rely on session memory. Give the assistant an outside where information lives (a knowledge base, a dated log, work folders) and make sure that outside stays current through use, not through a separate archiving chore.
The free AI newsletter