Agent memory, RAG, or fine-tuning: what should you change?

Start with the failed task

Use memory when the agent needs durable state from earlier work. Use retrieval-augmented generation, or RAG, when it needs relevant external material at answer time. Consider fine-tuning when you need a repeatable behavior change and have suitable examples. Fix the tools when correct intent still produces bad execution.

These mechanisms can be combined. They are not stages that every product must pass through. We would choose the smallest intervention that explains a repeated failure, then test it against the current agent.

For example, if an agent knows which file to open but the file tool returns stale bytes, adding memories will not fix the underlying read. If it opens the right document and still writes the wrong output format, retrieval may not be the missing piece.

A decision table for common failures

Name what the agent lacked at the moment it failed. Then test whether supplying only that missing input changes the outcome. This quick diagnostic can save a training run.

Memory preserves state; retrieval selects evidence

A useful memory should include provenance, scope, and a way to become stale. A note that a workspace used one path last month is not a guarantee that the path still exists. The agent should verify changing facts before acting on them.

In a RAG system, retrieval supplies material that conditions the answer. The original RAG paper combines parametric and non-parametric memory. In an application, the immediate question is whether the needed source was found and whether the answer used it correctly.

Evaluate retrieval and generation separately. If the correct passage never reaches the model, inspect indexing, chunking, filters, or ranking. If the right passage is present but the answer contradicts it, test the prompt and answer process. Do not collapse both failures into one vague accuracy score.

• Read the original RAG paper

Fine-tuning changes learned behavior

Fine-tuning updates model parameters or trainable adapters. LoRA is one approach that trains a smaller set of low-rank updates. This can make adaptation more practical, but the data and evaluation problem remain.

Before training, assemble examples of the behavior you actually want. Separate training and test cases by source or task family where possible, so near-duplicate examples do not create a false sense of generalization. Retain a baseline prompt that uses the same model without the adapter.

Training is not a reliable substitute for looking up rapidly changing facts. If a specification changes every week, maintaining a retrievable source may be easier to audit than repeatedly encoding changes into weights. A trained model can still benefit from retrieval.

• Hugging Face's explanation of LoRA

Run a controlled test before combining everything

Create four conditions: the current agent, the agent with relevant memory, the agent with retrieved sources, and the agent with the behavior change you are testing. Use the same tasks and a declared budget. If you change the prompt, tools, and model together, you will not know which change helped.

Track task completion, unsupported claims, stale-state errors, latency, and cost. Add cases in which memory is irrelevant or retrieval returns a misleading passage. The system needs to ignore bad context as well as use good context.

Then combine the interventions that earned their place. Check the combination again: a larger prompt may add latency or bury a useful instruction. The best individual techniques do not automatically form the best overall system.

Which of these is recursive self-improvement?

None is automatically recursive. An agent can read a memory or use an adapter without changing how it improves itself. The recursive link appears when a retained change helps the system produce or assess further changes.

Suppose an agent learns a better process for selecting evidence, then uses that process to improve its own tools. That is a testable candidate for a recursive loop. You would still need a comparison showing that the changed process helped, not merely that the next run had more context.

For most teams, the immediate win is reliable task completion. Keep memory, source retrieval, learned behavior, and tool correctness as separate experimental variables. That makes each improvement easier to understand and reverse.

• Define the recursive step and its evidence

Start with state you can inspect

Before training anything, check what the workspace retains and what the agent must verify again. Meshia documents its memory behavior so you can plan that boundary.