Why does my LLM give wrong answers?

Check in this order

1. 01

   Collect the failing cases into an eval set

   Layer 09 · Evaluation & Observability

   Twenty real failures tell you more than a thousand invented ones, and every fix below needs them to prove it worked.

   • promptfoo

     Use when

     Declarative red-teaming and regression tests that run in CI.

     Skip when

     You need a managed UI rather than a test runner.

   • DeepEval

     Use when

     pytest-style evaluation you can run next to your unit tests.

     Skip when

     You need a platform rather than a library.

   • Braintrust

     Use when

     Making evaluation, rather than tracing, the primary workflow.

     Skip when

     You need open formats and self-hosting.

2. 02

   Read the retrieved context for each failure

   Layer 03 · Retrieval & Vector Stores

   If the answer was not in the context, the ranking is wrong, not the model. Reranking is the cheapest fix with the largest effect.

   • Rerankers

     Use when

     A cross-encoder in the second retrieval stage, to lift precision cheaply.

     Skip when

     You have no way to measure whether recall actually improved.

3. 03

   Check parsing and chunking of the source documents

   Layer 03 · Retrieval & Vector Stores

   Tables flattened into prose and chunks cut mid-sentence produce context that looks relevant and answers nothing.

   • Docling

     Use when

     Layout-aware parsing that preserves tables and heading structure.

     Skip when

     Your documents are plain text.

   • Unstructured

     Use when

     Partitioning messy documents into retrievable chunks before indexing.

     Skip when

     Your input is already structured.

4. 04

   Fix the prompt: grounding, abstention, output schema

   Layer 08 · Prompt Engineering

   When the fact was present and misused, instructions and examples are cheap to change and cheap to revert.

   • DSPy

     Use when

     Compiling prompts into optimised programs from labelled examples.

     Skip when

     You have no examples and no way to score them.

   • Instructor

     Use when

     Schema-validated extraction with typed retry and partial streaming.

     Skip when

     You need a hard token-level guarantee rather than validation.

5. 05

   Fine-tune last, for narrow and stable behaviour

   Layer 04 · Fine-tuning & Training

   Fine-tuning teaches behaviour, not facts, and leaves you a new model to serve and retrain. Use it when prompting has measurably plateaued.

   • Unsloth

     Use when

     LoRA fine-tuning on a single GPU, with memory and time roughly halved.

     Skip when

     You need a training stack you can audit line by line.

   • Axolotl

     Use when

     Configuration-driven fine-tuning across many architectures.

     Skip when

     You need to hand-tune the training loop itself.

Looks like a fix, is not

• ×Rewriting the prompt when the retrieved context never contained the answer.
• ×Fine-tuning to teach facts — anything that changes monthly belongs in retrieval.
• ×Moving to a larger model before checking what context the current one was given.

Quick answers

What should I check first?

Collect the failing cases into an eval set. Twenty real failures tell you more than a thousand invented ones, and every fix below needs them to prove it worked.

What looks like a fix but is not?

Rewriting the prompt when the retrieved context never contained the answer. Fine-tuning to teach facts — anything that changes monthly belongs in retrieval. Moving to a larger model before checking what context the current one was given.