DEV Community

Manohar Papasani
Manohar Papasani

Posted on

You are seeing XYZ issue in your ai responses, walk me through what you change in your process/pipeline to resolve it.

#aiinterview #genai #llm #rag

Asked in a Gen AI engineer interview.

What they are really testing:

Whether you diagnose before changing anything, and whether you can locate the fault in retrieval, context assembly, prompting, or the model itself.

A strong answer:

  • Reproduce and classify the failure first. Collect a small set of failing queries and tag each one: retrieval miss, correct chunk ignored, stale or wrong source, instruction or format failure, or reasoning error. Each tag points to a different fix.
  • Trace the full pipeline. Log the query rewrite, retrieved chunks with scores, reranker output, the final assembled prompt, and the raw completion. Without this, every change is a guess.
  • If retrieval is the problem, fix it before touching the model: revisit chunk boundaries (structure-aware splits, roughly 300 to 500 tokens with 10 to 20 percent overlap as a starting point), add metadata filters, combine BM25 with dense retrieval, and add a cross-encoder reranker over a wider candidate set (for example 20 narrowed to 4 to 6).
  • If generation is the problem, tighten the context contract: require answers grounded in the supplied passages with citations, include only the context the question needs, and provide an explicit "not found" path so the model abstains instead of filling gaps.
  • Close the loop with evaluation. Measure retrieval recall at k and groundedness separately on a fixed regression set, so a fix for one failure does not quietly break another. Validate any LLM judge against human labels before trusting it.
  • Roll the change out behind a flag or canary, and monitor the failure tags in production so the issue cannot silently return.

Common mistakes:

  • Reaching for a larger model or fine-tuning before confirming whether retrieval returned the right evidence.
  • Changing chunking, prompts, and the reranker all at once, so you cannot tell which change fixed the problem.
  • Evaluating on a handful of hand-picked examples that happen to pass.

How would you split this issue between retrieval and generation? Share your approach below.

Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (0)

⌂ 换网址