Skip to content
All case studies

Case study 042025

LLM Support Copilot

A retrieval-augmented assistant that drafts cited answers for support agents, gated by evals in CI.

Role
Lead engineer
Team
3 engineers
Timeline
5 months
Focus
AI, Backend, Frontend

−35%

median handle time

94%

answers with valid citations

−48%

cost per resolved ticket

Every draft is built only from reranked passages; a golden-set eval suite scores each change before it can merge.Agent UI to Copilot API (ticket); Copilot API to Hybrid retrieval (query); Hybrid retrieval to Reranker; Copilot API to Model router (grounded prompt); Model router to Eval suite (scored in CI)Agent UINext.js · streamingCopilot APIFastAPIHybrid retrievalBM25 + pgvectorRerankertop-k passagesModel routersmall / largeEval suite600 cases · CI gateticketquerygrounded promptscored in CI
Every draft is built only from reranked passages; a golden-set eval suite scores each change before it can merge.

The problem

Handle times were climbing, and the first LLM prototype sounded confident while citing nothing. Leadership wanted AI in the support flow, and the support team wanted nothing to do with answers it couldn't verify.

What I owned

  • Retrieval: hybrid BM25 and vector search with a reranker.
  • The golden dataset built from 600 historical tickets, and the eval harness that gates merges.
  • The streaming agent UI, where every accept, edit or reject feeds the eval set.
  • Cost and latency tracking, plus routing simple intents to a smaller model.

Key decisions

  1. 1

    Grounded or silent

    The model only answers from retrieved passages and must cite them. If retrieval comes back weak, the copilot says so instead of guessing.

  2. 2

    Evals as a merge gate

    Prompt and model changes run against the golden set in CI. A drop in accuracy or citation coverage fails the build like any other test.

  3. 3

    Route by intent

    Most tickets are simple. A classifier sends them to a smaller, cheaper model and reserves the large model for hard cases.

Outcome

The copilot is part of the everyday support flow. Agents trust it because they can check every claim, and the eval suite now guards two other teams' LLM features too.

  • Python
  • FastAPI
  • pgvector
  • Next.js
  • LLM APIs
  • Evals