Production RAG and agents — cost, reliability, EU vendors

Why the demo dies

Production RAG is retrieval plus generation with a measured error rate, a budget and an owner. Production agents add tools and multi-step control. Demos die because they skip three things: a frozen evaluation set, traces that show which chunk was used, and a cost model that survives more than a pilot week. LMXAI treats those as launch criteria, not “phase two”.

This is the same discipline behind LMXAI’s agentic patterns and clinical work such as Savion — LangGraph orchestration, MCP tools, and eval (including BFCL-style tool-use checks) before anyone calls it reliable.

Cost: model the system, not the token price

• Embed once, query often. Re-embedding a messy share drive every night will dwarf generation cost.
• Context is the silent bill. Stuffing 40 chunks into a 32B model is usually worse and more expensive than 6 good ones.
• Agent steps multiply. A five-tool loop can be 5–15 model calls. Cap steps, cache tool results, fail closed.
• Idle GPUs vs API invoices. Self-hosted vLLM wins at steady internal load; APIs win at spiky experiments. Mixed is normal.

Reliability: what we measure before go-live

Gate	What “good” looks like
Retrieval hit rate	Gold questions retrieve the intended doc in top-k
Faithfulness	Answer is supported by retrieved text, or the model abstains
Tool success	Function calls match schema; no invented arguments
Latency SLO	p95 time-to-first-token and end-to-end agreed with the business
Trace completeness	Every answer has query, chunks, model version, tool I/O
Fallback	Empty retrieval → “I don’t know” or a human queue, not a hallucination

EU-friendly vendor posture

Prefer components you can run in your cluster: Elasticsearch or OpenSearch, Qdrant, or pgvector when the corpus is small; vLLM for generation; LangGraph for control flow; OpenTelemetry plus Phoenix or an equivalent for traces. Managed EU offerings are fine when the checklist in the vLLM vendor page passes — especially no silent training on your prompts and a contract under EU law.

Treat US-hosted “agent platforms” as prototypes unless procurement has accepted the data path. The product surface is rarely the problem. The index and the traces are.