← All insights
RAG · LangGraph · Evaluation

Production RAG and agents — cost, reliability, EU vendors

A notebook that answers three PDFs is not a system. Production RAG and agents need a cost model, an eval harness and a vendor list that does not force your index into a US SaaS.

LMXAI · Leiden, NL · Published · Revised

Nederlandse versie  ·  Türkçe versiyon

Orchestration
LangGraph · MCP
Retrieval
Hybrid · your index
Risk
Cost + silent failure
EU stance
Self-host first

Abstract

Demos of retrieval-augmented generation and agents fail in production for three reasons: no frozen evaluation set, no traces, and no cost model. This article sets out the cost drivers, the reliability gates LMXAI measures before go-live, and a vendor posture that keeps the index and traces inside the EU. Each gate is related to published results on retrieval, evaluation and tool use.

Keywords RAG · agents · evaluation · cost model · tracing · EU vendors

Contents

  1. Background and related work
  2. Why the demo dies
  3. Cost: model the system, not the token price
  4. Reliability: what we measure before go-live
  5. EU-friendly vendor posture
01

Background and related work

RAG improves specificity and factuality on knowledge-intensive tasks by letting a generator draw on a retrievable index[1], but an experience report from three case studies lists seven failure points and concludes that validation is only feasible during operation and that robustness evolves over time[2].

Several measurement results shape the gates below. Models perform best when relevant information is at the start or end of the input and worse when it is in the middle[3]. Reference-free metrics can score faithfulness, answer relevance and context relevance without a hand-written answer for every question[4]. Where a model is used as a judge, strong judges agreed with human preferences more than 80% of the time but showed position, verbosity and self-enhancement biases[5]. For agents, structural checks of function calls show good single-turn performance and persistent trouble with long-horizon tasks[6]. On cost, a cascade of cheaper and more expensive models matched the best single model at up to 98% lower cost in the authors' experiments[7].

02

Why the demo dies

Production RAG is retrieval plus generation with a measured error rate, a budget and an owner. Production agents add tools and multi-step control. Demos die because they skip three things: a frozen evaluation set, traces that show which chunk was used, and a cost model that survives more than a pilot week. LMXAI treats those as launch criteria, not “phase two”.

This is the same discipline behind LMXAI’s agentic patterns and clinical work such as AI dietitian — LangGraph orchestration, MCP tools, and eval (including BFCL-style tool-use checks) before anyone calls it reliable.

03

Cost: model the system, not the token price

  • Embed once, query often. Re-embedding a messy share drive every night will dwarf generation cost.
  • Context is the silent bill. Stuffing 40 chunks into a 32B model is usually worse and more expensive than 6 good ones.[3]
  • Agent steps multiply. A five-tool loop can be 5–15 model calls. Cap steps, cache tool results, fail closed.
  • Idle GPUs vs API invoices. Self-hosted vLLM wins at steady internal load; APIs win at spiky experiments. Mixed is normal.
04

Reliability: what we measure before go-live

Gate What “good” looks like
Retrieval hit rate Gold questions retrieve the intended doc in top-k
Faithfulness Answer is supported by retrieved text, or the model abstains
Tool success Function calls match schema; no invented arguments
Latency SLO p95 time-to-first-token and end-to-end agreed with the business
Trace completeness Every answer has query, chunks, model version, tool I/O
Fallback Empty retrieval → “I don’t know” or a human queue, not a hallucination
05

EU-friendly vendor posture

Prefer components you can run in your cluster: Elasticsearch or OpenSearch, Qdrant, or pgvector when the corpus is small; vLLM for generation; LangGraph for control flow; OpenTelemetry plus Phoenix or an equivalent for traces. Managed EU offerings are fine when the checklist in the vLLM vendor page passes — especially no silent training on your prompts and a contract under EU law.

Treat US-hosted “agent platforms” as prototypes unless procurement has accepted the data path. The product surface is rarely the problem. The index and the traces are.

LMXAI default: hybrid retrieval you own, an agent graph with hard step limits, a gateway that meters tokens per team, and an eval suite that runs in CI when prompts or chunking change. Cost and reliability are the same engineering problem.

References

  1. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). arxiv.org/abs/2005.11401
  2. Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., & Abdelrazek, M. (2024). Seven failure points when engineering a retrieval augmented generation system. Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering – Software Engineering for AI (CAIN 2024), 194–199. doi:10.1145/3644815.3644945 arxiv.org/abs/2401.05856
  3. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12. aclanthology.org/2024.tacl-1.9/
  4. Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. Proceedings of EACL 2024: System Demonstrations. aclanthology.org/2024.eacl-demo.16/
  5. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track. arxiv.org/abs/2306.05685
  6. Patil, S. G., Mao, H., Yan, F., Ji, C. C.-J., Suresh, V., Stoica, I., & Gonzalez, J. E. (2025). The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models. Proceedings of the 42nd International Conference on Machine Learning (ICML 2025), PMLR 267. proceedings.mlr.press/v267/patil25a.html
  7. Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176. arxiv.org/abs/2305.05176

Each reference was checked against its publisher, arXiv or official record.

How to cite

LMXAI. (2026). Production RAG and agents — cost, reliability, EU vendors. LMXAI Insights. https://lmxai.com/insights/production-rag-agents.html