UpTrain

UpTrain is an Apache 2.0 Python evaluation framework built by Sourabh Agrawal, Shikha Mohanty, and Vipul Gupta, launched through Y Combinator's W23 batch. The framework covers 20+ preconfigured evaluation checks with a diagnostic layer that identifies whether a failure originates from retrieval quality, context reranking, context utilization, or instruction-following — a distinction most evaluation tools leave to manual inspection. Developers should note that the founding team has largely shifted focus to a separate YC company, CombineHealth, and UpTrain currently operates with three employees. The repository received a v0.7.1 release on May 14, 2026, confirming the project remains functional, though active feature development has slowed significantly.

Key Capabilities

• Root cause analysis for RAG failures: Beyond returning a score, UpTrain diagnoses which pipeline component produced a failure, distinguishing between retrieval gaps, reranking problems, poor context utilization, and instruction misalignment

• Self-hosted Docker dashboard: A no-code web interface runs locally via bash run_uptrain.sh with no cloud dependency, suited for teams that require evaluation data to stay within their own infrastructure

• 20+ preconfigured evaluation checks: Pre-built checks span language quality, code correctness, and embedding-based use cases, alongside support for custom metrics through an extendable framework

• Classical NLP and LLM-based scoring: Metrics run through both LLM-as-judge and classical NLP methods, enabling cost-controlled evaluation without requiring frontier API calls for every check

• Vector database integrations: Direct integrations with Qdrant, ChromaDB, and FAISS allow retrieval quality evaluation against the actual vector stores powering a RAG pipeline

• Automated regression testing with prompt versioning: Tests run automatically on prompt or configuration changes, with versioned prompt snapshots that support rollback when regressions are detected