Corrective Rag With Automatic Retrieval Quality Assessment

via “evaluation and benchmarking of rag pipelines”

<p align="center"> <img height="100" width="100" alt="LlamaIndex logo" src="https://ts.llamaindex.ai/square.svg" /> </p> <h1 align="center">LlamaIndex.TS</h1> <h3 align="center"> Data framework for your LLM application. </h3>

Unique: Provides RAG-specific evaluation metrics (retrieval precision/recall, answer relevance) alongside standard NLP metrics, with integration to external evaluation services and built-in regression detection

vs others: More comprehensive than LangChain's evaluation tools because it includes RAG-specific metrics (not just generation metrics) and supports integration with specialized RAG evaluation frameworks like Ragas

via “llm-based rag evaluation with multi-metric synthesis”

RAG evaluation framework — faithfulness, relevancy, context precision/recall metrics.

Unique: Combines PydanticPrompt-based structured output extraction with Instructor adapter pattern for reliable LLM metric scoring, paired with async Executor pattern for efficient batch evaluation. Requires only questions and answers (not full retrieval traces), making it applicable to existing RAG systems without instrumentation changes.

vs others: More practical than human evaluation (no annotation cost) and more interpretable than black-box ML-based metrics because each score is tied to explicit LLM reasoning via prompts.

via “rag system component-level evaluation with automated test generation”

AI testing for quality, safety, compliance — vulnerability scanning, bias/toxicity detection.

Unique: Decomposes RAG systems into independently evaluable components (Retriever, Generator, Rewriter, Router) rather than treating them as black boxes, enabling root-cause analysis of performance degradation. Automatically generates diverse question types from knowledge bases using LLM-based generation rather than requiring manual test curation.

vs others: More granular than generic LLM evaluation frameworks like LangSmith because it provides component-level metrics and automatic test generation specific to RAG architectures, rather than generic output comparison.

via “feedback-loop-for-rag-quality-improvement”

AI-powered internal knowledge base dashboard template.

Unique: Integrates feedback collection directly into the chat and search UIs with minimal friction (single-click ratings). Automatically correlates feedback with RAG configuration (model, chunk size, prompt) to identify which changes improve quality.

vs others: More actionable than generic user satisfaction surveys because it captures feedback in context; more efficient than manual quality audits because it scales to thousands of interactions.

via “rag health diagnostics and retrieval quality monitoring”

Enterprise AI observability with explainability and fairness for regulated industries.

Unique: Fiddler's RAG diagnostics integrate retrieval quality monitoring with answer grounding analysis and LLM-as-a-Judge evaluation, providing end-to-end RAG pipeline visibility — differentiating from retrieval-only monitoring tools by connecting retrieval quality to answer quality and hallucination detection

vs others: More comprehensive than retrieval-only monitoring because it analyzes both retrieval quality and answer grounding, enabling detection of failures at multiple points in the RAG pipeline (bad retrieval, good retrieval but poor grounding, etc.)

via “context adherence scoring for rag systems”

AI evaluation platform with automated hallucination detection and RAG metrics.

Unique: Treats context adherence as a first-class observability metric integrated into production monitoring dashboards rather than a batch evaluation metric, enabling real-time detection of when retrieval quality degrades and impacts answer grounding

vs others: Provides context-specific grounding metrics whereas generic LLM evaluation platforms like Weights & Biases focus on output quality without measuring retrieval utilization

via “evaluation framework for rag quality metrics”

LangChain reference RAG implementation from scratch.

Unique: Demonstrates multi-dimensional evaluation covering retrieval quality (precision, recall, NDCG), generation quality (BLEU, ROUGE, semantic similarity), and end-to-end correctness, enabling developers to identify bottlenecks (e.g., poor retrieval vs. poor generation) and optimize accordingly.

vs others: More comprehensive than single-metric evaluation because it measures retrieval, generation, and end-to-end quality separately; more practical than manual evaluation because automated metrics enable rapid iteration and regression detection.

via “evaluation and benchmarking of rag pipeline quality”

LlamaIndex starter pack for common RAG use cases.

Unique: LlamaIndex's evaluation framework integrates retrieval and generation metrics in a single pipeline, enabling end-to-end quality assessment, whereas most RAG systems require separate evaluation tools for retrieval and generation

vs others: More comprehensive than generic NLG evaluation because LlamaIndex's metrics include retrieval-specific measures (precision, recall) alongside generation metrics, providing holistic RAG quality assessment

via “corrective and hybrid rag with relevance grading and multi-strategy retrieval”

100+ AI Agent & RAG apps you can actually run — clone, customize, ship.

Unique: Provides implementations of corrective RAG (with relevance grading and query reformulation) and hybrid RAG (combining vector and keyword search) with explicit trade-offs between quality and latency. Demonstrates how to define and implement relevance criteria. Most RAG tutorials show only basic vector search; this library treats quality improvement as a design pattern.

vs others: More sophisticated than basic RAG but with documented latency costs; more practical than academic RAG papers with working code

via “self-correcting-rag-with-answer-validation”

This repository showcases various advanced techniques for Retrieval-Augmented Generation (RAG) systems. Each technique has a detailed notebook tutorial.

Unique: Implements Self-RAG and CRAG techniques that validate generated answers against retrieved context and trigger self-correction (re-retrieval and regeneration) if validation fails, creating an internal feedback loop that detects and corrects hallucinations without external validators

vs others: More proactive than post-hoc fact-checking because it validates during generation and corrects immediately, and more practical than requiring external validators because it uses the LLM itself for validation

via “evaluation and metrics tracking for rag quality”

Unified framework for building enterprise RAG pipelines with small, specialized models

Unique: Built-in evaluation utilities for measuring RAG quality (retrieval precision/recall, answer relevance) with automatic prompt-response logging and source attribution tracking. Integrates with external evaluation frameworks (RAGAS, DeepEval) for standardized metrics, enabling systematic RAG optimization.

vs others: Integrated evaluation vs external frameworks; automatic prompt-response logging for compliance vs manual tracking; built-in source attribution metrics vs generic LLM evaluation tools.

via “evaluation framework for rag quality assessment and benchmarking”

Open-source LLM knowledge platform: turn raw documents into a queryable RAG, an autonomous reasoning agent, and a self-maintaining Wiki.

Unique: Integrates evaluation as a built-in capability, allowing RAG quality to be measured and tracked over time. Supports comparing multiple configurations and storing historical results.

vs others: More systematic than manual testing (automated metrics), more comprehensive than single-metric evaluation (multiple metrics), and more actionable than offline metrics (enables configuration comparison).

via “retrieval re-ranking with cross-encoder models and crag”

Everything you need to know to build your own RAG application

Unique: Combines cross-encoder re-ranking with Corrective RAG (CRAG) using LangGraph state machines, enabling iterative retrieval refinement with explicit quality validation rather than single-pass retrieval

vs others: More effective than embedding-only ranking for complex queries, and more robust than static retrieval because CRAG detects and corrects failures automatically

In-depth tutorials on LLMs, RAGs and real-world AI agent applications.

Unique: Implements automatic quality feedback loops using LLM-based relevance scoring rather than static retrieval pipelines, enabling dynamic strategy adjustment without manual intervention or threshold tuning

vs others: More robust than single-pass retrieval because it detects and corrects failures automatically; faster than exhaustive multi-strategy retrieval because it only applies corrections when needed based on quality assessment

via “evaluation and metrics for rag quality”

A data framework for building LLM applications over external data.

Unique: Provides a unified evaluation framework with multiple metric types (retrieval, generation, end-to-end) and support for both automated and human evaluation. Integrates with evaluation datasets and enables systematic quality tracking without custom metric implementation.

vs others: More comprehensive evaluation coverage than ad-hoc metric scripts; built-in integration with evaluation datasets and benchmarks reduces setup time for quality assessment.

via “corrective agentic rag with feedback-driven iterative refinement”

Agentic-RAG explores advanced Retrieval-Augmented Generation systems enhanced with AI LLM agents.

Unique: Implements error correction as an autonomous capability where agents detect failures and trigger corrective actions without external feedback, rather than treating errors as terminal failures, enabling self-improving systems that adapt retrieval and generation strategies based on quality feedback.

vs others: More autonomous than systems requiring human feedback by implementing automatic error detection and correction, and more adaptive than fixed retrieval strategies by adjusting approach based on detected failures.

via “evaluation framework for rag and qa systems”

LLM framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data.

Unique: Integrated evaluation framework supporting retrieval metrics (NDCG, MRR, precision@k), generation metrics (BLEU, ROUGE, semantic similarity), and custom evaluators — enabling quantitative RAG system assessment without external tools

vs others: More RAG-specific than generic ML evaluation frameworks; simpler than building custom evaluation pipelines

via “rag quality evaluation framework with retrieval metrics”

[EMNLP2025] "LightRAG: Simple and Fast Retrieval-Augmented Generation"

Unique: Provides a built-in evaluation framework with ground-truth comparison and synthetic dataset generation, enabling measurement of retrieval quality without external evaluation tools. Integrates with the RAG pipeline to measure quality improvements as documents are added.

vs others: More integrated than external evaluation tools; enables in-system quality measurement and tracking, though less comprehensive than dedicated RAG evaluation platforms.

via “evaluation and metrics collection for rag quality”

Retrieval Augmented Generation (RAG) support for NestJS AI

Unique: Implements RAG evaluation as NestJS services with pluggable evaluation strategies (ground truth, LLM-as-judge, human feedback) and metrics collection, allowing systematic measurement and comparison of retrieval and generation quality

vs others: More comprehensive than ad-hoc logging — provides structured evaluation framework with support for multiple evaluation strategies and A/B testing, rather than requiring manual metrics implementation

via “multi-metric rag evaluation with llm-as-judge scoring”

Evaluation framework for RAG and LLM applications

Unique: Implements domain-specific metrics (faithfulness, answer relevance, context precision) designed for RAG evaluation rather than generic NLG metrics; uses LLM-as-judge pattern with configurable judge models, enabling evaluation without human annotation while maintaining interpretability through metric-specific prompting strategies

vs others: More specialized for RAG than generic LLM evaluation frameworks (like DeepEval or LangSmith), with metrics specifically designed to catch retrieval failures and hallucinations in context-grounded generation tasks