Multi Provider Judge Model Integration With Decoder Registry

via “multi-provider judge model integration with decoder registry”

Automatic LLM evaluation — instruction-following, LLM-as-judge, length-controlled, cost-effective.

Unique: Implements a pluggable Decoder registry pattern that unifies OpenAI, Anthropic, Hugging Face, vLLM, and Ollama under a single interface, with built-in caching and retry logic. The decoder abstraction allows swapping judge models without changing evaluation logic, and supports both cloud APIs and local inference in the same framework.

More flexible than single-provider benchmarks (e.g., LMSys Chatbot Arena which uses only GPT-4); cheaper than cloud-only solutions by supporting local open-source judges