chinese-llm-benchmark

Evaluates Chinese LLMs across 8 major domains (Medical, Education, Finance, Law, Administrative Affairs, Psychological Health, Reasoning & Math, Language & Instruction Following) using approximately 300 specific evaluation dimensions. Each domain assessment aggregates task-specific scores (1-5 scale per question) normalized to 0-100 point scale, then combines domain scores to produce overall model rankings. The framework uses domain-specific test questions designed to measure real-world capability rather than general language understanding.

Organizes 298 evaluated models into hierarchical leaderboards using primary classification (commercial vs open-source) and secondary tiers (price tier for commercial models, parameter size for open-source models). The system maintains separate ranked lists for each category, enabling users to compare models within similar cost/capability profiles. Leaderboard data is stored in markdown files (commerce2.md, reasonmodel.md, alldata.md) with model metadata (name, version, provider, parameters, pricing) and performance scores aggregated from domain evaluations.

Maintains comprehensive metadata for 298+ evaluated models including name, version, provider/developer organization, model type (commercial/open-source), parameter count, pricing information, release date, and availability status. Metadata is stored alongside evaluation scores in leaderboard files and enables filtering, sorting, and comparison based on model attributes. The system tracks model evolution (versions, updates) and maintains historical metadata for deprecated or superseded models.

Maintains a defect library containing over 2 million documented model errors collected during evaluation across all domains and models. The system indexes failures by model, domain, question type, and error category, enabling researchers to identify systematic failure patterns. Defect records link specific model errors to evaluation questions, domain context, and error classification, supporting root-cause analysis and model improvement research. The library serves as a queryable knowledge base for understanding model weaknesses rather than just performance scores.

Implements specialized evaluation for Chinese language understanding and instruction following, including Gaokao (Chinese college entrance exam) level questions that test reading comprehension, writing quality, and complex reasoning in Chinese. The evaluation framework includes domain-specific language tasks (medical terminology understanding, legal document interpretation, financial report analysis) alongside general Chinese language proficiency assessment. Scoring incorporates both accuracy and response quality (1-5 scale) to capture nuanced language performance beyond binary correctness.

Evaluates models on mathematical computation, logical reasoning, and complex problem-solving through domain-specific test questions in the 'Reasoning & Math' category. The evaluation framework assesses both correctness of final answers and quality of reasoning steps (1-5 scale), capturing partial credit for correct methodology with computational errors. Supports multi-step reasoning problems, symbolic manipulation, and logical inference tasks designed to test mathematical capability beyond simple arithmetic.

Implements specialized evaluation across four professional domains (Medical, Finance, Law, Administrative Affairs) with domain-expert-designed test questions requiring specialized knowledge and reasoning. Each domain assessment uses realistic scenarios (medical case studies, financial analysis problems, legal document interpretation, administrative policy questions) to evaluate practical professional capability rather than general knowledge. Scoring incorporates domain-specific rubrics reflecting professional standards and best practices in each field.

Evaluates models on psychological health concepts, mental health counseling knowledge, and psychological reasoning through specialized test questions in the 'Psychological Health' domain. Assessment covers mental health terminology, therapeutic approaches, psychological assessment, and ethical counseling practices. Scoring incorporates both knowledge accuracy and quality of psychological reasoning (1-5 scale) to evaluate capability for mental health support applications.

Implements continuous evaluation pipeline that regularly re-evaluates models and updates leaderboards with new results, maintaining 'Really Reliable Live Evaluation' (ReLE) as the system name indicates. The pipeline processes new model versions, newly released models, and periodic re-evaluation of existing models to keep rankings current. Updates are published to markdown leaderboard files (commerce2.md, reasonmodel.md, alldata.md) enabling version-controlled tracking of ranking changes over time.

Enables direct comparison between commercial models (ChatGPT, Claude, Gemini, Qwen, etc.) and open-source models (DeepSeek, Llama, Phi, etc.) by organizing leaderboards with separate commercial and open-source tiers. Commercial models are further categorized by pricing tier (e.g., ultra-cheap, standard, premium), while open-source models are categorized by parameter size (7B, 13B, 70B, etc.). This structure enables price-performance analysis comparing commercial API costs against open-source deployment costs.

Identifies and separately ranks models with specialized reasoning capabilities (e.g., DeepSeek-R1, o1-mini, reasoning-optimized variants) in dedicated leaderboard (reasonmodel.md). These models are evaluated on the same domain tasks but ranked separately to highlight their specialized reasoning strengths. The system recognizes that reasoning-specialized models may have different performance profiles (stronger on math/logic, potentially weaker on general knowledge) and enables comparison within the reasoning-specialist category.