codeburn vs SafetyBench Eval

Automatically locates and parses session logs from Claude Code, Cursor, GitHub Copilot, Codex, and other AI coding tools by scanning platform-specific directories (~/.claude, ~/.config, etc.). Implements a provider plugin system with standardized parsers that convert heterogeneous log formats into a unified ParsedTurn and Session object model, enabling downstream analysis across multiple tools without manual configuration.

Unique: Implements a provider plugin architecture that decouples provider-specific parsing logic from the core analysis engine, allowing new providers to be added via standardized interfaces (discoverAllSessions, parseSessionFile) without modifying core code. Uses LiteLLM's pricing database as the canonical source for model cost data across 100+ models.

vs alternatives: Supports 5+ AI coding tools natively with a pluggable architecture, whereas most token trackers are single-tool specific or require API proxies that add latency and privacy concerns.

Analyzes parsed session turns and classifies them into TaskCategory buckets (coding, testing, terminal usage, debugging, etc.) using heuristic rules based on turn content, tool invocations, and file types. Implements a classifyTurn function that examines API calls, file modifications, and context patterns to assign semantic meaning to raw token consumption, enabling cost breakdown by activity type rather than just by model.

Unique: Uses multi-signal heuristic classification (file types, tool invocations, context patterns) rather than simple keyword matching, enabling semantic understanding of turn purpose. Tracks one-shot success rate per task category to identify which activity types benefit most from AI assistance.

vs alternatives: Provides task-level cost visibility that generic token counters cannot offer, allowing developers to optimize by activity type rather than just by model or project.

Provides CLI commands (codeburn status, codeburn report) that generate detailed reports on session discovery status, parsing errors, and data quality metrics. Implements metadata inspection capabilities that allow developers to examine individual session files, view parsing errors, and understand data completeness. Generates status summaries showing how many sessions were discovered, parsed successfully, and skipped due to errors.

Unique: Provides transparent visibility into the data ingestion pipeline, showing exactly which sessions were discovered, parsed, and skipped with detailed error messages. Enables developers to audit data quality before relying on cost calculations.

vs alternatives: Offers detailed status and error reporting that helps developers understand data completeness, whereas black-box tools that silently skip sessions make it difficult to detect data quality issues.

Implements a plugin-based architecture that allows new AI coding providers to be added without modifying core CodeBurn code. Each provider plugin implements standardized interfaces (discoverAllSessions, parseSessionFile) that return normalized ParsedTurn and Session objects. Plugins are loaded dynamically at runtime and can be distributed as npm packages, enabling community contributions and custom provider support.

Unique: Defines a minimal, standardized plugin interface (discoverAllSessions, parseSessionFile) that decouples provider-specific logic from the core analysis engine, enabling community contributions without core code changes. Plugins are loaded dynamically at runtime.

vs alternatives: Enables extensibility without forking or modifying core code, whereas monolithic tools that hardcode provider support require core maintainers to add each new provider.

Calculates USD costs for each turn by multiplying token counts (input + output) by model-specific pricing rates sourced from LiteLLM's pricing database, which covers 100+ models across OpenAI, Anthropic, and other providers. Implements a calculateCost function that handles variable pricing tiers, currency conversion, and subscription plan adjustments (e.g., Claude Pro discounts), ensuring accurate financial visibility without requiring API calls to pricing services.

Unique: Integrates LiteLLM's comprehensive pricing database as a built-in data source rather than requiring external API calls, enabling offline cost calculation and eliminating latency. Handles subscription plan adjustments (Claude Pro discounts) and multi-currency support natively.

vs alternatives: Provides accurate, offline cost calculation across 100+ models without API dependencies, whereas most token trackers either hardcode pricing or require cloud lookups that add latency and privacy exposure.

Renders a terminal-based interactive dashboard (TUI) using a framework like Ink or Blessed that displays aggregated token usage, costs, and efficiency metrics across multiple time periods (Today, 7 Days, 30 Days, All Time). Implements keyboard-driven navigation, filtering by project/model/task category, and drill-down capabilities that allow developers to explore cost patterns without leaving the terminal. Updates metrics in real-time as new session data is discovered.

Unique: Implements a keyboard-driven TUI dashboard that runs entirely in the terminal without external dependencies, enabling cost monitoring in headless environments and SSH sessions. Provides drill-down navigation from aggregate metrics to individual turns without context switching.

vs alternatives: Offers a native terminal experience for developers who live in the CLI, whereas web-based dashboards require browser context switching and are inaccessible in SSH/headless environments.

Aggregates parsed session turns into daily buckets and higher-level time periods (7 Days, 30 Days, All Time) using an aggregateProjectsIntoDays function that groups by date, project, and model. Implements a caching layer that stores aggregated results to avoid recomputing statistics on every dashboard load, with cache invalidation triggered by new session data discovery. Supports efficient querying of cost trends across arbitrary time windows.

Unique: Implements a two-level aggregation strategy (daily buckets + period summaries) with intelligent cache invalidation that rebuilds only affected time periods when new sessions are discovered, avoiding full recomputation. Uses immutable daily aggregates as the foundation for all higher-level queries.

vs alternatives: Provides fast metric queries even with large datasets by pre-aggregating and caching, whereas naive approaches that recalculate from raw turns on every query become slow with 1000+ turns.

Scans session history to identify inefficient token usage patterns such as redundant file reads, bloated context windows, unused MCP tool invocations, and low one-shot success rates. Implements an optimization engine (codeburn optimize) that analyzes turn sequences, detects repeated operations on the same files, and generates actionable recommendations to reduce token waste. Uses heuristic rules and statistical analysis to flag anomalies in token consumption.

Unique: Analyzes turn sequences and file access patterns to detect structural inefficiencies (e.g., reading the same file 5 times in a single session) rather than just flagging high token counts. Tracks one-shot success rate as a proxy for efficiency and correlates it with context size and tool usage.

vs alternatives: Provides actionable optimization recommendations based on actual usage patterns, whereas generic cost-cutting advice (e.g., 'use smaller models') ignores the specific inefficiencies in a developer's workflow.

Evaluates LLM safety across 7 distinct categories (offensiveness, unfairness, physical health, mental health, illegal activities, ethics, privacy) using 11,435 curated multiple-choice questions available in both Chinese and English. The benchmark constructs category-specific prompts, sends them to target models, extracts predicted answers from model responses, and compares against ground-truth labels (0->A, 1->B, 2->C, 3->D) to compute accuracy metrics per category and overall safety score.

Unique: Combines 11,435 questions across 7 safety categories with explicit Chinese-English parallel coverage and a filtered subset (test_zh_subset.json) for sensitive keyword handling, enabling systematic cross-lingual safety assessment. Uses category-stratified few-shot examples (5 per category) to support both zero-shot and five-shot evaluation paradigms within a single framework.

vs alternatives: Larger and more category-diverse than single-domain safety benchmarks (e.g., ToxiGen for toxicity only), and explicitly supports Chinese alongside English, addressing a gap in multilingual safety evaluation infrastructure.

Supports two distinct evaluation paradigms: zero-shot (questions presented directly without examples) and five-shot (5 category-specific examples provided before each test question). The framework conditionally constructs prompts using dev_en.json/dev_zh.json few-shot examples or omits them entirely, allowing researchers to measure how in-context learning affects safety performance. Prompt templates are language-aware and can be customized per model to improve answer extraction accuracy.

Unique: Provides curated few-shot examples stratified by safety category (5 per category) rather than random sampling, ensuring balanced representation of each harm type. Prompt templates are explicitly customizable per model (e.g., evaluate_baichuan.py shows Baichuan-specific extraction logic), acknowledging that different architectures require different prompting strategies.

vs alternatives: More systematic than ad-hoc few-shot selection; category-stratified examples ensure consistent coverage of all safety dimensions rather than potentially biased random sampling.

Manages parallel Chinese and English datasets (test_en.json, test_zh.json, dev_en.json, dev_zh.json) with a filtered Chinese subset (test_zh_subset.json, 300 questions per category) for sensitive keyword handling. Data acquisition uses Hugging Face hosting with dual download methods (shell script download_data.sh or Python download_data.py with datasets library). Each question maintains consistent structure (id, category, question, options, answer) across languages, enabling direct cross-lingual comparison of model safety performance.

Unique: Provides both full Chinese dataset (test_zh.json) and a filtered subset (test_zh_subset.json with 300 questions per category) explicitly designed to avoid sensitive keywords, addressing practical concerns about evaluating on content that may trigger platform policies. Dual download methods (shell script and Python) reduce friction for different user workflows.

vs alternatives: More comprehensive multilingual coverage than English-only benchmarks; filtered subset is a pragmatic addition for teams needing to evaluate without policy violations.

Computes accuracy metrics per safety category (offensiveness, unfairness, physical health, mental health, illegal activities, ethics, privacy) and aggregates to an overall safety score. Supports standardized leaderboard submission via JSON format (question_id -> predicted_answer). Metrics are computed by comparing predicted answers (extracted from model responses) against ground-truth labels, enabling fine-grained analysis of which safety dimensions a model excels or fails on. Results can be submitted to llmbench.ai/safety leaderboard for public comparison.

Unique: Stratifies metrics across 7 explicit safety categories rather than computing a single aggregate score, enabling fine-grained diagnosis of safety weaknesses. Leaderboard integration (llmbench.ai/safety) provides public benchmarking infrastructure, creating accountability and enabling direct model comparison.

vs alternatives: Category-level metrics provide more actionable insights than single-number safety scores; leaderboard integration drives standardization and reproducibility across the research community.

Implements a standardized evaluation pipeline (exemplified in evaluate_baichuan.py) that constructs prompts, sends them to a target model via API or local inference, extracts predicted answers from model responses using model-specific parsing logic, and validates extracted answers against expected format (0->A, 1->B, 2->C, 3->D). The pipeline handles model-specific response formats and can be customized per model architecture. Supports batch evaluation of all 11,435 questions with error handling and logging.

Unique: Provides a concrete, model-specific evaluation implementation (evaluate_baichuan.py) that can be forked and adapted, rather than just a dataset. Acknowledges that different models require different answer extraction logic and provides a template for customization. Supports both zero-shot and few-shot evaluation within the same pipeline.

vs alternatives: More practical than dataset-only benchmarks because it includes reference evaluation code; reduces barrier to entry for teams without evaluation infrastructure.

Defines a structured taxonomy of 7 safety categories (offensiveness, unfairness, physical health, mental health, illegal activities, ethics, privacy) and curates 11,435 diverse multiple-choice questions mapped to these categories. Each question is designed to test whether a model correctly handles or refuses harmful content within that category. The taxonomy is explicit and mutually exclusive, enabling fine-grained safety analysis. Questions are curated to be challenging and representative of real-world safety concerns.

Unique: Explicitly defines 7 non-overlapping safety categories and curates 11,435 questions to cover them systematically, providing a structured taxonomy rather than ad-hoc safety testing. The taxonomy is comprehensive enough to cover major harm types (physical, mental, legal, ethical, privacy) while remaining tractable for evaluation.

vs alternatives: More comprehensive and structured than single-category benchmarks (e.g., toxicity-only); provides a holistic safety assessment framework that aligns with regulatory and safety research perspectives.

Provides two download methods for SafetyBench datasets: shell script (download_data.sh) and Python script (download_data.py using Hugging Face datasets library). The architecture leverages Hugging Face Hub for dataset hosting and distribution, enabling one-command dataset acquisition with automatic decompression and directory structure creation. The Python method uses the datasets library for programmatic access, supporting integration into automated evaluation pipelines without manual file management.

Unique: Provides dual download methods (shell script and Python) leveraging Hugging Face Hub for distribution, enabling both manual and programmatic dataset acquisition with automatic decompression and directory structure creation.

vs alternatives: More convenient than manual downloads by providing automated acquisition scripts, and more reproducible than email-based dataset distribution by using Hugging Face Hub as a stable, versioned repository

Computes accuracy metrics stratified by safety category, enabling per-dimension performance analysis. The evaluation pipeline aggregates predictions across all questions in each category (offensiveness, unfairness, physical health, mental health, illegal activities, ethics, privacy) and computes category-specific accuracy scores. This architecture enables identification of category-specific vulnerabilities (e.g., a model may be robust on ethics but weak on physical health) without requiring separate evaluation runs.

Unique: Automatically stratifies accuracy metrics by safety category, enabling fine-grained vulnerability analysis without requiring separate evaluation runs. Provides per-category scores that reveal category-specific weaknesses.

vs alternatives: More diagnostic than aggregate safety scores by breaking down performance by harm category, enabling targeted safety improvements rather than black-box optimization