JARVIS

Uses an LLM controller to analyze user requests, decompose them into subtasks, select appropriate expert models from HuggingFace Hub based on model descriptions, execute those models sequentially or in parallel, and synthesize results into coherent responses. The LLM acts as a central planner and coordinator, maintaining context across all execution stages and making dynamic model selection decisions based on task requirements.

Implements a structured four-stage pipeline where Stage 1 (Task Planning) decomposes user requests into subtasks, Stage 2 (Model Selection) identifies appropriate HuggingFace models, Stage 3 (Task Execution) runs selected models and collects outputs, and Stage 4 (Response Generation) synthesizes results. Each stage produces inspectable intermediate outputs, enabling debugging and partial result retrieval without completing the full pipeline.

Synthesizes final natural language responses by aggregating outputs from multiple executed models. The synthesis stage uses the LLM controller to interpret model predictions, resolve conflicts between models, integrate results into a coherent narrative, and generate human-readable responses. Synthesis is context-aware, incorporating task decomposition and model selection reasoning from earlier stages.

Uses YAML configuration files to specify deployment modes (local/remote/hybrid), local deployment scales (minimal/standard/full), model registry definitions, and inference parameters. Configuration is declarative and version-controllable, enabling reproducible deployments and easy switching between configurations without code changes. Supports environment variable substitution for sensitive credentials.

Queries HuggingFace Hub's model registry to discover available models, retrieves their metadata (descriptions, tags, task types), and uses the LLM controller to match task requirements against model capabilities. Selection is performed by embedding task descriptions and model descriptions in semantic space or via LLM reasoning, enabling dynamic model discovery without hardcoded model lists.

Supports three deployment modes configurable via YAML: Local Mode executes all models on local hardware, HuggingFace Mode uses only remote HuggingFace inference endpoints, and Hybrid Mode mixes local and remote execution. Local deployments offer three scales (minimal, standard, full) with different RAM requirements (12GB, 16GB, 42GB) and model coverage, enabling resource-constrained deployments.

Exposes JARVIS functionality through three interfaces: Server API mode provides HTTP endpoints (/hugginggpt for full service, /tasks for Stage 1 results, /results for Stages 1-3 results), CLI mode offers text-based interaction, and Web UI provides browser-based access. All interfaces share the same underlying four-stage workflow, enabling different user personas to interact with the system.

Provides a benchmark dataset and evaluation framework for measuring LLM performance on task automation and multi-model orchestration. TaskBench includes task instances with ground-truth model selections and expected outputs, enabling quantitative evaluation of JARVIS's task planning, model selection, and execution accuracy. The framework measures both task completion rate and quality of intermediate reasoning steps.

Generates concise, structured tool instructions from model descriptions to improve LLM tool-calling accuracy. EasyTool formats HuggingFace model descriptions into standardized instruction templates that highlight key capabilities, input/output formats, and usage constraints, reducing ambiguity in model selection and improving LLM reasoning about which models to invoke.

Generates synthetic task instances for training and evaluation by sampling from task templates, creating diverse task variations with corresponding ground-truth model selections. The pipeline produces structured datasets with task descriptions, expected subtask decompositions, selected models, and execution results, enabling creation of large-scale benchmarks without manual annotation.

Manages context and state throughout the four-stage inference pipeline, maintaining task descriptions, intermediate results, and model outputs across stages. The inference engine passes context from task planning through model selection and execution to response generation, enabling the LLM to reason about relationships between subtasks and model outputs. Context is managed in-memory with optional serialization for debugging.

Executes selected HuggingFace models with standardized error handling, timeout management, and result collection. The execution engine invokes models via HuggingFace inference APIs or local deployments, captures outputs in a standardized format, handles failures gracefully (timeouts, OOM, API errors), and collects results for synthesis. Supports both synchronous execution and asynchronous batching.