Local Inference With Variable Model Sizes

via “scalable multi-size model family with configurable context windows”

IBM's enterprise-focused open foundation models.

Unique: Unified architecture across four parameter sizes (3B-34B) with consistent tokenization and training methodology, enabling zero-retraining model swapping. Each size variant is available with multiple context window options (2K, 4K, 8K), allowing fine-grained hardware/latency optimization without model retraining.

vs others: More granular size options than Codex (which has fewer variants) and more flexible context windows than fixed-context models; allows organizations to optimize for specific hardware constraints and latency requirements without sacrificing model consistency.

via “dynamic shape inference and handling for variable-length inputs”

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

Unique: Implements shape inference logic that propagates dynamic shapes through the graph, enabling inference with variable-length inputs without recompilation. The shape inference engine handles both static and dynamic dimensions, adapting to input variations at runtime.

vs others: Provides more flexible dynamic shape support than TensorFlow's static graph model and better shape inference than ONNX Runtime's limited dynamic shape support.

via “local-inference-with-variable-model-sizes”

LLaVA — vision-language model combining CLIP and Vicuna — vision-capable

Unique: Offers three distinct model sizes (7B/13B/34B) distributed through Ollama's unified runtime, enabling hardware-aware deployment choices; 7B variant provides 32K context window (8x larger than 13B/34B) despite smaller parameter count, optimizing for conversation length over reasoning depth

vs others: Eliminates cloud API dependencies and costs compared to GPT-4V or Claude Vision; provides granular hardware-to-model-size matching (7B for consumer GPUs, 34B for enterprise) unlike single-size cloud models

via “model size flexibility with parameter-matched performance tiers”

Meta's Llama 3.1 — high-quality text generation and reasoning

Unique: All three parameter sizes (8B, 70B, 405B) share identical 128K context window and API interface, enabling zero-code-change model swapping. Developers can optimize for latency (8B on consumer hardware) or quality (405B on enterprise hardware) without refactoring.

vs others: More flexible than single-size models (GPT-4, Claude 3.5 Sonnet) which force one-size-fits-all trade-offs. Comparable to OpenAI's GPT-4 Turbo vs. GPT-4o mini, but with full control over model selection and local deployment options.

via “multi-size model variants for performance-efficiency tradeoffs”

* ⏫ 09/2023: [RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (RLAIF)](https://arxiv.org/abs/2309.00267)

Unique: Provides four distinct parameter sizes (7B, 13B, 34B, 70B) with differentiated capabilities (infilling available only in 7B, 13B, 70B), enabling explicit performance-accuracy tradeoffs

vs others: Multiple size options enable deployment across hardware spectrum from edge devices (7B) to high-end servers (70B), offering more flexibility than single-size models like GPT-3.5 or single-size open models

via “local-inference-with-variable-model-sizes-0-5b-to-32b”

Alibaba's Qwen 2.5 specialized for code generation and understanding — code-specialized

Unique: Six model size options (0.5B-32B) enable fine-grained hardware/quality trade-offs without requiring separate model families. All variants share the same 32K context window and instruction-tuning approach, ensuring consistent behavior across sizes despite quality differences.

vs others: More flexible than single-size models (e.g., Mistral 7B) because users can choose appropriate size for their hardware, and more cost-effective than cloud APIs because inference runs locally without per-token charges.

via “model variant selection with accuracy-latency tradeoffs”

Robust Speech Recognition via Large-Scale Weak Supervision

Unique: Unified model family with consistent API across all sizes, allowing single codebase to target devices from smartphones (tiny) to servers (large) without architecture changes. Weak supervision training enables smaller models to maintain reasonable accuracy without task-specific fine-tuning.

vs others: More flexible than fixed-size competitors (Google Cloud offers only one model); smaller models outperform language-specific open-source alternatives like DeepSpeech due to better training data, though larger models are slower than commercial APIs on CPU.

via “lightweight 7b and 13b parameter model variants for hardware-constrained deployment”

BakLLaVA — lightweight vision-language model — vision-capable

Unique: BakLLaVA's 7B variant achieves multimodal reasoning in 4.7GB, significantly smaller than LLaVA 13B or larger VLMs, enabling deployment on consumer GPUs and edge devices where larger models are infeasible.

vs others: More memory-efficient than LLaVA 13B or Qwen-VL for edge deployment, but likely less accurate on complex visual reasoning tasks compared to larger open-source models or proprietary APIs like GPT-4V.

via “model variant selection across parameter sizes (3b, 7b, 13b, 70b)”

Orca Mini — compact instruction-following model

Unique: Provides four model variants with different parameter counts under a single model family name, enabling users to select size via model tag (e.g., `orca-mini:7b`) without managing separate model names or configurations

vs others: More flexible than single-size models (Llama 2 Chat 7B only) and easier to switch between sizes than downloading separate models, but lacks guidance on variant selection vs commercial APIs with automatic model selection

via “multi-model size selection with speed-capability tradeoff”

Unique: Provides explicit model size selection across a 160x parameter range (125M to 20B) with transparent per-token pricing for each tier, enabling developers to optimize for specific latency/cost/quality targets without vendor lock-in to a single model

vs others: More granular model selection than OpenAI (which offers only GPT-3.5/4 variants) but less diverse than open-source model hubs; pricing advantage strongest on smaller models, eroding on 20B tier