ai-agents-from-scratch

Executes quantized GGUF language models locally using node-llama-cpp bindings to the llama.cpp C++ runtime, with platform-specific acceleration (Metal on macOS, CUDA/Vulkan on Linux/Windows). Models run entirely on-device without cloud API calls, enabling privacy-preserving inference with configurable temperature, token limits, and streaming output. The architecture abstracts the underlying C++ runtime through JavaScript bindings, handling model loading, memory management, and token generation.

Implements structured function calling by embedding tool schemas in system prompts and parsing LLM-generated function calls from text output. The architecture defines tools as JavaScript objects with name, description, and parameters, then instructs the LLM to output function calls in a parseable format (typically JSON or XML). A tool execution framework intercepts these outputs, validates them against the schema, and executes the corresponding JavaScript functions, returning results back to the LLM for further reasoning.

Enables switching between local LLMs (via node-llama-cpp) and cloud APIs (OpenAI, Anthropic) through a unified interface, allowing developers to compare quality/speed tradeoffs or fall back to cloud when local inference is insufficient. The architecture abstracts the model backend behind a common interface, with conditional logic to route requests to either local or cloud providers based on configuration. This pattern allows the same agent code to work with different model sources without modification.

Structures agent development as a nine-module learning progression, where each module introduces exactly one new concept (basic LLM interaction → function calling → memory → ReAct). The architecture uses consistent module structure (executable .js file, detailed CODE.md walkthrough, conceptual CONCEPT.md explanation) to enable self-paced learning with multiple entry points. Each module builds on previous ones, creating a scaffolded learning experience from fundamentals to autonomous agents.

Maintains conversation state by storing message history (user and assistant messages) in memory or persistent storage, then including the full or windowed history in each LLM prompt. The architecture uses a message buffer that tracks role (user/assistant), content, and optionally metadata (timestamps, tool calls). Between turns, the system appends new user messages and LLM responses to this buffer, then passes the entire history to the LLM context window, enabling multi-turn reasoning and context awareness.

Implements the ReAct (Reasoning + Acting) pattern by orchestrating a loop where the LLM reasons about the next step, decides whether to call a tool or return a final answer, executes the tool if needed, and incorporates the result back into the conversation history. The architecture maintains a reasoning trace (visible to the LLM) that shows thought processes, tool calls, and observations, enabling the agent to self-correct and refine its approach iteratively. Each loop iteration appends the LLM's reasoning and tool results to the message history, creating a transparent audit trail.

Streams LLM output tokens in real-time using async iterators, allowing applications to display partial responses as they are generated rather than waiting for the full completion. The architecture uses node-llama-cpp's streaming API to yield tokens as they are produced by the inference engine, enabling progressive rendering, early stopping, and responsive user interfaces. Each token is yielded individually, allowing callers to accumulate them into a full response or process them incrementally.

Adapts LLM behavior by injecting task-specific system prompts that define role, constraints, output format, and reasoning style. The architecture treats system prompts as the primary control mechanism for agent specialization, allowing different prompts to transform the same base model into different specialized agents (translator, reasoner, code generator, etc.). System prompts are prepended to the message history and remain constant across conversation turns, establishing the agent's persona and operational guidelines.

Processes multiple independent requests concurrently using Promise.all() or similar patterns, allowing multiple inference tasks to run in parallel (subject to hardware constraints). The architecture spawns multiple LLM inference tasks simultaneously, each with its own prompt and context, then collects results as they complete. This pattern is useful for embarrassingly parallel workloads (e.g., processing a batch of documents, generating multiple variations) where tasks are independent and can share the same model instance.

Provides educational guidance on selecting appropriate quantized GGUF models based on task requirements, hardware constraints, and quality/speed tradeoffs. The architecture documents model characteristics (parameter count, quantization level, context window, inference speed) and helps developers choose between models like Mistral, Llama 2, Phi, and others. The repository includes a model download utility (npx node-llama-cpp pull) that surfaces model options and their specifications, enabling informed selection without trial-and-error.

Exposes temperature, top-p, and other sampling parameters to control LLM output randomness and creativity. The architecture allows developers to tune these parameters per request, enabling different behaviors for different tasks (e.g., low temperature for deterministic code generation, high temperature for creative writing). Parameters are passed to the node-llama-cpp inference engine, which uses them to control the probability distribution over next tokens during generation.

Provides utilities and patterns for tracking token usage and managing context window constraints to prevent exceeding model limits. The architecture includes token counting logic (either through node-llama-cpp's built-in tokenizer or external libraries) that estimates prompt and response token counts before generation. Developers can use this information to implement context windowing strategies (e.g., dropping oldest messages when approaching limit) or warn users when approaching capacity.