get-llms-txt

Parses markdown and MDX files from a documentation source directory and extracts semantic content blocks (headings, paragraphs, code blocks, lists) into a structured format optimized for LLM consumption. Uses AST-based parsing to preserve document hierarchy and metadata, then flattens content into a single llms.txt file with clear delimiters and context markers that help LLMs understand document structure without needing to parse raw markdown syntax.

Automatically detects and adapts to different documentation framework conventions (Next.js, Astro, Docusaurus, VitePress, Gatsby) by identifying framework-specific directory patterns, configuration files, and content organization schemes. Uses heuristic-based framework detection (checking for framework config files like next.config.js, astro.config.mjs, docusaurus.config.js) to determine the correct source directory and content structure without requiring explicit configuration.

Walks through nested directory structures starting from a detected or configured source directory, recursively discovers all markdown and MDX files, and applies filtering rules to include/exclude content based on file patterns, directory names, and metadata. Uses file system APIs with configurable glob patterns or ignore rules to skip common non-content directories (node_modules, .git, build output) and focus only on documentation source files.

Transforms markdown syntax into plain text while preserving semantic meaning and document structure through strategic formatting choices. Converts markdown headers to uppercase labels with separators, converts lists to indented plain text, strips inline formatting (bold, italic) while keeping content, removes markdown-specific syntax (backticks, brackets), and preserves code blocks as indented text blocks. This approach ensures LLMs can understand content hierarchy without needing to parse markdown syntax.

Processes MDX files containing embedded JSX components and React code by extracting text content from component props, rendering component descriptions, and handling interactive elements as plain text descriptions. Parses JSX syntax to identify component boundaries, extracts meaningful text from component children and props, and generates fallback text descriptions for components that don't have direct text equivalents (e.g., 'Interactive Code Example' for a CodeSandbox embed).

Generates llms.txt output with customizable formatting options including configurable section delimiters, header formatting styles, content separators, and metadata inclusion. Allows users to specify how headers are formatted (e.g., '# HEADER' vs '=== HEADER ==='), what separators divide sections, whether to include file paths or metadata, and how to structure the final output. Supports multiple output format presets (compact, verbose, structured) to optimize for different LLM consumption patterns.

Processes multiple markdown and MDX files in a single operation, aggregates their content into a unified llms.txt output, and maintains file-level organization through metadata or section markers. Reads all discovered files, parses each independently, concatenates converted content with clear file boundaries, and optionally includes file path information or table of contents to help LLMs navigate the aggregated content. Handles file ordering (alphabetical, by modification time, or custom) to ensure consistent output.

Distributes get-llms-txt as an npm package with a command-line interface that can be invoked directly or integrated into build scripts and CI/CD pipelines. Provides both programmatic API (for Node.js projects) and CLI commands (for shell scripts and automation), supports configuration via command-line arguments or config files, and integrates with npm scripts in package.json for automated llms.txt generation during builds or deployments.

Detects changes in source markdown files since the last llms.txt generation and optionally regenerates only affected sections or the entire file based on modification timestamps or content hashing. Tracks file modification times or computes hashes of source files to determine if regeneration is necessary, enabling faster builds by skipping unchanged documentation. Can be configured to always regenerate or to use change detection for optimization.

Extracts YAML or TOML front matter from markdown files (metadata like title, description, tags, date) and optionally includes this metadata in the llms.txt output or uses it to filter/organize content. Parses front matter blocks at the beginning of files, extracts key-value pairs, and can use metadata to determine file importance, ordering, or inclusion in the output. Supports filtering files based on metadata (e.g., exclude draft files, include only published content).