Fetch MCP Server

Implements MCP tool registration for HTTP GET/POST requests to arbitrary URLs, using the Python MCP SDK's tool decorator pattern to expose fetch operations as callable tools within the MCP server-client communication model. The server registers fetch as a tool primitive that LLM clients can invoke through JSON-RPC, handling request/response serialization and error propagation through the MCP transport layer (stdio, SSE, or custom).

Transforms fetched HTML content into Markdown format using a conversion library (likely html2text or similar), optimizing the output for LLM token efficiency and readability. The conversion strips unnecessary HTML tags, preserves semantic structure (headings, lists, links), and produces compact Markdown that reduces token overhead compared to raw HTML. This is applied automatically to fetch results before returning to the LLM client.

Validates fetch requests against the target domain's robots.txt file before making HTTP requests, parsing the robots.txt rules and checking if the requested URL path is allowed for the user-agent. If robots.txt forbids access, the fetch is rejected with an appropriate error message. This is implemented as a pre-flight validation step that runs before the actual HTTP request is made, respecting web scraping conventions and legal/ethical standards.

Registers the fetch capability as an MCP tool primitive with a JSON schema that describes input parameters (URL, method, headers, body) and output format. The server exposes this schema through MCP's tools/list endpoint, allowing clients to discover the fetch tool and understand its interface without documentation. The schema includes parameter validation rules and type information that clients use to construct valid tool invocations.

Captures HTTP errors (4xx, 5xx status codes, network timeouts, DNS failures) and propagates them back to the MCP client with descriptive error messages and status codes. Network errors are caught and wrapped in MCP error responses, allowing clients to distinguish between successful fetches, client errors (bad URL), server errors (target down), and network issues. Error messages include the original HTTP status code and reason phrase when available.

Allows MCP clients to specify custom HTTP headers (User-Agent, Authorization, Accept, etc.) when invoking the fetch tool, enabling authentication to protected resources and customization of request behavior. Headers are passed as a parameter to the fetch tool and merged with any default headers set by the server. This enables use cases like API authentication, cookie-based sessions, and content negotiation without requiring separate tool variants.

Extends the fetch tool to support HTTP POST requests with custom request bodies, enabling interaction with APIs that require POST for data submission. Clients can specify the HTTP method (GET/POST), request body content, and Content-Type header. The server constructs the POST request with the provided body and sends it to the target URL, returning the response body and status code. This enables agents to submit forms, call APIs, and interact with web services that require POST.

Detects the Content-Type of HTTP responses and applies appropriate handling: HTML is converted to Markdown, JSON is returned as-is, plain text is returned as-is, and binary content (images, PDFs) is rejected or summarized. The server reads the Content-Type header from the HTTP response and routes the content through the appropriate handler. This ensures that responses are in a format suitable for LLM consumption regardless of the target URL's content type.

Implements the fetch server as an MCP server that communicates with clients through the MCP protocol using a transport layer (stdio, SSE, or custom). The server uses the Python MCP SDK to handle protocol-level concerns (JSON-RPC message framing, request/response routing, error serialization) while the application code focuses on tool logic. The transport layer is abstracted away, allowing the same server code to work with different transport mechanisms without modification.