OpenAI: GPT-4o

GPT-4o processes both text and image inputs through a single unified transformer backbone, eliminating separate vision and language encoders. Images are tokenized into visual patches and embedded into the same token sequence as text, allowing the model to reason jointly over mixed modalities without explicit fusion layers. This architecture enables pixel-level image understanding (OCR, spatial reasoning, object detection) while maintaining full language comprehension in a single forward pass.

GPT-4o maintains a 128,000-token context window, allowing it to process and generate responses based on very long documents, codebases, or conversation histories in a single request. The model uses rotary positional embeddings (RoPE) and efficient attention mechanisms to handle this extended context without quadratic memory explosion. Developers can submit entire books, API documentation, or multi-file code repositories and ask questions that require reasoning across the full context.

GPT-4o can be fine-tuned on custom training data to adapt the model to specific domains, writing styles, or task-specific behaviors. Fine-tuning uses supervised learning to update model weights based on provided examples, allowing developers to create specialized versions of GPT-4o. The fine-tuning process is managed via the OpenAI API, with training data provided as JSONL files containing prompt-completion pairs.

GPT-4o supports constrained generation via JSON schema specification, ensuring output strictly adheres to a provided schema without post-processing or validation. The model uses grammar-constrained decoding (similar to outlines.ai or llama.cpp's approach) to enforce token-level constraints during generation, guaranteeing valid JSON that matches the schema. Developers specify a JSON schema in the API request, and the model generates only tokens that produce valid schema-compliant output.

GPT-4o supports server-sent events (SSE) streaming, delivering generated tokens to the client as they are produced rather than waiting for the full response. The API streams tokens individually, allowing developers to display text progressively, implement real-time chat interfaces, or cancel requests mid-generation. Streaming uses HTTP chunked transfer encoding with JSON-formatted token events, enabling low-latency user feedback.

GPT-4o supports function calling via a schema-based tool registry, where developers define functions as JSON schemas and the model decides which tools to invoke and with what arguments. The model can call multiple functions in parallel within a single response, and the API supports automatic tool result injection for multi-turn tool use. The implementation uses a special token vocabulary for function calls, allowing the model to reason about tool use without generating raw function names.

GPT-4o performs spatial reasoning over images, understanding object locations, relationships, and hierarchies without explicit bounding box annotations. The model can identify objects, read text at various scales, understand diagrams and charts, and reason about spatial relationships (above, below, inside, overlapping). This capability is built into the unified multimodal architecture, allowing the model to ground language understanding in visual context.

GPT-4o generates code across 40+ programming languages, supporting both full function generation and inline completion. The model understands language-specific syntax, idioms, and best practices, and can generate code that integrates with existing codebases when provided with sufficient context. Code generation uses the same transformer backbone as text generation, allowing the model to reason about code structure and dependencies.

GPT-4o can be prompted to generate detailed reasoning chains before providing final answers, using explicit chain-of-thought (CoT) patterns. The model breaks down complex problems into steps, shows intermediate reasoning, and arrives at conclusions through explicit logical progression. This capability is enabled through prompt engineering rather than architectural changes, but the model's training makes it particularly effective at following CoT instructions.

GPT-4o includes built-in content moderation that filters harmful outputs (violence, hate speech, sexual content, etc.) based on OpenAI's usage policies. The moderation is applied at the output level, preventing the model from generating prohibited content. Developers can also use OpenAI's Moderation API to classify user inputs and filter requests before sending them to GPT-4o, creating a two-layer safety approach.

GPT-4o supports batch processing via the OpenAI Batch API, allowing developers to submit hundreds or thousands of requests in a single batch and receive results asynchronously. Batch requests are processed at off-peak times and cost 50% less than standard API calls, making them ideal for non-time-sensitive workloads. Requests are submitted as JSONL files, processed in parallel, and results are returned in a single output file.