CodeT5

Generates code from natural language descriptions using a T5-based encoder-decoder architecture enhanced with instruction-tuning objectives. InstructCodeT5+ 16B variant processes natural language input through the encoder, then decodes syntactically valid code sequences using teacher-forced training with code-specific tokenization. The model achieves 36.1% Pass@1 on HumanEval by learning to follow structured programming instructions rather than pure next-token prediction.

Extracts dense vector embeddings from code snippets using a specialized 110M parameter embedding model that encodes semantic meaning of code into fixed-dimension vectors. The model processes code through a shared encoder and projects outputs to embedding space, enabling fast approximate nearest-neighbor search for code retrieval tasks. Achieves 74.23 average MRR across six programming languages by learning language-agnostic code semantics.

Tokenizes code from multiple programming languages (Python, Java, JavaScript, Go, Ruby, PHP, C++) using a unified vocabulary that captures language-agnostic code patterns. The tokenizer preserves code structure (indentation, brackets) while normalizing language-specific syntax, enabling a single model to process code across languages. Unified vocabulary reduces model size compared to language-specific tokenizers while maintaining code semantics.

Provides a configuration system that abstracts model loading, tokenization, and inference across different CodeT5+ variants (110M embedding, 220M bimodal, 770M general, 2B/6B/16B generation, InstructCodeT5+ 16B). Developers specify model variant and task in configuration files, and the framework automatically loads correct weights, tokenizer, and inference pipeline. Enables switching between models without code changes.

Retrieves similar code snippets from a codebase using code-to-code similarity computed via embedding vectors. The embedding model learns code semantics that capture functional similarity beyond syntactic matching, enabling detection of code clones with different variable names or control flow. Useful for identifying duplicate implementations, refactoring opportunities, and security vulnerabilities.

Summarizes code into natural language descriptions using a 220M bimodal encoder-decoder that jointly processes code and text representations. The encoder learns unified representations of code syntax and semantics, while the decoder generates abstractive summaries in natural language. Bimodal training on code-summary pairs enables the model to capture both structural and semantic aspects of code without language-specific tokenizers.

Provides a family of pre-trained models (110M embedding, 220M bimodal, 770M general, 2B/6B/16B generation, InstructCodeT5+ 16B) allowing developers to select variants based on latency-accuracy tradeoffs. Each variant is pre-trained on the same code corpus but optimized for different tasks and inference constraints. The architecture enables progressive scaling from lightweight embedding models (2GB VRAM) to large generation models (32GB VRAM) without retraining.

Evaluates code generation models using the HumanEval benchmark, which tests functional correctness on 164 hand-written programming problems. The evaluation framework computes Pass@k metrics (Pass@1, Pass@10, Pass@100) by sampling k code completions and checking if any passes unit tests. CodeT5+ 16B achieves 30.9% Pass@1 and 76.7% Pass@100, demonstrating the gap between single-attempt and multi-sample generation.

Computes CodeBLEU scores to evaluate code generation quality by measuring n-gram overlap, syntax tree matching, and dataflow matching between generated and reference code. The metric combines BLEU-style token matching with code-specific structural features (AST nodes, variable dataflow) to capture both syntactic and semantic correctness. Provides more nuanced evaluation than BLEU alone by rewarding structurally similar code even with different variable names.

Retrieves code snippets matching natural language queries by computing similarity between query embeddings and code embeddings across six programming languages. The 220M matching variant uses a bimodal encoder to align text and code representations in shared embedding space, enabling zero-shot retrieval across languages without language-specific fine-tuning. Achieves 75.85 average MRR by learning language-agnostic code semantics.

Provides a configurable fine-tuning pipeline that adapts pre-trained CodeT5+ models to downstream tasks (code generation, summarization, retrieval) using task-specific loss functions and data formats. The framework handles data loading, tokenization, training loop management, and evaluation, supporting both supervised fine-tuning and instruction-tuning objectives. Enables developers to customize training without reimplementing the full training loop.

Trains models to follow natural language instructions for code generation using instruction-tuning objectives that pair code with structured instructions (e.g., 'Generate a function that sorts an array'). InstructCodeT5+ 16B learns to parse instructions, decompose them into subtasks, and generate corresponding code. The approach achieves 36.1% Pass@1 on HumanEval by optimizing for instruction-following rather than raw next-token prediction.

Pre-trains CodeT5 models using masked span prediction (similar to BERT) combined with next-token prediction objectives on large code corpora. The encoder learns bidirectional code representations by predicting masked token spans, while the decoder learns to generate code autoregressively. This dual-objective pre-training enables the model to understand code structure (encoder) and generate code sequences (decoder) without task-specific supervision.