CodeSearchNet

Extracts 6 million functions from public GitHub repositories across Python, Java, JavaScript, PHP, Ruby, and Go using language-specific AST parsers and tokenizers. Each function is normalized to a canonical representation with consistent formatting, removing language-specific syntax variations while preserving semantic structure. The extraction pipeline handles edge cases like nested functions, lambdas, and anonymous classes through recursive AST traversal and scope-aware filtering.

Pairs extracted functions with their associated docstrings (docstrings, comments, and inline documentation) to create 6 million code-documentation tuples. The pairing logic uses heuristic matching (proximity-based, AST-aware comment association) and filtering to ensure semantic alignment between code and documentation. Removes low-quality pairs (undocumented functions, trivial stubs) through statistical filtering and manual validation on a subset.

Provides a standardized evaluation framework with train/validation/test splits and metrics (Mean Reciprocal Rank, NDCG, precision@k) for assessing code search system performance. The benchmark includes query sets (natural language queries paired with relevant code functions) and baseline implementations, enabling reproducible comparison of different code search approaches. Evaluation is performed at function-level granularity with relevance judgments derived from docstring-query similarity and manual validation.

Enables training of polyglot code understanding models that learn a shared embedding space across 6 programming languages. The representation is derived from normalized function code and documentation, allowing models to map semantically equivalent functions in different languages to nearby points in embedding space. This is achieved through contrastive learning objectives (e.g., code-documentation pairs as positive examples, random negatives) that learn language-invariant code semantics.

Enables training and evaluation of code clone detection systems by providing a large corpus of functions with implicit similarity relationships derived from documentation and code structure. The dataset can be used to identify Type-1 (exact) and Type-2 (syntactically similar) clones through embedding similarity, and to train models that detect semantic clones (Type-3/4) that perform similar functionality despite different syntax. Similarity is computed via cosine distance in embedding space or explicit clone annotation.

Serves as a large-scale, pre-training corpus for code understanding models like CodeBERT and GraphCodeBERT. The dataset provides 6 million code-documentation pairs that enable self-supervised and supervised pre-training objectives (masked language modeling, code-documentation matching, contrastive learning). The corpus is diverse across languages and domains, reducing domain bias and improving generalization to downstream tasks.

Provides mechanisms to generate natural language queries from code functions and assess relevance between queries and code. Queries are generated from docstrings and function signatures through extractive and abstractive summarization, or manually curated. Relevance assessment uses docstring-query similarity (BM25, embedding-based) and optional manual validation to create ground truth for evaluation. This enables creation of query-code relevance judgments for benchmark evaluation.

Provides language-aware tokenization and shared vocabulary for code across 6 programming languages. Tokenization handles language-specific syntax (operators, keywords, delimiters) while creating a unified vocabulary that maps tokens from different languages to shared semantic categories. This enables models to process code from any supported language using a single tokenizer and vocabulary, reducing model complexity and enabling cross-language transfer.