BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (BERT)

BERT learns deep contextual embeddings for text tokens by pre-training on unlabeled corpora using a masked language model (MLM) objective: 15% of input tokens are randomly masked, and the model predicts masked tokens using bidirectional context from both left and right neighbors across all Transformer encoder layers. This contrasts with unidirectional models (GPT-style) that condition only on preceding or following context, enabling richer semantic representations that capture full syntactic and semantic context for each token.

BERT pre-trains a secondary binary classification objective (Next Sentence Prediction, NSP) that learns to predict whether sentence B immediately follows sentence A in the training corpus. This task operates at the sequence level using the [CLS] token representation and forces the model to learn discourse-level coherence patterns, sentence boundaries, and semantic relationships between consecutive sentences beyond token-level masked prediction.

BERT can be fine-tuned for semantic role labeling (SRL) by predicting argument spans and their semantic roles (agent, patient, instrument, etc.) for a given predicate. The model learns to identify argument boundaries and classify their semantic roles using token-level representations, leveraging bidirectional context to understand predicate-argument relationships without explicit syntactic parsing.

BERT enables transfer learning by providing a shared pre-trained representation that can be fine-tuned for diverse downstream tasks (classification, tagging, span selection, etc.) with minimal task-specific modifications. The pre-trained bidirectional context captures general linguistic knowledge (syntax, semantics, discourse) that transfers effectively across tasks, reducing the amount of labeled data required for each task and accelerating convergence during fine-tuning.

BERT can be extended to multilingual settings by pre-training on unlabeled text from multiple languages using the same masked language modeling objective. The shared vocabulary and bidirectional context enable the model to learn language-agnostic representations that capture universal linguistic patterns, enabling zero-shot or few-shot transfer across languages. While not explicitly detailed in the abstract, multilingual BERT (mBERT) extends the approach to 104+ languages.

BERT enables task-specific adaptation by adding a single task-specific output layer on top of pre-trained representations and fine-tuning the entire model (or a subset) on labeled task data. The architecture requires minimal modification: for classification tasks, the [CLS] token representation feeds into a softmax layer; for span selection (e.g., question answering), token-level representations are scored directly. This approach contrasts with prior methods requiring substantial task-specific architecture engineering.

BERT is evaluated on a comprehensive suite of 11 NLP benchmarks spanning text classification (GLUE), natural language inference (MultiNLI), question answering (SQuAD v1.1 and v2.0), and semantic similarity tasks. The evaluation demonstrates consistent improvements over prior state-of-the-art baselines (e.g., +7.7 percentage points on GLUE, +1.5 F1 on SQuAD v1.1), validating the pre-training approach across diverse task types and data scales.

BERT fine-tunes for extractive question answering (SQuAD) by predicting start and end token positions within a passage using token-level representations. The model scores each token's probability of being a span start or end position, leveraging bidirectional context to disambiguate correct answer spans. Performance improvements on SQuAD v1.1 (+1.5 F1) and v2.0 (+5.1 F1, which includes unanswerable questions) demonstrate the effectiveness of bidirectional context for span selection.

BERT fine-tunes for natural language inference (NLI) tasks like MultiNLI by classifying sentence pairs into entailment, contradiction, or neutral categories. The [CLS] token representation (optimized during pre-training via NSP) feeds into a softmax layer for 3-way classification. The bidirectional context enables the model to understand semantic relationships between premise and hypothesis without explicit alignment mechanisms.

BERT fine-tunes for text classification tasks (part of GLUE benchmark) by using the [CLS] token's contextual representation as a fixed-size feature vector that feeds into a softmax classification layer. The [CLS] token is positioned at the start of every input sequence and its representation is optimized during pre-training (via NSP) to capture sequence-level semantics, making it a natural choice for classification without requiring pooling or aggregation strategies.

BERT fine-tunes for semantic textual similarity (STS) tasks by predicting a continuous similarity score (typically 0-5) for sentence pairs. The [CLS] token representation or a pooled representation feeds into a regression head that outputs a single similarity score. The bidirectional context enables the model to understand nuanced semantic relationships between sentences (paraphrase, entailment, contradiction) and map them to a continuous similarity scale.

BERT fine-tunes for named entity recognition (NER) by applying a classification layer to each token's representation, predicting entity tags (e.g., B-PER, I-PER, B-LOC, O) for each token. The bidirectional context enables the model to disambiguate entity boundaries and types using full sentence context, improving accuracy on NER benchmarks compared to unidirectional models or shallow sequence labeling approaches.

BERT can be fine-tuned for coreference resolution by learning to identify and link coreferent mention spans (e.g., 'John' and 'he' referring to the same entity). The model learns span representations by combining token representations (e.g., start token, end token, span width embeddings) and predicts coreference links between spans using pairwise scoring. Bidirectional context enables the model to understand entity mentions and their relationships across long-range dependencies.