koelectra-small-v3-nsmc

Performs binary sentiment classification (positive/negative) on Korean text using a small ELECTRA discriminator model fine-tuned on the NSMC (Naver Sentiment Movie Comments) dataset. The model leverages ELECTRA's replaced-token detection pretraining approach combined with task-specific fine-tuning on 200K Korean movie reviews, enabling efficient sentiment inference with 23.5M parameters. Inference runs locally via PyTorch/Hugging Face Transformers without requiring API calls, supporting batch processing and custom confidence thresholds.

Processes multiple Korean text samples in parallel batches using Hugging Face Transformers' DataCollator with dynamic padding, which pads sequences to the longest sample in each batch rather than a fixed max length. This reduces computational waste and memory overhead when processing variable-length Korean text. Supports configurable batch sizes and automatic device placement (CPU/GPU), enabling efficient throughput for production inference pipelines without manual padding logic.

Loads model weights from Hugging Face Hub using safetensors format (a secure, fast serialization standard) instead of pickle, with automatic version management and caching. The model is stored as a public repository with git-based versioning, allowing reproducible downloads of specific commits/tags. Safetensors format enables faster deserialization (~10x vs pickle) and eliminates arbitrary code execution risks during weight loading, making it suitable for production and untrusted environments.

Tokenizes Korean text using ELECTRA's pretrained WordPiece tokenizer, which was trained on Korean corpora and includes morphological awareness for Korean-specific linguistic patterns (e.g., particles, verb conjugations, compound words). The tokenizer handles Korean-specific edge cases like spacing conventions, Hangul decomposition, and subword segmentation optimized for Korean morphology. Supports both encoding (text → token IDs) and decoding (token IDs → text) with configurable special tokens and truncation strategies.

Provides a pretrained ELECTRA discriminator checkpoint that can be fine-tuned for downstream Korean text classification tasks beyond sentiment analysis. The model's learned representations capture Korean linguistic patterns from pretraining, enabling efficient transfer learning with minimal labeled data. Supports standard fine-tuning workflows (adding task-specific head, freezing/unfreezing layers, learning rate scheduling) via Hugging Face Transformers' Trainer API or custom PyTorch training loops.

Outputs softmax-normalized probability distributions over sentiment classes (positive/negative), enabling confidence-based filtering and decision-making. The model produces logits that are converted to probabilities via softmax, allowing downstream systems to reject low-confidence predictions or apply different handling strategies based on confidence thresholds. Supports both hard predictions (argmax class) and soft predictions (probability distributions) for flexible integration into decision pipelines.