Unigram Language Model Tokenization With Probability Based Selection

via “token-level probability and uncertainty estimation”

text-generation model by undefined. 71,06,872 downloads.

Unique: Exposes full vocabulary probability distributions at inference time without requiring model modification, enabling post-hoc confidence filtering and uncertainty quantification that works with any decoding strategy (greedy, beam, sampling)

vs others: More transparent than black-box confidence scoring but less calibrated than ensemble methods or Bayesian approaches; faster than external uncertainty quantification but requires manual threshold tuning