Tokenization in NLP: An Academic Guide

How text is split into tokens and converted to model inputs.