Tokenization in NLP: An Academic Guide

An introduction to how language models turn raw text into tokens and numerical representations, including word and subword units and Word2Vec-style embeddings.