Tokenization vs. Embeddings: From Text Pieces to Meaning

Most explanations of language models blur two very different operations: tokenization, which decides what the model will read, and embedding, which gives what it reads a numerical representation.

The distinction is simple but consequential. A tokenizer turns text into a sequence of discrete symbols and IDs. An embedding model turns those symbols—or a whole passage assembled from them—into dense vectors whose geometry can encode patterns of meaning.

The visual captures the handoff: human-readable text is first segmented into model-readable units; those units can then be mapped into a numerical space. The first step is an encoding convention. The second is a learned representation.

The short version

Suppose the input is:

Tokenization and embeddings solve different problems.

A tokenizer might split it into pieces resembling:

• Token
• ization
• and
• embed
• dings
• solve
• different
• problems
• .

Each piece is assigned an integer from a fixed vocabulary. Those integers are token IDs. They are labels, not coordinates: ID 8,421 is not inherently closer in meaning to ID 8,422 than it is to ID 17.

An embedding layer or embedding model then maps the input to vectors such as [0.12, -0.48, …]. In that space, distance and direction can become useful. Related words, sentences, or documents may land near one another, depending on what the model was trained to represent.

So the division of labor is:

1. Tokenization: text → tokens → integer IDs
2. Embedding: tokens or text → dense numerical vectors
3. Model computation: vectors → contextual representations, predictions, generations, classifications, or retrieval results

Tokenization determines the units. Embedding supplies the geometry.

What tokenization actually does

Language models do not usually process raw strings directly. They work with a vocabulary: a finite inventory of pieces the model knows how to identify. The tokenizer applies normalization rules, segments the input, looks up each segment in that vocabulary, and may add special tokens that mark boundaries or roles.

A good tokenizer balances competing goals. A vocabulary made only of whole words is efficient for common language but fails gracefully only if every possible word, spelling, name, and typo is known. A character vocabulary can represent almost anything, but makes sequences much longer. Subword tokenization occupies the middle ground: common words remain compact, while uncommon words can be assembled from smaller pieces.

This is why token count is not word count. The same sentence can consume different numbers of tokens under different tokenizers, and rare names, code, or underrepresented languages may fragment more heavily.

The key tokenization approaches

Word and whitespace tokenization

The earliest approach is to split at spaces and punctuation. It is intuitive and easy to inspect, but brittle. “Run,” “runs,” and “running” become separate vocabulary entries; URLs, emoji, contractions, and languages without spaces complicate the rules; unseen words require an unknown-token fallback.

Word tokenization remains useful in simple pipelines, but modern general-purpose language models usually favor subwords.

Character and byte tokenization

Character tokenizers treat individual characters as units. Byte tokenizers go lower and operate on the bytes used to encode text. Both have strong coverage: nearly any input can be represented without an unknown word.

The tradeoff is length. A common word that could be one token may require several characters or bytes. Longer sequences cost more computation and leave less room in a fixed context window.

Pure byte models exist, but bytes are also commonly used as the safe foundation for a subword algorithm.

Byte Pair Encoding

Byte Pair Encoding, or BPE, begins with small units and repeatedly merges the adjacent pair that appears most often in a training corpus. Over many merges, frequent patterns become single vocabulary items while rare text remains decomposable.

For example, a corpus might lead BPE to merge t + h into th, then th + e into the. The learned merge list is applied in a defined order when new text is encoded.

Modern implementations often start from bytes. Byte-level BPE can therefore represent arbitrary Unicode text while still compressing frequent sequences into larger tokens. GPT-style tokenizers, including implementations such as tiktoken, are prominent examples of this family.

WordPiece

WordPiece, associated with BERT-style models, also builds a subword vocabulary, but its vocabulary-building objective differs from classic BPE. Instead of selecting a merge only because a pair is frequent, it favors pieces that improve the likelihood of the training data under the vocabulary model.

At encoding time, WordPiece commonly uses a longest-match-first strategy: it takes the longest vocabulary item that fits, then continues through the word. Continuation pieces are often displayed with markers such as ##.

The result resembles BPE to a reader, but the scoring and segmentation rules are not identical.

Unigram language-model tokenization

The Unigram approach starts with a large set of candidate pieces and repeatedly removes the least useful ones. It assigns probabilities to pieces and chooses the segmentation with the strongest likelihood, often using dynamic programming.

Unlike a fixed greedy merge path, Unigram can retain multiple plausible segmentations. That makes techniques such as subword regularization possible: training can sample alternate segmentations, encouraging the model to be less dependent on one exact split.

SentencePiece

SentencePiece is best understood as a tokenizer framework rather than one segmentation algorithm. It can train BPE or Unigram tokenizers directly from raw text, treats spaces as explicit symbols, and avoids requiring language-specific pre-tokenization.

That design is valuable for multilingual systems and scripts where space-based word splitting is a poor assumption. SentencePiece is widely used in T5-, Llama-, and other model families, though the exact algorithm and vocabulary differ by model.

What an embedding is

An embedding is a learned vector: a list of floating-point values whose position is useful because of patterns learned from data. “Dense” means that most dimensions contain a value, unlike a sparse one-hot vector with a single active position.

There are several meanings of embedding, and mixing them causes confusion.

A model begins with a token embedding table. Each token ID indexes one row of that learned matrix. If the vocabulary has V entries and the hidden width is d, the table has roughly V × d learned values. Position information is then added or incorporated so the model can distinguish “dog bites man” from “man bites dog.”

After Transformer layers process the sequence, each token has a contextual embedding. The vector for “bank” can differ between “river bank” and “investment bank” because attention has incorporated the surrounding words.

A separate text embedding model usually turns a whole sentence, passage, or document into one fixed-length vector. That vector is designed for comparison in search, clustering, recommendations, deduplication, or classification.

These objects are related, but they are not interchangeable.

The key embedding model families

Word2vec

word2vec learns one vector per word from local context. In the skip-gram formulation, the model uses a center word to predict nearby words. In continuous bag of words, or CBOW, it uses nearby words to predict the center.

Words that appear in similar contexts acquire similar vectors. This produced famous demonstrations of vector relationships, but each word still gets one context-independent representation. “Bank” has the same vector in every sentence, and out-of-vocabulary words remain a problem.

GloVe

GloVe, or Global Vectors, learns from aggregate word co-occurrence statistics. It builds a matrix recording how often words appear near one another, then learns vectors whose dot products approximate transformed co-occurrence ratios.

Where word2vec emphasizes local prediction, GloVe makes global corpus statistics explicit. Like word2vec, it produces static word vectors rather than different vectors for different uses of a word.

fastText

fastText represents a word through character n-grams as well as a whole-word vector. “Embedding,” for example, can share subword features with “embeddings” and “embedded.”

This helps with morphology, misspellings, and unseen words because a vector can be composed from familiar character fragments. Its representations are still generally static, but its subword construction gives it better coverage than a word-only lookup.

Contextual Transformer embeddings

Models such as BERT and RoBERTa create contextual token representations. They begin with token embeddings, pass them through stacked self-attention layers, and update each position using information from the rest of the sequence.

Their pretraining objectives teach them to infer linguistic and semantic patterns from context. A hidden state can support classification, extraction, or other downstream tasks, but an arbitrary pooled BERT output is not automatically an excellent retrieval embedding.

Sentence Transformers

Sentence-BERT and the broader sentence-transformers approach adapt Transformer encoders to produce comparable sentence or passage vectors. A pooling step—often a mean across token representations—creates one vector, while paired or triplet training teaches related texts to move closer and unrelated texts to move apart.

This makes semantic similarity practical at scale: documents can be embedded once, stored in a vector index, and compared with an embedded query without running every query-document pair through a larger cross-encoder.

Modern retrieval embeddings

Families such as E5 and BGE, along with hosted embedding services, extend this pattern with large-scale contrastive training and task-oriented data. Many distinguish between query and document roles, sometimes through instruction prefixes, so the model learns an asymmetric retrieval task rather than generic similarity alone.

The basic mechanism remains consistent: tokenize the input, encode the token sequence with a neural network, pool it into a fixed-size vector, and train the vector space so useful pairs have high similarity.

How the two stages work together

In a Transformer pipeline, the flow is roughly:

1. The tokenizer applies its normalization and segmentation rules.
2. Vocabulary lookup converts tokens to IDs.
3. IDs select learned token vectors from an embedding table.
4. Position information and model layers turn those initial vectors into contextual representations.
5. For retrieval, a pooling or projection step may produce one vector for the entire input.
6. A similarity function—often cosine similarity or a dot product—compares that vector with others.

The tokenizer is therefore upstream of the embedding model. Changing the tokenizer without retraining or carefully adapting the model breaks the expected ID-to-vector mapping. Token ID 500 only means what a particular vocabulary and model jointly learned it to mean.

Likewise, vectors from different embedding models should not be mixed in the same index merely because they have the same number of dimensions. Each model defines its own coordinate system.

Why the distinction matters in practice

For generation, tokenizer choice affects latency, cost, and usable context. A phrase that becomes three tokens in one model may become eight in another. Fragmentation can be especially uneven across languages, specialized terminology, and source code.

For retrieval, tokenization still affects what the encoder sees, but the embedding model and its training objective determine whether the final vector captures topical similarity, paraphrase, intent, or some other relationship. Chunk size matters too: a long document squeezed into one vector can blur several subjects, while tiny chunks may lose the context needed to interpret them.

A few rules prevent common mistakes:

• Do not treat token IDs as semantic measurements.
• Do not assume tokens correspond cleanly to words.
• Do not compare vectors produced by different embedding spaces.
• Do not choose an embedding model only by dimension count.
• Do evaluate retrieval on representative queries, languages, and document types.
• Do store the model name and version with every vector index so it can be rebuilt deliberately.

A useful mental model

Think of tokenization as making a set of tiles and spelling the input with them. Embedding assigns each tile an initial location in a learned numerical space. The model then moves those representations according to context. If the task needs one vector for a sentence or document, an embedding model gathers that contextual information into a point designed for comparison.

Tokenization answers: “What pieces will the model read?”

Embeddings answer: “How will those pieces—or the text they form—be represented numerically?”

Both are foundational. They solve different problems, and the quality of an AI system depends on understanding where one ends and the other begins.