From Raw Text to Token IDs and Embeddings

In Your First LLM Walkthrough: From Prompt to Answer, you saw a prompt go into a language model and tokens come out. That lesson showed the outer loop: what the user sees. This lesson opens the first step inside the model—how raw text becomes numbers the model can calculate with.

The fundamental problem is this: a neural network is made of mathematical operations—additions and multiplications of numbers. It cannot directly compute on the text string "The cat sat". A computer program does not inherently know what letters or words mean; it only knows numerical values.

The pipeline in one sentence: Raw text → Tokens → Token IDs → Embedding Table Lookup → Input Tensor.

---

Step 1: Breaking Text into Tokens

A token is a single unit or chunk of text processed by a language model. It can be a whole word, part of a word (a subword), a punctuation mark, or a space.

A tokenizer is the program or algorithm that chops a raw string into tokens.

Let’s start with our running prompt:

Prompt: "The cat sat"

In our toy tokenizer for this lesson, we split by whole words:

Tokens: ["The", "cat", "sat"]

That gives a sequence of 3 tokens. This whole-word split is an illustrative example designed to keep the numbers simple; real tokenizers use subwords, as we will see later.

---

Step 2: From Tokens to Token IDs (Vocabulary and Table Indices)

A vocabulary is the fixed dictionary or catalog of every unique token the model knows, where each token is assigned a unique integer index.

A token ID is an integer used solely as an index (a row number) to look up a token in the model’s vocabulary and embedding table.

Important: Token IDs are strictly categorical indices, not numerical measurements.

• Token ID 2 is not “twice as large” or “twice as meaningful” as Token ID 1.
• Adding ID(The) + ID(cat) = 1 + 2 = 3 (sat) has zero semantic or linguistic meaning.
• Arithmetic on ID values does not represent semantic relationships.
• IDs cannot be fed directly into math operations because treating table row indices as numbers would imply false numerical relationships.

Our toy vocabulary has 7 tokens (IDs 0 to 6):

Token ID	Token String	Meaning / Note
0	[PAD]	Special padding token (placeholder)
1	The	Article
2	cat	Noun
3	sat	Verb
4	dog	Noun
5	on	Preposition
6	mat	Noun

Mapping our prompt "The cat sat":

Token	Token ID
The	1
cat	2
sat	3

Resulting sequence of Token IDs: [1, 2, 3]

---

Step 3: From Token IDs to Vectors (The Embedding Table)

A vector is an ordered list of numbers (coordinates) that describe features or position in a mathematical space.

An embedding is a vector of learned floating-point numbers assigned to a token that represents its static features to the model (Mikolov et al., 2013).

The embedding dimension ($d$, often written as d_model in literature) is the number of values inside each token’s vector (Vaswani et al., 2017). In real models, $d$ might be 768 (GPT-2 Small) or 4,096 (LLaMA-7B). In our toy model, we choose $d = 4$ so every number can be inspected by eye.

The embedding table (embedding matrix) is a lookup table with $V$ rows (one for each vocabulary ID) and $d$ columns (one for each feature).

Note on Toy Vectors: The numbers in the table below are illustrative values chosen to show the mechanics of table lookups and tensor shapes. They are not trained model weights and do not prove real-world semantic similarity.

Our complete toy embedding table ($V=7, d=4$):

Token ID	Token	Feature 0	Feature 1	Feature 2	Feature 3	Full Vector
0	[PAD]	0.00	0.00	0.00	0.00	[ 0.00, 0.00, 0.00, 0.00]
1	The	0.42	-0.15	0.88	0.05	[ 0.42, -0.15, 0.88, 0.05]
2	cat	-0.73	0.61	0.12	0.94	[-0.73, 0.61, 0.12, 0.94]
3	sat	0.25	-0.80	0.33	-0.18	[ 0.25, -0.80, 0.33, -0.18]
4	dog	-0.68	0.55	0.19	0.87	[-0.68, 0.55, 0.19, 0.87]
5	on	0.10	0.02	-0.45	0.31	[ 0.10, 0.02, -0.45, 0.31]
6	mat	-0.30	0.18	0.52	-0.64	[-0.30, 0.18, 0.52, -0.64]

The Lookup Operation

The lookup is simply picking the row corresponding to the Token ID index:

• Token ID 1 (The) retrieves Row 1: [ 0.42, -0.15, 0.88, 0.05]
• Token ID 2 (cat) retrieves Row 2: [-0.73, 0.61, 0.12, 0.94]
• Token ID 3 (sat) retrieves Row 3: [ 0.25, -0.80, 0.33, -0.18]

---

Step 4: Building the Input Tensor & Labeling Dimensions

A tensor is a multi-dimensional grid or container of numbers. A scalar has 0 dimensions, a vector has 1 dimension, a matrix has 2 dimensions, and a 3D tensor has 3 dimensions.

The input tensor has 3 dimensions:

1. Batch size ($B$): How many independent text prompts are being processed together. For our single prompt, $B = 1$.
2. Sequence length ($T$): How many tokens are in the prompt. For "The cat sat", $T = 3$.
3. Embedding dimension ($d$): How many numbers represent each token. In our toy model, $d = 4$.

The resulting tensor shape is written as [Batch, Sequence_Length, Embedding_Dimension] or [B, T, d]:

[1, 3, 4]

The Complete 3D Tensor

[
  [
    [ 0.42, -0.15,  0.88,  0.05],  # Token 1 (ID 1: "The")
    [-0.73,  0.61,  0.12,  0.94],  # Token 2 (ID 2: "cat")
    [ 0.25, -0.80,  0.33, -0.18]   # Token 3 (ID 3: "sat")
  ]
]

Worked Calculation

Total numeric values stored in this tensor: $B \times T \times d = 1 \times 3 \times 4 = 12$ numbers.

---

Critical Concept: Static Embeddings vs. Contextual Representations

An embedding lookup is static.

In our lookup table, "cat" always gets [-0.73, 0.61, 0.12, 0.94], whether it appears in "The cat sat" or "cat on mat". The table alone does not know grammar, sentence structure, or who is doing what.

The transformer’s attention layers (covered in later lessons) take these static vectors and allow tokens to interact, creating contextual representations—vectors that change based on surrounding context.

---

Toy Tokenizer vs. Real Production Tokenizers

To build a solid intuition, we used a toy dictionary. Here is how real production tokenizers compare:

Aspect	Our Toy Tokenizer	Real Production Tokenizers
Vocabulary Size	7 tokens	GPT-2: 50,257 (Radford et al., 2019); GPT-4/tiktoken (cl100k_base): ~100,000; LLaMA 3: 128,256
Splitting Strategy	Whole words only	Subwords via algorithms such as Byte Pair Encoding (BPE; Sennrich et al., 2016). Unfamiliar words split into known subwords (for example, an illustrative split might divide "unhappiness" into ["un", "happi", "ness"]).
Tokenizer Library	Hardcoded script	Tools like SentencePiece (Kudo & Richardson, 2018), a tokenizer library that can train either BPE or Unigram models directly on raw text.
Spaces & Punctuation	Whole words without spaces	Real tokenizers preserve whitespace. SentencePiece marks whitespace with the meta-symbol (U+2581, e.g., cat). GPT-2 / tiktoken’s byte-level BPE maps spaces to Ġ via a byte-to-unicode table (OpenAI encoder.py).
Unknown Tokens	Fails or drops unknown words	GPT-2 / tiktoken byte-level BPE includes all 256 individual UTF-8 byte values as base tokens in its vocabulary. Any text string can be decomposed into raw bytes if no larger subword matches, guaranteeing no out-of-vocabulary ([UNK]) errors. In SentencePiece, byte fallback is a configuration option.
Special Tokens	[PAD] only	Dedicated control markers such as <|endoftext|> (GPT-2 / tiktoken), <s> and </s> (LLaMA / SentencePiece), or [CLS] and [SEP] (BERT) to mark document beginnings, endings, and boundaries.

---

Primary Source Citations

• Subword BPE in NLP: Sennrich, Haddow, & Birch (2016), Neural Machine Translation of Rare Words with Subword Units, ACL 2016 (ACL Anthology / arXiv:1508.07909).
• Transformer Embeddings ($V \times d$): Vaswani et al. (2017), Attention Is All You Need, NeurIPS 2017 (arXiv:1706.03762). Formalized embedding tables of size $V \times d$ (often referenced as d_model).
• Distributed Word Vectors: Mikolov et al. (2013), Distributed Representations of Words and Phrases and their Compositionality, NeurIPS 2013 (arXiv:1310.4546).
• Byte-level BPE in Practice: Radford et al. (2019), Language Models are Unsupervised Multitask Learners, OpenAI technical report (PDF) and GPT-2 encoder.py.
• SentencePiece Library: Kudo & Richardson (2018), SentencePiece: A simple and language independent subword tokenizer for Neural Text Processing, EMNLP 2018 (ACL Anthology / GitHub).

---

Check Your Understanding: Worked Exercises & Solutions

Exercise 1 (Tracing a new prompt)

Prompt: "dog sat on mat"

1. List the tokens.
2. List their Token IDs using the toy vocabulary table.
3. What is the resulting tensor shape [B, T, d]?
4. What is the total count of numbers in the tensor?

Solution:

1. Tokens: ["dog", "sat", "on", "mat"]
2. Token IDs: [4, 3, 5, 6] (dog=4, sat=3, on=5, mat=6)
3. Tensor shape: [1, 4, 4] ($B=1$, $T=4$, $d=4$)
4. Total numbers: $1 \times 4 \times 4 = 16$

---

Exercise 2 (The nature of Token IDs)

In our toy vocabulary, Token ID 4 is "dog" and Token ID 2 is "cat". A programmer writes code that calculates:

ID("dog") - ID("cat") = 4 - 2 = 2  ("cat")

Explain why arithmetic on Token IDs does not represent semantic relationships.

Solution:

Token IDs are integers used solely as embedding-table row indices. The number 4 has no semantic relationship to the number 2; their integer values simply reflect the arbitrary order in which tokens were listed in the dictionary. Subtracting them computes a difference in table row position, not a difference in real-world meaning. Doing math on token IDs would be like claiming that subtracting the phone number of a pet store from the phone number of a veterinary clinic yields the phone number of a cat.

Furthermore, note that the vectors in our toy table are illustrative numbers designed to demonstrate table lookups and tensor shapes; they do not prove real-world semantic relationships.

---

Exercise 3 (Static vectors across different prompts)

Consider the token "cat" in two different prompts:

• Prompt A: "The cat sat"
• Prompt B: "cat on mat"

1. What vector is retrieved from the embedding table for "cat" in Prompt A?
2. What vector is retrieved from the embedding table for "cat" in Prompt B?
3. Does the embedding table know whether "cat" is performing an action or sitting on an object?

Solution:

1. In Prompt A, "cat" has Token ID 2 and retrieves Row 2: [-0.73, 0.61, 0.12, 0.94].
2. In Prompt B, "cat" has Token ID 2 and retrieves Row 2: [-0.73, 0.61, 0.12, 0.94].
3. No. The embedding table lookup is purely static: it returns the exact same vector for Token ID 2 every single time, regardless of which prompt it appears in or its grammatical role in the sentence. Contextual awareness is created later when transformer attention layers allow tokens in the sequence to exchange information.

---

Exercise 4 (Handling unknown words)

If someone enters "The fish sat", what happens in our 7-word toy tokenizer? How does a production byte-level BPE tokenizer (such as GPT-2’s) handle "fish" without crashing?

Solution:

In our toy tokenizer, "fish" is not in the vocabulary. The tokenizer has no Token ID to assign, so it fails unless an explicit [UNK] (unknown) token is added.

In contrast, GPT-2 and tiktoken’s byte-level BPE includes all 256 individual UTF-8 bytes as base tokens in its vocabulary. If "fish" were not present as a whole word or common subword, the tokenizer would break it down into smaller known pieces or down to its individual UTF-8 byte tokens (["f", "i", "s", "h"]). Because any text string can be expressed as a sequence of bytes, byte-level BPE guarantees that the tokenizer can encode any input without ever hitting an out-of-vocabulary error.

---

Summary & What Comes Next

Pipeline recap: Raw text $\rightarrow$ Tokens $\rightarrow$ Token IDs $\rightarrow$ Embedding lookup $\rightarrow$ [B, T, d] tensor.

Next steps:

• Next lesson: From Logits to the Next Token
• Back to prerequisite: Your First LLM Walkthrough: From Prompt to Answer
• Track overview: Start Here: LLM Foundations
• Optional deeper study: If you want to see how production BPE merge rules and PyTorch matrices work at an intermediate level, read Tokenisation and Embeddings: From Text to Tensors.