Q9 — AWS AIF-C01 Ch.2

Question 9 of 100 | ← Chapter 2

In the context of generative AI models, what are tokens?

• A. Tokens are the fundamental units of input and output processed by generative AI models; they can represent words, subwords, or other linguistic units. ✓
• B. In generative AI models, tokens are mathematical representations of words or concepts.
• C. Tokens refer to pre-trained weights in generative AI models that are fine-tuned for specific tasks.
• D. Tokens are specific prompts or instructions given to a generative AI model to produce output.

Correct Answer: A. Tokens are the fundamental units of input and output processed by generative AI models; they can represent words, subwords, or other linguistic units.

Explanation

In generative AI models, tokens are the basic units of text that the model processes as input and generates as output. Tokenization breaks down raw text into discrete elements—such as whole words, subwords (e.g., 'unhappiness' → 'un', 'happi', 'ness'), or characters—enabling efficient model training and inference. Tokens serve as the atomic building blocks for attention mechanisms and sequence modeling. Option A correctly defines tokens in this context, whereas options B, C, and D mischaracterize tokens as embeddings, weights, or prompts—none of which align with the standard definition.