高级Advanced可打印Printable点子Idea
自己的小语言模型：从计数到生成
Your Own Tiny Language Model: Count, Then Generate
下载一个只用 Python 标准库的小模型。改变上下文长度、训练语料和随机种子，比较没见过的句子上的表现。
Download a model using only the Python standard library. Change context length, training text and random seed, then test on unseen sentences.
试教点子：已提供活动、学习单和双语指南，尚未标为 Verified。可按学生情况分两次完成。
Teaching idea: activities, worksheets and bilingual guides are available. It is not yet marked Verified; split it into two sessions if helpful.
先造一个小到每个计数都能检查的语言模型，再决定为什么需要 Transformer。它按词计数，是 n-gram 模型；它不是聊天 AI，也不是 Transformer。
Build a language model small enough to inspect every count before asking why we need Transformers. This word-counting n-gram model is neither a chat AI nor a Transformer.
开始之前Before you start
实际运行一个词 n-gram 模型，在固定测试集上比较上下文长度并解释未见词。
Run a word n-gram model, compare context lengths on fixed held-out data and explain unseen words.
先学猜词、模型评估；本地需 Python 3.10+，不需第三方库或 GPU。
Complete Guess the Word and model evaluation; local Python 3.10+ is needed, without packages or a GPU.
一步一步试试看Try it step by step
- 先猜：手数“the cat naps”两次、“the cat plays”一次。只看 cat 后一个词，不平滑时 naps 的频率是多少？Predict: count “the cat naps” twice and “the cat plays” once. Without smoothing, how often does naps follow cat?
- 动手：在终端运行
python3 tiny_language_model.py --context 1 --seed 7。读词表、候选词概率、训练与测试每词负对数似然，再看生成句。Try: runpython3 tiny_language_model.py --context 1 --seed 7. Read the vocabulary, candidate probabilities and train/test negative log-likelihood before the sample. - 观察：改为
--context 2，保持语料、留出句和种子不变。更多上下文更稀疏，不保证损失更低。如果据此挑设置，这份留出数据就参与了验证；最终评价要另留新测试。Observe: use--context 2, keeping corpus, held-out sentences and seed fixed. More context is sparser and need not lower loss. If you select settings from this comparison, these held-out data serve as validation; reserve a fresh final test. - 挑战：用
--train my_train.txt --test my_test.txt，一行一句，先分好数据再看结果。只改种子不会重新学习，主要改变抽样结果。Challenge: use--train my_train.txt --test my_test.txt, one sentence per line. Split data before inspecting results. Changing only the seed changes sampling, not learned counts.
一起算一个例子Work through one example
不平滑时 P(naps|cat)=2/3。下载程序使用 add-one 平滑和训练词表，因此程序的概率还包含每个候选加 1 以及 <unk>、结束符的计数。
Without smoothing, P(naps|cat)=2/3. The program uses add-one smoothing and a training-only vocabulary, so its probabilities also include one extra count per candidate, <unk> and an end token.
先想一想，再看答案Think first, then check
生成句读起来很顺，能证明测试损失低或事实正确吗？把测试文本也拿去计数会怎样？
Does fluent output prove low test loss or factual truth? What happens if test text is counted during training?
查看参考答案Show the expected answer
都不能；只用训练集学词表和计数，测试词表外词映射到 <unk>。NLL 越低表示这组测试后续词的平均负对数概率更低（几何平均概率更高），不能当作真实知识准确率。重复查看测试来挑参数也会泄漏。 手算例中 plays 的概率为 1/3，未见后续词为 0，若测试命中该零概率，NLL 无穷大。程序内置 12 句比手算多：cat 后 naps/plays/runs 的计数为 3/2/1，词表大小 11；平滑后 P(naps|cat)=4/17≈0.2353。可查看 TRAIN 逐项核对。
Neither. Learn vocabulary/counts only from training and map unseen test words to <unk>. Lower NLL means lower average negative log probability (higher geometric-mean probability) on those test continuations, not factual accuracy. Repeatedly selecting parameters against test data also leaks information. In the hand example, plays has probability 1/3 and an unseen continuation zero; a zero-probability test event gives infinite NLL. The built-in 12 sentences differ: after cat, naps/plays/runs count 3/2/1. Vocabulary size 11 gives smoothed P(naps|cat)=4/17≈0.2353. Inspect TRAIN to check each count.
这个实验说明了什么What this experiment shows
这是计数型 n-gram 基线，不是神经网络。没有下载外部模型、联网推理或 GPU。下一步的本地 Transformer 训练见页面进阶点子，需另行准备依赖和更完整评估。
This is a count-based n-gram baseline, not a neural network. It downloads no external model, makes no inference requests and uses no GPU. A local Transformer is a further project requiring dependencies and fuller evaluation.
更多课堂玩法与原理More classroom activities and explanations
接下来可做的进阶项目（尚未实现）Further project ideas (not implemented yet)
- 本地小 Transformer：用自己有权使用的语料，先实现一个很短上下文、少量参数的模型。对照 n-gram，在同样的独立测试上比较每词损失、参数量、内存和用时；运算较大时在本地运行。A local tiny Transformer: use text you may use, short contexts and few parameters. Compare against n-grams on the same held-out task, tracking token loss, parameter count, memory and runtime. Run larger experiments locally.
- 先检索再回答：从自己写的短文章中检索相关段落，要求每个答案指回原文；加入“资料里没有答案”的题，检查能否承认不知道。Retrieve before answering: search short original articles and require answers to point to source passages. Include questions with no answer in the material and check whether the system abstains.
- 从硬币游戏到强化学习：让一个表格型智能体从对局更新动作价值，再和已知的倒推策略比较；多换几个随机种子，分别记录训练对手和测试对手。From coins to reinforcement learning: update tabular action values from games and compare against the known backward solution. Repeat with several seeds and keep training opponents separate from test opponents.
学习单Worksheets
学习单用英文，可打印给孩子。教师指南随页面语言下载：当前为中文版。
Print the English worksheets for the children. The guide follows the page language: English is selected.
下载学习单（PDF）Download worksheets (PDF)教师指南（中文 PDF）Teacher guide (English PDF)
下载Downloads
视频与文章Videos and reading
- Jurafsky & Martin: N-gram Language Models 教师参考：上下文、平滑与测试teacher reference: context, smoothing and evaluation