Week 9 — Inverse RL, Social Cognition & POMDPs第9週 — 逆強化学習・社会的認知・POMDP

Friday, June 26, 20262026年6月26日（金）

Prof. Joseph Austerweilオウステウェイル　ジョセフ教授

Run the camera backwardsカメラを逆回しする

Opening section break, clock 0:00. The title for the whole week: we’re going to “run the camera backwards.” Last week we filmed a goal turning into actions (forward RL); this week we play that film in reverse to recover the goal/belief/reward. Set the tone in one line, then go to admin — don’t elaborate yet, the pivot slide does that.

Admin — deadlines連絡 — 締め切り

• ★ Final-project proposal due Sunday June 28, 8:00 PM (pass/fail) — ~1 page: background, your question, method, ≥3 references.
• Assignment 4 (Reinforcement Learning) is out — due Fri Jul 10, 8 PM (Q-learning on the GardenPath grid).
• Monte Carlo assignment also due Fri Jul 10, 8 PM — pace your work around the proposal.
• Reflections: you need 5 of 11 (Weeks 2–12). After today, 3 chances remain — Weeks 10, 11, 12. Check you’re on track.

• ★ 最終プロジェクト提案書：6月28日（日）20:00締切（合否）— 約1ページ：背景・問い・手法・参考文献3件以上。
• 課題4（強化学習）公開中 — 締切 7月10日（金）20:00（GardenPathグリッド上のQ学習）。
• モンテカルロ課題も締切 7月10日（金）20:00 — 提案書を挟んでペース配分を。
• リフレクション： 11週中5回必要（第2〜12週）。今日以降、残り3回 — 第10・11・12週。進捗を確認しよう。

Welcome block, clock 0:00 — we have ~7 min here. Lead with the headline: the final-project proposal is due Sunday June 28, 8 PM, two days out — make sure everyone has a question and three references lined up. Note the RL assignment (Assignment 4) and Monte Carlo are BOTH due Jul 10, 8 PM, so they should pace work around the proposal this weekend. Reflections: students need 5 of 11 (Weeks 2–12); by class time Week 9’s window has closed, so only 3 chances remain (Weeks 10–12) — nudge anyone behind to plan ahead. Keep it to two minutes; don’t linger.

Next week is async — AI Safety workshop来週は非同期 — AIセーフティ・ワークショップ

• Week 10 (Fri Jul 3) runs asynchronously — no in-person session.
• There’s an AI Safety workshop that week — you’re warmly invited if it interests you. Heads up: it runs fully in English.
• I’ll record the lecture (Bayesian nonparametrics) and post a link to the recording — watch it on your own time.
• The weekly discussion thread still runs — post and reply as usual (Week 10 is reflection-eligible).

• 第10週（7月3日・金）は非同期 — 対面授業はありません。
• その週にAIセーフティのワークショップがあります — 興味があればぜひご参加を。注：全て英語で行われます。
• 講義（ベイズ・ノンパラメトリクス）を録画し、録画リンクを共有します — 各自のペースで視聴を。
• 毎週のディスカッションスレッドは通常通り — いつものように投稿・返信を（第10週はリフレクション対象）。

Logistics for next week — flag it clearly so nobody shows up to an empty room. Week 10 (Jul 3) is ASYNC because there’s an AI Safety workshop that week worth attending; students are invited if interested, but warn them it’s entirely in English. You’ll record the Bayesian-nonparametrics lecture and post the link; the Slack discussion thread runs as normal and Week 10 still counts as one of their three remaining reflection weeks. ~30 seconds; point them to the thread for the workshop details/link.

From acting to reading minds行動から心を読むへ

Last week we ran reinforcement learning forward: a goal becomes a policy becomes actions.

This week we run the same machinery backward — watch the actions, infer the goal, belief, or reward behind them. That inversion is inverse RL, and when the agent is a person, it is the computational story of social cognition.

先週は強化学習を順方向に動かした：目標 → 方策 → 行動。

今週は同じ仕組みを逆方向に動かす — 行動を観察し、その背後の目標・信念・報酬を推論する。この反転が逆強化学習であり、相手が人間のときそれは社会的認知の計算論的な物語そのものだ。

This is the pivot slide — the spine of the whole week. Last week we ran RL FORWARD: a goal became a policy became actions, and the students literally coded that (Q-learning on GardenPath). Today we run the SAME machinery BACKWARD: watch the actions, infer the hidden goal/belief/reward. Say the name-drop out loud — “this inversion is inverse RL, and when the agent is a person, it IS the computational theory of social cognition” (Jara-Ettinger 2019). Point at the two columns of the figure: forward (goal→π→actions) on the left, inverse (actions→goal) on the right. Everything today is this one arrow reversed, applied to a richer hidden variable each block.

The one equation for the whole week今週を貫く一つの式

\[P(\text{goal}\mid\text{actions}) \;\propto\; \underbrace{P(\text{actions}\mid\text{goal})}_{\text{a policy, in reverse}}\;\cdot\;P(\text{goal})\]

The likelihood P(actions | goal) is just a policy — how a goal-seeker would act. Last week we worked out how good each action is for a goal; this week we assume the agent mostly takes the good actions, but slips up now and then — and we read that backward. Everything today is this one inversion, on a richer hidden variable each time: goal → belief → reward.

尤度 P(行動 | 目標) は単なる方策 — 目標を目指す者の動き方だ。先週は各行動が目標に対してどれだけ良いかを求めた。今週はエージェントがたいてい良い行動を取るが、時々ミスをすると仮定し、それを逆向きに読む。今日はこの一つの反転を、隠れ変数を目標 → 信念 → 報酬と豊かにしながら適用する。

The single equation the whole deck hangs on — write it big and say it once slowly. The key move to land: the likelihood P(actions | goal) is JUST a policy — exactly what we computed last week (value iteration → Q-values). The one new ingredient this week is making the agent noisily rational: wrap Q in a softmax with temperature β, so mistakes are possible and the inference has something to chew on. Promise them the punchline now: today we re-run this exact inversion on goal, then belief, then reward — same equation, richer hidden variable each time. Don’t derive anything yet; that’s the next block.

The agent model — a POMDPエージェントのモデル — POMDP

Every agent today fits one picture: a goal/reward \(R\) drives a policy \(\pi\) that picks an action \(a\); the world has a hidden state \(s\) the agent sees only through a noisy observation \(o\), so it tracks a belief \(b(s)\). That is a POMDP — a partially observable MDP. Week 8 ran it forward (every piece known → behavior); all week we invert it — watch behavior, infer one hidden piece.

今日のエージェントは全て一枚の図に収まる：目標＝報酬 \(R\) が方策 \(\pi\) を動かし行動 \(a\) を選ぶ。世界には隠れた状態 \(s\) があり、エージェントはノイズのある観測 \(o\) を通してしか見えないので、信念 \(b(s)\) を保持する。これが POMDP（部分観測MDP）だ。先週は順方向（全要素既知→行動）、今週はこれを反転する — 行動を見て隠れた一要素を推論する。

The master frame — the single most important slide of this redesign. Trace the loop on the figure: reward R → policy π → action a (the only OBSERVED piece, green); the world state s is hidden (red); the agent sees it only through observation o and tracks belief b(s). Define the acronym at first use: POMDP = Partially Observable Markov Decision Process — an MDP where the agent doesn’t see the true state. The line to land: Week 8 ran this FORWARD (all pieces known → behavior); all of Week 9 is the INVERSE — observe behavior, infer ONE hidden piece. Tell them this diagram returns every block with the piece we’re inferring lit up — so they expect it. No Bayes math yet (next slides); this is the map.

A map of the unknowns未知の地図

The rest of the day is this one model with a different piece hidden each time:

• reward \(R\) hidden, world seen → goal inference / IRL (next)
• belief \(b\) hidden, possibly false → theory of mind, POMDPs
• reward \(R\) at scale → MaxEnt IRL, RLHF
• the observer’s belief is what you steer → teaching / showing
• two agents, one shared hidden \(R\) → alignment

Same inversion, a richer unknown each time — we point at the lit box, we don’t re-derive it.

今日の残りは同じモデルで、毎回隠す要素を変えるだけ：

• 報酬 \(R\) が隠れ、世界は見える → 目標推論 / 逆強化学習（次）
• 信念 \(b\) が隠れ、時に誤り → 心の理論、POMDP
• 大規模な報酬 \(R\) → 最大エントロピーIRL、RLHF
• 観察者の信念をあなたが操る → 教示 / 見せる
• 二者が一つの隠れた \(R\) を共有 → アラインメント

同じ反転、毎回より豊かな未知 — 光る箱を指すだけで、導出し直さない。

The itinerary — “one model, five unknowns.” This is the slide that fixes the “no unifying framework / it kept repeating” complaint: each block is the SAME diagram with a different box lit. Say it out loud — we establish the Bayes inversion ONCE (next block), then every later block only changes which latent is hidden; we point at the lit box instead of re-deriving Bayes. Keep it quick; it’s a signpost.

Agenda — 2 hours本日の予定 — 2時間

• Run the camera backwardsカメラを逆回しする 0:00
• Goal inference as inverse planning逆プランニングとしての目標推論 0:07
• What is theory of mind?心の理論とは？ 0:37
• POMDPs — inferring beliefsPOMDP — 信念を推論する 0:45
• Break休憩 1:07
• Inverse RL — recovering rewards逆強化学習 — 報酬の復元 1:12
• Teaching, flipped教示の反転 1:26
• Alignment & LLM Theory of MindアラインメントとLLMの心の理論 1:38
• Close & Week 10まとめと第10週 1:52

The full 2-hour map. Highlighted row is where we are (Welcome, 0:00). Five-beat arc: goal → ToM → belief → teaching → alignment, each re-running the same inversion. Flag the break at 1:07. Don’t read every line — just orient them: “two halves, break in the middle, the back half is teaching and the modern alignment tail.” Move on quickly.

Goal inference as inverse planning逆プランニングとしての目標推論

Section break into Block 2, the longest concept block (~20 min, clock 0:07). This is where we actually build the inversion — Baker, Tenenbaum & Saxe 2007 (the assigned reading) and Baker, Saxe & Tenenbaum 2009. Transition line: “we have the one equation; now let’s actually invert an MDP and recover a goal from a path.”

Where we areここまでの流れ

• Run the camera backwardsカメラを逆回しする 0:00
• Goal inference as inverse planning逆プランニングとしての目標推論 0:07
• What is theory of mind?心の理論とは？ 0:37
• POMDPs — inferring beliefsPOMDP — 信念を推論する 0:45
• Break休憩 1:07
• Inverse RL — recovering rewards逆強化学習 — 報酬の復元 1:12
• Teaching, flipped教示の反転 1:26
• Alignment & LLM Theory of MindアラインメントとLLMの心の理論 1:38
• Close & Week 10まとめと第10週 1:52

Recap slide — brief. We’ve done the welcome/pivot (~7 min in); now entering goal inference as inverse planning. One sentence and move on.

Recall: the forward model復習：順方向モデル

Week 8’s forward solve: value iteration turns a reward into a value, then a policy. This is the machine we now run backward.先週の順方向：価値反復が報酬を価値に、そして方策に変える。これを今から逆に回す。

Callback widget (cut if behind) — a 30-second re-grounding before we invert. This is the Week-8 forward model: value iteration maps a reward → values → a greedy/softmax policy. Say “today we watch the actions and run THIS backward to recover the reward.” It makes the inversion concrete and reuses a widget the students already saw. Fallback: skip it. Then notation.

Notation lock-in記法の確認

• trajectory \(\tau\) — the path of states and actions we observe
• goal \(g\) — the hidden thing we infer; reward \(R\), cost \(C\)
• posterior \(P(g \mid \tau)\) — our belief over goals after seeing \(\tau\)
• likelihood \(P(\tau \mid g)\) — how a \(g\)-seeking agent would move
• rationality \(\beta\) — how decisively the agent picks the best-valued action (\(\beta\to\infty\) greedy / pure exploit, \(\beta\to 0\) random); formula next slide
• carried from Week 8: state \(s\), action \(a\), policy \(\pi\), value \(Q\), discount \(\gamma\)

• 軌跡 \(\tau\) — 観察する状態と行動の系列
• 目標 \(g\) — 推論したい隠れ変数；報酬 \(R\)、コスト \(C\)
• 事後分布 \(P(g \mid \tau)\) — \(\tau\) を見た後の目標についての信念
• 尤度 \(P(\tau \mid g)\) — 目標 \(g\) を目指すエージェントの動き方
• 合理性 \(\beta\) — 価値最大の行動をどれだけ決然と選ぶか（\(\beta\to\infty\) で貪欲＝純粋な活用、\(\beta\to 0\) でランダム）；式は次のスライド
• 先週から：状態 \(s\)、行動 \(a\)、方策 \(\pi\)、価値 \(Q\)、割引 \(\gamma\)

This is the heaviest-notation week after Week 8 — the CLAUDE.md rule is to define every new symbol at first use, so this is the lock-in box. Walk the new symbols slowly: τ (trajectory = path of states+actions we observe), g (the hidden goal), the posterior P(g|τ) and the likelihood P(τ|g), and especially β, the rationality knob (its softmax formula lands two slides on, in Build the inversion) — emphasize the two limits: β→∞ is greedy/pure exploit, β→0 is random. The rest (s, a, π, Q, γ) carries over from last week; just remind them. Tell students this box is a reference they can glance back at; don’t make them memorize it.

The inversion is Bayes’ rule反転はベイズの定理

Posterior (what we infer) \(\propto\) likelihood (how a goal-seeker acts — a softmax policy, read in reverse) \(\times\) prior (over candidate goals). Next: build each piece.

事後分布（推論したいもの）\(\propto\) 尤度（目標を追う者の行動 — ソフトマックス方策を逆に読む）\(\times\) 事前分布（候補目標について）。次に各要素を組み立てる。

The high-level anatomy before the recipe: goal inference IS Bayes’ rule. Point at the color-coding — posterior (yellow, what we infer) ∝ likelihood (blue) × prior (green). The one thing to stress: the LIKELIHOOD is a policy — “how a goal-seeker acts,” i.e. last week’s softmax-over-Q run in reverse. That’s the whole trick; the next slide makes each of the four pieces concrete.

Build the inversion反転を組み立てる

1. Forward — value iteration (Week 8) gives the \(Q\)-values \(Q_g\); wrap them in a softmax (best action most likely, the rest still possible) → a noisy-rational policy \(\pi_g(a\mid s)\propto \exp(\beta\,Q_g(s,a))\) (new this week; one per candidate goal)
2. Likelihood — \(P(\tau\mid g)=\prod_t \pi_g(a_t\mid s_t)\)
3. Prior — \(P(g)\) over candidate goals
4. Posterior — \(P(g\mid\tau)\propto P(\tau\mid g)\,P(g)\)

1. 順方向 — 価値反復（先週）が \(Q\) 値 \(Q_g\) を与える；それをソフトマックス（最良の行動が最も起こりやすいが、他もあり得る）で包む → ノイズのある合理的方策 \(\pi_g(a\mid s)\propto \exp(\beta\,Q_g(s,a))\) （今週から；候補目標ごとに一つ）
2. 尤度 — \(P(\tau\mid g)=\prod_t \pi_g(a_t\mid s_t)\)
3. 事前分布 — 候補目標についての \(P(g)\)
4. 事後分布 — \(P(g\mid\tau)\propto P(\tau\mid g)\,P(g)\)

The four-step recipe — this is the conceptual core of the block; take your time. Step 1 is the only new compute: last week’s value iteration gives Q-values per candidate goal, and we wrap each in a softmax to get a noisy-rational policy π_g — one forward solve PER hypothesized goal. Steps 2–4 are just Bayes: multiply the per-step policy probabilities into a likelihood, multiply by the prior, normalize to the posterior. Emphasize “the likelihood is a policy” — that is the whole trick (it’s the blue piece of the Bayes-rule anatomy we just saw). Then promise: “now let’s watch this run.”

In GenJAX — invert the MDPGenJAXで — MDPを反転する

LOGITS = beta * Q_per_goal              # β·Q logits; categorical softmaxes them → the policy
@gen
def observer(states):                   # observed actions = the data
    g = categorical(jnp.log(goal_prior)) @ "goal"     # the hidden goal
    for t in range(T):
        categorical(LOGITS[g, states[t]]) @ f"a_{t}"  # softmax policy of g
post = normalize([exp(observer.assess(cm(g, acts), (states,))[0]) for g in GOALS])

The hidden variable we recover is “which MDP are we in?”復元する隠れ変数こそ「どのMDPにいるか」である。

The GenJAX code slide — the recipe in ~7 runnable lines. Point at the three things: LOGITS = β·Q (the policy is a softmax over Q-values from last week’s solver); the @gen model draws the hidden goal g, then samples each action from the softmax policy of g; and post enumerates the candidate goals and normalizes the assessed scores into the posterior. The backbone is genjax_goal_inference.py in this directory — verified to run (genjax 0.10.3) — so you CAN open it and execute it live if the room is engaged. Land the tagline on the slide: the hidden variable we recover IS “which MDP are we in?” — that’s the unifying frame for the whole week.

Watch it infer — live推論を実演 — ライブ

Drive the agent (arrow keys / 🎮) — the posterior over goals updates each step; drag β to feel the rationality assumption bite.エージェントを操作（矢印キー / 🎮）— 各ステップで目標の事後分布が更新。β を動かすと合理性の仮定が効く。

WIDGET A — live goal inference (Baker’s “freeze the animation” paradigm made interactive). Demo script: start with the agent stepping under its softmax policy and watch the posterior bars over goals update each step — point out it’s AMBIGUOUS early (mid vs right tie around 0.50) and SHARPENS as the agent commits; the verified freeze-frame sequence on “right” is 0.37 → 0.47 → 0.50 → 0.54. Then drag the β (rationality) slider: near-random β flattens the posterior to ~0.33 each, high β sharpens to ~0.72 on the true goal — “the inference is only as good as the rationality assumption.” NEW: there’s a “🎮 you drive” mode — switch to it and steer the agent with the arrow keys / WASD; the posterior updates live as the audience watches you betray your goal by where you head. If the widget fails to load, the static fallback is goal-inference-posterior.png. Keep it to ~2 min.

The inversion is ill-posed反転は不良設定問題

A few steps fit many goals. The single confident answer comes from the prior + the rationality assumption — it’s a posterior, not a unique solution.

Baker, Saxe & Tenenbaum (2009): the changing-goal model tracks human goal judgments at r = 0.97 (vs 0.82 single-goal).

数ステップは多くの目標に当てはまる。一つの確信ある答えは事前分布と合理性の仮定から来る — 唯一解ではなく事後分布だ。

Baker, Saxe & Tenenbaum (2009)：目標が変わるモデルは人間の判断を r = 0.97 で説明（単一目標は0.82）。

The crucial caveat — don’t skip it. A few steps fit MANY goals; the single confident answer is NOT a unique solution, it’s a posterior that the prior plus the rationality assumption disambiguate (Baker, Saxe & Tenenbaum 2009: “the inverse problem is ill-posed; its solution requires strong prior knowledge”). Common student confusion to preempt: they’ll think the model “figured out” the goal — correct them, it WEIGHTED multiple explanations by fit. The r=0.97 number is the payoff: the changing-goal model tracks human frame-by-frame goal judgments at r=0.97 (Experiment 1), versus 0.82 for a single-fixed-goal model — say “modeling that goals can CHANGE is what fits people best.” Tee up the next slide: people infer not just the goal but the cost-reward tradeoff behind it.

People reason about cost vs reward人はコストと報酬で推論する

Naive utility calculus (Jara-Ettinger et al. 2016): we assume agents maximize reward minus cost, and we infer both from behavior — present from infancy through adulthood. Inferring the reward is the same move as inferring the goal.

Watch Chibany trek all the way across town for a bento box and open it beaming — you infer two things at once: the trip was costly, so the contents must have been worth it. You’d be surprised if it weren’t his beloved tonkatsu.

素朴効用計算（Jara-Ettinger et al. 2016）：私たちはエージェントが報酬からコストを引いた値を最大化すると仮定し、行動からその両方を推論する — 乳児期から成人まで存在する。報酬の推論は目標の推論と同じ操作だ。

チバニーが弁当のために町の反対側まで歩き、満面の笑みで開けるのを見れば、二つを同時に推論する：その道のりはコストが高く、だから中身はそれに見合う価値があったはず。大好きなとんかつでなければ驚くだろう。

Naive utility calculus (Jara-Ettinger, Gweon, Schulz & Tenenbaum 2016): we assume agents maximize reward MINUS cost, and we infer BOTH from behavior — and this runs from infancy through adulthood. The Chibany bento example is the intuition pump: a long costly trip + a beaming face → you infer the contents were worth it (you’d be surprised if it weren’t his beloved tonkatsu). The one thing to emphasize: inferring the reward is the SAME inversion move as inferring the goal — just a richer latent. This is the bridge from the cog-sci ToM material to machine reward-recovery (Widget D later). Then run the poll.

Poll — which is not inverse RL?ポール — 逆強化学習でないのは？

• A. Find an optimal policy by acting in an MDP
• B. Learn a reward from an agent’s behavior
• C. Learn an agent’s preferences from behavior
• D. Learn an MDP’s transitions from behavior

• A. MDPの中で行動して最適方策を求める
• B. 行動から報酬関数を学習する
• C. 行動から選好を学習する
• D. 行動から遷移関数を学習する

POLL 1 (commit before reveal). Source: SP25 “Social Cognition and Inverse Reinforcement Learning” quiz, Q2. Have everyone commit to a letter before you reveal. The likely split: B and D draw a few votes from students who hear “learn from behavior” and pattern-match to “inverse.” The misconception this surfaces: confusing the FORWARD problem (acting in an MDP to find a policy — last week) with the INVERSE problem (recovering something hidden from behavior). Hold the reveal until everyone has committed.

Poll — revealポール — 解答

A — that’s forward RL.A — それは順強化学習。

Acting in an MDP to find a policy is the forward problem (Week 8). B, C, D all recover something hidden from behavior — that’s the inverse problem.

MDPの中で行動して方策を求めるのは順問題（先週）。B・C・Dはすべて行動から隠れたものを復元する — それが逆問題だ。

Reveal: A — that’s FORWARD RL (acting in an MDP to find a policy = Week 8’s problem). One-line justification: B, C, D all recover something hidden FROM behavior — that’s the inverse problem. Source: SP25 IRL quiz Q2. Reinforce the forward/inverse distinction one more time, since the rest of the day depends on it, then move into the softmax / noisy-rationality detail.

Noisy rationalityノイズのある合理性

\(\pi(a\mid s)\propto \exp(\beta\,Q(s,a))\) — favors high-\(Q\) actions, imperfectly. Take the rightward path (→ → ↑ ↑) from the goal-inference demo: at high \(\beta\) it’s decisive evidence the goal is top-right; at \(\beta\to 0\) (a random agent) the same path says nothing.

So when an agent blunders or detours, the model rationalizes it by inferring a weirder goal — or, soon, an unseen obstacle. “Assume rationality, blame the goal.”

\(\pi(a\mid s)\propto \exp(\beta\,Q(s,a))\) — 高\(Q\)の行動を好むが完璧ではない。目標推論デモの右向きの経路（→ → ↑ ↑）：高い \(\beta\) では目標が右上だという決定的な証拠、\(\beta\to 0\)（ランダム）では同じ経路が無意味。

だからエージェントが失敗・遠回りすると、モデルはより奇妙な目標を — まもなく見えない障害物を — 推論して辻褄を合わせる。「合理性を仮定し、目標のせいにする」。

This slide carries Block 3 (noisy rationality) — under time pressure it folds into the build-the-inversion slide, but it deserves its own beat if there’s room (~0:27 mark). The core idea: the softmax exponent β is the rationality knob — the SAME path is decisive evidence at high β and useless at β→0. The big payoff phrase, say it out loud: “Assume rationality, blame the goal.” When an agent blunders or detours, the model doesn’t conclude it’s irrational — it rationalizes the detour by inferring a WEIRDER goal, or (foreshadow Block 5) an unseen obstacle. This is the exact mechanism BToM will reuse for false beliefs, so plant it here. If a student objects “but maybe the agent just had wrong info,” that’s the perfect lead-in to POMDPs — acknowledge and say “hold that, it’s the next beat.”

What is theory of mind?心の理論とは？

Section break into Block 4 (~8 min, clock 0:37). Short transition: “we’ve shown the machinery — now the claim about minds.” This block names the thesis: Theory of Mind IS inverse RL.

Where we areここまでの流れ

• Run the camera backwardsカメラを逆回しする 0:00
• Goal inference as inverse planning逆プランニングとしての目標推論 0:07
• What is theory of mind?心の理論とは？ 0:37
• POMDPs — inferring beliefsPOMDP — 信念を推論する 0:45
• Break休憩 1:07
• Inverse RL — recovering rewards逆強化学習 — 報酬の復元 1:12
• Teaching, flipped教示の反転 1:26
• Alignment & LLM Theory of MindアラインメントとLLMの心の理論 1:38
• Close & Week 10まとめと第10週 1:52

Recap — brief. Goal inference and noisy rationality are done (~37 min in); next is the ToM = IRL framing. One sentence, keep moving.

Theory of mind: the false-belief test心の理論：誤信念テスト

Theory of mind = attributing mental states — beliefs, desires, goals — to others (Premack & Woodruff 1978).

The classic probe is the false-belief task — Sally-Anne (Baron-Cohen, Leslie & Frith 1985):

Sally hides her marble in a basket and leaves. Anne moves it to a box. Sally returns. Where will Sally look?

Passing means representing a belief that differs from reality (she looks in the basket). Most children pass around age 4.

心の理論 = 他者に心的状態 — 信念・欲求・目標 — を帰属させること（Premack & Woodruff 1978）。

古典的な検査が誤信念課題 — サリーとアン（Baron-Cohen, Leslie & Frith 1985）：

サリーがビー玉をかごに隠して出て行く。アンが箱に移す。サリーが戻る。サリーはどこを探す？

正解は、現実と異なる信念を表現すること（かごを探す）。多くの子は4歳頃に通過する。

Open the block by DEFINING theory of mind — the first delivery never did, which is the root of complaints 4 and 7. ToM = attributing mental states to others (Premack & Woodruff 1978, “Does the chimpanzee have a theory of mind?”). The canonical test is the false-belief task — Sally-Anne (Baron-Cohen, Leslie & Frith 1985). POSE it and have the room answer “where will Sally look?” out loud before any theory. The correct answer is the basket (the OLD location) — it requires holding Sally’s belief separate from reality. Adults find it trivial; that triviality is the point. Next: a real child being tested. We need this precise definition again for the LLM debate.

Watch: the Sally-Anne task映像：サリーとアン課題

Watch when the child’s answer flips from “the box” (where it really is) to “the basket” (where Sally thinks it is).子どもの答えが「箱」（実際の場所）から「かご」（サリーがいると思う場所）へ切り替わる年齢に注目。

~2 min. Kaltura clip (UW-Madison media; short link https://go.wisc.edu/zz79z8). Plays inline in the HTML deck; in the PPTX export it won’t embed — open the short link in a browser for the PPTX backup. Play ~1-2 min, then discuss: young children answer “box” (reality) because they can’t yet hold Sally’s false belief separate from their own knowledge.

Reading a mind is inverse planning心を読むことは逆プランニング

How does mature theory of mind work? Baker, Saxe & Tenenbaum (2009) — action understanding as inverse planning:

• forward: beliefs + goals → action (planning)
• ToM: action → beliefs + goals (inverse planning)

Theory of mind is your own decision theory, run backwards on someone else — reviewed as “ToM = inverse RL” (Jara-Ettinger 2019).

成熟した心の理論はどう働く？ Baker, Saxe & Tenenbaum (2009) — 逆プランニングとしての行動理解：

• 順方向：信念 + 目標 → 行動 （プランニング）
• 心の理論：行動 → 信念 + 目標 （逆プランニング）

心の理論とは、自分の意思決定理論を他者に逆向きに走らせること — 「ToM = 逆強化学習」と総説（Jara-Ettinger 2019）。

The computational answer, AFTER ToM is defined behaviorally. The framework is Baker & Tenenbaum’s (Baker, Saxe & Tenenbaum 2009, “action understanding as inverse planning”) — forward = simulate the planner, inverse = invert it to recover beliefs + goals. The unifying line: theory of mind is your own decision theory run BACKWARDS on someone else — literally the same inversion we did for goals last block, now the latent is a whole mind. Jara-Ettinger 2019 is the recent REVIEW that names it “ToM = inverse RL” — cite as the review, not the originator. This is the seam between the cog-sci ToM material and the machine IRL after the break.

The machine is old; the evidence is new仕組みは古い、新しいのは証拠

“Infer preferences from choices, assuming utility maximization” is just revealed preference — economics run backwards. The theory isn’t new. What is striking is the empirical reach:

• Toddlers infer others’ costs and rewards from a single choice (Jara-Ettinger et al. 2016; Liu et al. 2017).
• Inverse-planning models quantitatively predict adults’ moment-to-moment belief and desire attributions (Baker et al. 2017).

「効用最大化を仮定して選択から選好を推論する」は顕示選好 — 経済学を逆に回しただけ。理論は新しくない。新しいのは実証の射程：

• 幼児でさえ、一度の選択から他者のコストと報酬を推論する（Jara-Ettinger et al. 2016；Liu et al. 2017）。
• 逆プランニングモデルは成人の刻々の信念・欲求の帰属を定量的に予測する（Baker et al. 2017）。

~1.5 min. Honest framing (per the source deck): the naive-utility-calculus THEORY is essentially revealed preference / utility maximization run as inference — say so plainly, it builds credibility. The real contribution is EMPIRICAL: infants/toddlers already do it (Liu, Ullman, Tenenbaum & Spelke 2017, Science — infants infer a goal’s value from the cost an agent pays), and quantitative fits to adult judgments (Baker et al. 2017, Nat Hum Behav). Under time pressure this folds into the previous slide.

A harder test: the faux pasより難しいテスト：失言（フォーパ）

A faux pas: someone says something they don’t realize they shouldn’t — and it lands badly. Recognizing one needs two mental states at once:

• the speaker holds a false belief (they don’t know the hurtful fact), and
• the listener knows what the speaker doesn’t, so the remark stings.

A faux pas is a nested false belief — harder than Sally-Anne, passed later (~age 9–11). Hold this exact structure; the LLM debate turns on it.

失言（フォーパ）：本人がまずいと気づかずに言ってしまい、相手を傷つける発言。見抜くには二つの心的状態を同時に必要とする：

• 話し手は誤信念を持つ（傷つける事実を知らない）、かつ
• 聞き手は話し手が知らないことを知っているから刺さる。

失言は入れ子の誤信念 — サリーとアンより難しく、通過は遅い（約9〜11歳）。この構造をLLMの議論で使う。

NEW slide — the first delivery hand-waved “faux pas” without defining it, which is exactly why the LLM debate fell flat. Define it cleanly: recognizing a faux pas = recognizing a speaker said something inappropriate WITHOUT realizing it. It requires representing TWO minds at once — the speaker’s false belief (they don’t know) AND the listener’s knowledge (they do) — a nested / second-order false belief, harder than Sally-Anne, passed later (~9-11; Baron-Cohen et al. 1999 faux-pas test). Plant the structure now: in the LLM block we ask whether GPT-4 represents THIS structure or pattern-matches it.

Theory of mind & autism — and a reframe心の理論と自閉症 — そして再考

Sally-Anne was built to study autism (Baron-Cohen et al. 1985, “Does the autistic child have a theory of mind?”): autistic children passed control questions but failed the false-belief question more often — launching the “mindblindness” hypothesis.

Milton (2012), the double-empathy problem reframes it: the mismatch is bidirectional — neurotypical people are no better at reading autistic minds than the reverse; autistic-to-autistic communication is as effective as neurotypical-to-neurotypical.

Not a broken module — two differently-tuned inverse planners reading each other.

サリーとアンは自閉症研究のために作られた（Baron-Cohen et al. 1985）：自閉症児は統制質問には答えたが誤信念質問により多く不正解 — 「マインドブラインドネス」仮説を生んだ。

Milton (2012)の二重共感問題が再考を迫る：ミスマッチは双方向 — 定型発達者が自閉症の心を読むのも、その逆と同程度に難しい。自閉症者同士の意思疎通は定型発達者同士と同じくらい効果的。

壊れたモジュールではなく、互いを読み合う異なる調整の逆プランナー二つ。

~3 min. Present the historical mindblindness result faithfully first — careful language: “more often,” NOT “all autistic people lack ToM.” Then the double-empathy reframe (Milton 2012): the gap is bidirectional and symmetric — neurotypicals are no better at reading autistic minds. Tie to the spine: two differently-tuned inverse planners, each with a mis-calibrated prior over the other. Sets up the methods twist next.

What you measure decides what you conclude何を測るかが結論を決める

A clean prediction: deep strategic reasoning (“I think that you think…”) needs theory of mind, so autistic players should reason less deeply. Two studies:

• Pantelis & Kennedy (2017): outcomes are statistically indistinguishable — ASD mean ≈ 30.2, controls ≈ 31.8. Moderate evidence for the null.
• Król & Król (2019): same outcomes — but a process tracer shows autistic players were less strategic in process.

Look only at the outcome → “no difference.” Trace the process → a difference appears. The lesson that decides the LLM debate.

きれいな予測：深い戦略的推論（「私はあなたが考えると考える」）は心の理論を要するから、自閉症のプレイヤーはより浅く推論するはず。二つの研究：

• Pantelis & Kennedy (2017)： 結果は統計的に区別できない — ASD平均≈30.2、対照≈31.8。帰無を中程度に支持。
• Król & Król (2019)： 結果は同じ — だが過程追跡では自閉症のプレイヤーは過程で戦略性が低い。

結果だけ見れば「差なし」。過程を追えば差が出る。LLMの議論を決める教訓。

~3 min — load-bearing for the back half. The prediction (autistic players reason less deeply) is licensed by the classic mindblindness finding; Pantelis & Kennedy (2017, Cognition) found the OUTCOME null (ASD ~30.2 vs controls ~31.8; Bayes factor moderate-for-null), but Król & Król (2019) replicated the outcome AND traced the PROCESS, finding autistic players less strategic in process. Figure: top = outcomes overlap, bottom = process diverges. The portable lesson — say it slowly and tag it: “what you measure decides what you conclude” / “behavioral match is not mechanism.” This is EXACTLY the move we make against LLM-ToM claims after the break — flag the callback now.

Blame bends the inference: the side-effect effect非難が推論を曲げる：副作用効果

A chairman is told a program will raise profits and harm the environment as a side effect. He says: “I don’t care about the environment — I just want profit.” It runs; the environment is harmed. Did he harm it intentionally?

Knobe (2003): 82% say intentionally. Flip one word to help — same indifferent chairman — and only 23% say he helped intentionally.

Same mental state both times. Yet “intentional” tracks bad vs. good, not his actual intent — moral valence bends the mind-reading.

ある会長が、事業は利益を上げ副作用として環境を害すると告げられる。彼は言う：「環境はどうでもいい — 利益が欲しいだけだ」。実行され、環境は害される。彼は意図的に害したか？

Knobe (2003)： 82%が意図的と言う。一語を助けるに変えると — 同じ無関心な会長 — 23%しか意図的と言わない。

心的状態は両方で同じ。なのに「意図的」は彼の意図ではなく善悪を追う — 道徳的な価が心読みを曲げる。

~3 min. Run it as a live show of hands FIRST (whole room, harm version only — with ~6 students a between-subjects split is too thin): “Did he harm the environment intentionally?” Almost everyone says yes. THEN reveal Knobe (2003): 82% harm vs 23% help — same indifferent chairman, one word flipped. The point: intent attribution is morally loaded (the side-effect / Knobe effect) — moral valence of the OUTCOME bends the intent inference, which “pure” inverse planning wouldn’t predict. Mind-reading isn’t a clean Bayesian instrument; affect leaks in. Verified 82/23 (Knobe 2003).

Blame is structured — and needs an agent非難は構造化され — 主体を要する

Much blame is structured (Malle et al. 2014): negative event → causal agent? → intentional? → if so, blame scales with reasons; if not, with could they have prevented it. The inverse-planning machinery, with a moral layer.

Mind perception (Gray, Gray & Wegner 2007) has two axes: agency (the capacity to do) and experience (the capacity to feel).

Blame needs an agent with agency. That’s why we argue over blaming a company, an animal, or an AI — the alignment question, two blocks early.

非難の多くは構造化されている（Malle et al. 2014）：否定的な出来事 → 原因となる主体？ → 意図的？ → そうなら非難は理由に応じ、そうでなければ防げたかに応じる。逆プランニング＋道徳の層。

心の知覚（Gray, Gray & Wegner 2007）は二軸：行為主体性（行う能力）と経験（感じる能力）。

非難には行為主体性を持つ主体が要る。だから、企業・動物・AIを非難できるかを我々は論じる — アラインメントの問い、二ブロック早く。

~3 min. Two ideas, tied to the spine and the alignment tail. (1) Blame-late / Malle path model: blame is often structured reasoning — cause → intent → reasons (intentional) or obligation+capacity (unintentional). Walk the tree; it’s the inverse-planning machinery with a moral overlay (and Knobe shows affect also leaks in — both run). (2) Mind perception (Gray, Gray & Wegner 2007, Science): agency (capacity to act) vs experience (capacity to feel); blame needs an agent with agency. The live payoff: this is WHY blaming a company / animal / AI is contested — the exact agency question alignment raises. Under time pressure, keep only the mind-perception half.

False belief needs a hidden belief誤信念には隠れた信念が要る

Sally’s mistake is the key. In an MDP the agent sees the true state \(s\) — so it can never be wrong. A false belief means the belief \(b\) disagrees with the world: \(b \neq s\).

To represent that, belief must be its own hidden variable, separate from the world. That model is a POMDP — and Sally-Anne is a POMDP.

Next: infer the belief \(b\), not just the goal.

サリーの誤りが鍵だ。MDPではエージェントは真の状態 \(s\) を見る — だから決して間違えない。誤信念とは、信念 \(b\) が世界と食い違うこと：\(b \neq s\)。

それを表すには、信念は世界とは別の独自の隠れ変数でなければならない。そのモデルが POMDP — そしてサリーとアンはPOMDPだ。

次：目標だけでなく信念 \(b\) を推論する。

The hinge into POMDPs — and the rigorous payoff of defining ToM first. Sally’s false belief is impossible in an MDP, because there the agent sees the true state s and can’t be wrong. A false belief requires b ≠ s, which means belief must be its OWN hidden latent, separate from the world — that is exactly a POMDP. Say the line: “Sally-Anne IS a POMDP.” Point at the master diagram with belief lit (and the world state the agent no longer sees directly). This is the bridge the first delivery missed: it jumped to POMDPs as a new topic instead of deriving the need for them from false belief. Then break into the belief-inference block.

POMDPs — inferring beliefsPOMDP — 信念を推論する

Section break into Block 5 — THE HEART of the lecture (~22 min, clock 0:45). This is the longest and most important block; it never gets cut. Transition: “now the agent doesn’t see everything — let’s infer what it BELIEVES.”

Where we areここまでの流れ

• Run the camera backwardsカメラを逆回しする 0:00
• Goal inference as inverse planning逆プランニングとしての目標推論 0:07
• What is theory of mind?心の理論とは？ 0:37
• POMDPs — inferring beliefsPOMDP — 信念を推論する 0:45
• Break休憩 1:07
• Inverse RL — recovering rewards逆強化学習 — 報酬の復元 1:12
• Teaching, flipped教示の反転 1:26
• Alignment & LLM Theory of MindアラインメントとLLMの心の理論 1:38
• Close & Week 10まとめと第10週 1:52

Recap — brief. ToM = IRL is named (~45 min in); now we add partial/false beliefs via POMDPs. One sentence, keep moving — this is the longest block so don’t burn time on the recap.

The food-truck puzzleフードトラックの謎

Chibany likes Mexican > Korean > Lebanese. He can see a Korean truck nearby; a building hides the far spot.

Instead of going straight to the visible Korean, he detours to peek at the hidden spot — the moment he sees it’s Lebanese (not the Mexican he hoped for), he takes the direct path to the Korean.

Passing up a perfectly good Korean only makes sense if he was hoping for Mexican and didn’t know what was hidden. So you infer a desire for a truck not even in the scene — impossible without modeling his belief (Baker et al. 2017).

チバニーの好み：メキシコ > 韓国 > レバノン。近くの韓国料理トラックは見えるが、建物が遠い場所を隠している。

見えている韓国料理へ直行せず、わざわざ隠れた場所を覗きに行き — レバノン料理（期待したメキシコ料理ではない）と分かると — まっすぐ韓国料理へ向かう。

申し分ない韓国料理を素通りするのは、メキシコ料理を期待していて、隠れた場所に何があるか知らなかった場合のみ筋が通る。だからその場にすら無いトラックへの欲求を推論する — 彼の信念をモデル化しない限り不可能だ（Baker et al. 2017）。

Motivation BEFORE formalism — the food-truck scenario is why we need beliefs at all (Baker, Jara-Ettinger, Saxe & Tenenbaum 2017, Nature Human Behaviour). Walk it as a story: Chibany prefers Mexican > Korean > Lebanese; he can SEE a Korean truck nearby but a building HIDES the far spot. He skips going straight to the perfectly good visible Korean and detours to peek behind the building; the moment he can see it’s Lebanese (not Mexican), he takes the efficient straight path to the Korean — he does NOT walk all the way to the far truck (pre-empt the “why didn’t he take the efficient route back?” question — he does). The punchline to land: passing up the Korean only makes sense if he was HOPING for Mexican and DIDN’T KNOW what was hidden — so you infer a desire for a truck that isn’t even in the scene. No shortest-path goal account can explain this; you can only explain it by modeling what he BELIEVED. That’s the entire case for the POMDP. Spend time here — it’s the intuition that carries the formalism.

Forward POMDP — the belief state順方向POMDP — 信念状態

The agent never sees state \(s\), only an observation \(o\). It keeps a belief \(b(s)=P(s\mid\text{history})\), updated by Bayes after each \((a,o)\).

Don’t let the new symbol fool you: \(b(s)\) is nothing exotic — it’s the same Bayesian posterior we’ve worked with all term, just over the hidden state. We write \(b\) because it’s the field’s standard, and because it’s the one probability a POMDP agent must track.

The belief is a sufficient statistic → a POMDP is just an MDP over beliefs.

エージェントは状態 \(s\) を見ず、観測 \(o\) のみを得る。信念 \(b(s)=P(s\mid\text{履歴})\) を保持し、各 \((a,o)\) の後にベイズで更新する。

新しい記号に惑わされないで： \(b(s)\) は特別なものではない — 今学期ずっと扱ってきたのと同じベイズ事後分布で、ただ隠れた状態についてのものだ。\(b\) と書くのは分野の標準であり、POMDPエージェントが追うべき唯一の確率だからにすぎない。

信念は十分統計量 → POMDPは信念上のMDPに他ならない。

The formalism, kept conceptual. The agent never sees state s, only an observation o; it keeps a belief b(s) = P(s | history), updated by Bayes after each (a, o). The key reassurance — say it explicitly to preempt panic at the new symbol: b(s) is NOTHING exotic, it’s the same Bayesian posterior we’ve used all term, just over the hidden state. We write b because it’s the field’s standard and because it’s the ONE probability a POMDP agent must track. Then the big structural payoff: the belief is a sufficient statistic for history, so a POMDP is just an MDP over beliefs — planning in a POMDP = solving an MDP on the belief simplex. That sentence sets up the Tiger problem and the alpha-vector picture coming next.

The Tiger problemトラ問題

Two doors; a tiger behind one. Listen (cost −1) — the growl is right only 85% of the time. Open the wrong door: −100; the correct door: +10. Your belief slides by Bayes:

二つの扉、片方の奥にトラ。聞く（コスト−1）— うなり声が正しいのは85%だけ。間違った扉を開けると −100、正しいと +10。信念はベイズで動く：

The canonical 2-state POMDP (Kaelbling, Littman & Cassandra 1998) — small enough to draw the whole belief space as a line, which is its pedagogical gift. Lay out the numbers: two doors, a tiger behind one; Listen costs −1 but the growl is only 85% reliable; opening the wrong door is −100, the correct one +10 (γ = 0.95 canonical — note the R pomdp package uses 0.75 if anyone has seen it elsewhere). The belief slides by Bayes: each consistent growl multiplies the odds by 0.85/0.15 ≈ 5.67, so from 0.5 you go 0.5 → 0.85 → 0.9698 → 0.9945; a hear-left then hear-right cancels exactly back to 0.5. Walk the figure top (doors) to bottom (belief). Note: this slide is belief-ONLY (no α-vectors yet — those come two slides later, where they’re explained). Tell them: keep listening in the uncertain middle, open once you’re sure enough — and crucially, ONE growl (b=0.85) is NOT enough. That sets up Poll 2.

In GenJAX — belief update = conditioningGenJAXで — 信念更新 = 条件付け

OBS = jnp.array([[0.85, 0.15],     # tiger-left  -> hear-left .85 / hear-right .15
                 [0.15, 0.85]])    # tiger-right -> ...

@gen
def listen(belief):
    s = categorical(jnp.log(belief)) @ "s"      # the hidden state, drawn from belief
    categorical(jnp.log(OBS[s])) @ "o"          # the growl we actually hear

# b'(s) ∝ P(o | s) b(s): enumerate the 2 states, score with assess, normalize
def update(belief, obs):
    return normalize([exp(listen.assess(cm(s, obs), (belief,))[0]) for s in (0, 1)])
# 0.5 -> 0.85 -> 0.9698 -> 0.9945  as the growls agree

GenJAX code slide — the belief update IS Bayesian conditioning, nothing more. Point at OBS (the 0.85/0.15 observation matrix), the @gen listen model (draw the hidden state from the current belief, then the growl from OBS), and update() which enumerates the 2 states, scores each with assess, and normalizes — that’s exactly b’(s) ∝ P(o|s)·b(s). The comment shows the verified output 0.5 → 0.85 → 0.9698 → 0.9945. Backbone is genjax_tiger_pomdp.py in this directory, verified to run — you can execute it live to reproduce the belief sequence. The teaching point: “the belief update is just conditioning — the same thing we’ve done all term, now over a genuinely unobserved world state.”

Track the belief — live信念を追う — ライブ

Listen → the belief slides 0.5 → 0.85 → 0.97; hear-left then hear-right cancels back to 0.5; the α-vectors show when opening beats listening.「聞く」→ 信念が 0.5 → 0.85 → 0.97 と動く。左→右で 0.5 に戻る。αベクトルが「開ける」が「聞く」を上回る点を示す。

WIDGET B — the Tiger belief tracker. Demo script: start at 0.5/0.5; press Listen → hear-left a couple of times and watch the bars animate 0.5 → 0.85 → 0.97. The key beat: after ONE growl the belief is only 0.85 — point out that’s not yet at the open threshold, so one growl isn’t enough; a second agreeing growl pushes it past ~0.90 (where opening beats listening). Show the cancellation too: hear-left then hear-right snaps back to 0.5 — a satisfying “evidence cancels” moment. If the widget overlays the α-vector PWLC value lines with the shaded optimal-action region, point out the belief crossing from the “listen” region into the “open” region. This sets up Poll 2 directly — don’t reveal the poll answer yet. Fallback PNG: tiger-belief-update.png. ~2 min.

Belief tracking, in general一般の信念追跡

The Tiger’s two-number belief is the simplest case; in general a belief is a cloud over states, updated by the same Bayes rule — here, a particle filter.トラの二数の信念は最も単純な場合；一般に信念は状態上の雲で、同じベイズ則で更新される — ここではパーティクルフィルタ。

Callback widget (cut if behind) — generalize the belief beyond two numbers. The Tiger belief is a single number (P(tiger-left)); in a bigger world the belief is a distribution over many states, updated by the SAME Bayes rule (observe → reweight → renormalize). The particle filter shows that as a cloud of particles tracking a moving hidden state. One line: “same machinery, more states.” Fallback: skip it.

Value is a function of belief価値は信念の関数

Each action’s value is a weighted average of its two outcomes → a straight line in \(b\) (its alpha-vector). Open right = +10 if tiger-left (prob \(b\)), −100 if right: that’s \(110b-100\). At \(b=0.5\) it’s a −45 gamble — listen (\(-1\)) wins.

The optimal value is their upper envelope (the max). Its breakpoints are the decision thresholds: open the correct door once \(b\) passes ≈ 0.90 — where open-right’s line finally beats listen (\(-1\)).

Solvers scale this up: exact \(\alpha\)-vector VI → offline QMDP / SARSOP → online POMCP / DESPOT.

各行動の価値は二つの結末の加重平均 → \(b\) の直線（そのαベクトル）。右を開ける = トラが左（確率 \(b\)）なら +10、右なら −100：つまり \(110b-100\)。\(b=0.5\) では −45 の賭け — 聞く（\(-1\)）が勝つ。

最適価値はその上側包絡線（max）。その折れ点が決定の閾値：\(b\) が ≈ 0.90 を超えたら正しい扉を開ける — open-right の直線が聞く（\(-1\)）を上回る点だ。

ソルバが大規模化：厳密な\(\alpha\)ベクトル価値反復 → オフライン QMDP / SARSOP → オンライン POMCP / DESPOT。

Kochenderfer, Wheeler & Wray, Algorithms for Decision Making (MIT Press, 2022) — free at algorithmsbook.com

The α-vector / piecewise-linear-convex value picture. BUILD IT, don’t just show it — this is where students (and you!) get confused. Each line is the EXPECTED reward of committing to one action, which is a weighted average of its two possible outcomes, weighted by the belief: open-right pays +10 if the tiger’s on the left (prob b) and −100 if on the right (prob 1−b), so its value is 110b − 100. Walk the endpoints on the figure: at b=1 (tiger surely left) open-right is safe → +10; at b=0 (surely right) open-right is the tiger → −100; the line just connects them, and the value at any belief is the point on that line. Those two endpoint numbers ARE the action’s alpha-vector. Hit b=0.5 explicitly (students ask): both open lines sit at −45 there — a coin-flip on a tiger — while listen is −1, so you listen. The optimal value is the upper envelope (the max); its breakpoints are the decision thresholds (open-right takes over at b≈0.90). It’s piecewise-linear and convex for a finite horizon. The intuition to deliver: the regions tell you the optimal action — listen in the uncertain middle, open near the edges; the breakpoints are the belief thresholds. This is what makes “the policy is a function of belief” visually obvious. Then the solver taxonomy as a fast name-drop (Kochenderfer’s Algorithms for Decision Making, the Stanford AA228 line): exact α-vector VI → offline QMDP/SARSOP → online POMCP/DESPOT. Under time pressure this whole slide drops to a name-drop — protect the Tiger build-up and Poll 2 instead.

How the decision unfolds — ① uncertain決定の流れ — ① 不確か

You start at 50/50. The top line is listen (\(-1\)) — much better than a −45 gamble. So you listen, and a growl arrives.

50/50から始まる。一番上の線は聞く（\(-1\)）— −45の賭けよりずっと良い。だから聞く。すると、うなり声が届く。

Start the walk. The marker sits on the LISTEN line at b=0.5 — say “right now any door is a −45 coin-flip on a tiger, so the best line overhead is listen.” This is the dynamic payoff of the static plot: as observations come in, the marker is going to SLIDE along the x-axis. Click to the next slide as the growl arrives.

How the decision unfolds — ② still listening決定の流れ — ② まだ聞く

Each growl slides the belief right. After one “left” growl, \(b=0.85\) — still in the listen region (left of 0.90). So you listen again.

うなり声ごとに信念は右へ動く。一度「左」を聞くと \(b=0.85\) — まだ聞く領域（0.90の左）。だからもう一度聞く。

The belief jumped 0.5 → 0.85 (the faded dot + arrow show it). Key beat: the marker is STILL on the listen line — one growl isn’t enough to cross 0.90, so the optimal action is unchanged. This is exactly Poll 2’s setup. Click on as the second growl arrives.

How the decision unfolds — ③ open!決定の流れ — ③ 開ける！

A second agreeing growl → \(b=0.97\), past the 0.90 threshold. The marker hops up onto the open-right line — now opening beats listening. Open the right door.

二度目の一致するうなり声 → \(b=0.97\)、0.90の閾値を越える。マーカーはopen-rightの線へ飛び移る — 今や開ける方が良い。右の扉を開ける。

The payoff frame: the marker hops UP off the listen line onto the green open-right line — that hop IS the decision to act. The whole arc in one sentence: you ride the listen line (gathering cheap evidence) until the belief crosses a threshold where a terminal action’s line rises above it, then you commit. That’s how a POMDP policy works. The live Tiger widget two slides back lets them drive this themselves. Straight into Poll 2 (which is just this, asked before the reveal).

Poll — open, or listen again?ポール — 開ける？もう一度聞く？

You’ve heard one growl from the left. Do you open the right door now, or listen again?

左から一度うなり声を聞いた。今すぐ右の扉を開ける？もう一度聞く？

• A. Open the right door now
• B. Listen again

• A. 今すぐ右の扉を開ける
• B. もう一度聞く

POLL 2 (commit before reveal). Source: AUTHORED (Tiger, verified numbers — not from the SP25 quiz). Setup: you’ve heard ONE growl from the left. Have everyone commit: open the right door now, or listen again? The likely split: many will pick A (open) — the growl feels convincing, 85% sounds high. The misconception this surfaces: confusing the belief (0.85) with the EXPECTED VALUE of acting on it — 0.85 isn’t enough because the −100 downside is huge. Hold the reveal; the arithmetic on the next slide is the whole point.

Poll — revealポール — 解答

B — listen again.B — もう一度聞く。

Opening right wins (+10) if the tiger is left, loses (−100) if right: \(E[\text{open-right}] = 10\,b + (-100)(1-b) = 110b - 100\). One growl → \(b=0.85\) → \(E=\mathbf{-6.5}\), worse than the −1 of listening. Two agreeing growls → \(b=0.97\) → \(E=\mathbf{+6.7}\) → open.

一度のうなり → \(b=0.85\)、よって \(E[\text{右を開ける}] = 110(0.85)-100 = \mathbf{-6.5}\) — 聞く（−1）より悪い。二度一致 → \(b=0.97\)、\(E = \mathbf{+6.7}\) → 開ける。

Reveal: B — listen again. Walk the arithmetic slowly, it’s the payoff: opening right wins +10 if the tiger is left and loses −100 if right, so E[open-right] = 10b + (−100)(1−b) = 110b − 100. One growl → b=0.85 → E = −6.5, which is WORSE than the −1 cost of listening. TWO agreeing growls → b≈0.97 → E = +6.7 → now open. Emphasize: the belief was already “85% sure” after one growl, but the asymmetric −100 penalty means you still shouldn’t act — information has value. Source: authored, verified in genjax_tiger_pomdp.py. Then send them to break.

Break休憩

Break — 5 min, clock 1:07. Cat photo. We resume at 1:12 with the teaching block. Good moment to take a question that’s been brewing about beliefs/POMDPs before we flip the inversion.

Inverse RL — recovering rewards逆強化学習 — 報酬の復元

Section break into the machine side of inverse RL (~14 min, clock 1:12). Transition, post-break: “all morning we inferred discrete goals and beliefs by hand — now the SAME inverse problem at ML scale: recover a whole reward FUNCTION from behavior.” Sets up teaching (smart demonstrators shape behavior to be inverted) and the alignment capstone.

Where we areここまでの流れ

• Run the camera backwardsカメラを逆回しする 0:00
• Goal inference as inverse planning逆プランニングとしての目標推論 0:07
• What is theory of mind?心の理論とは？ 0:37
• POMDPs — inferring beliefsPOMDP — 信念を推論する 0:45
• Break休憩 1:07
• Inverse RL — recovering rewards逆強化学習 — 報酬の復元 1:12
• Teaching, flipped教示の反転 1:26
• Alignment & LLM Theory of MindアラインメントとLLMの心の理論 1:38
• Close & Week 10まとめと第10週 1:52

Recap, post-break (~1:12). Belief inference is done; now scale the inversion up — how machines recover a reward from behavior (MaxEnt → GAIL → AIRL) — then flip it (teaching), then the alignment capstone.

IRL methods = learning the reward逆強化学習の手法 = 報酬を学ぶ

“IRL methods” all do one thing: recover the reward function behind observed behavior. The catch — it’s underconstrained: many rewards explain the same actions. So each method adds an assumption to pin one down.

「逆強化学習の手法」がすることは一つ：観察された行動の背後にある報酬関数を復元する。問題は劣決定：同じ行動に多くの報酬が当てはまる。だから各手法は一つに絞るための仮定を加える。

MaxEnt IRL (Ziebart 2008) — among all rewards consistent with the data, pick the one that keeps maximum uncertainty (highest entropy): \(P(\tau)\propto e^{\text{reward}}\), a soft-optimal demonstrator (the same softmax as Block 2). Why a milestone: a principled fix for the ill-posedness — commit no more than the data forces → one well-defined reward by max-likelihood, and the bridge from IRL to soft-RL and RLHF.

最大エントロピー逆強化学習（Ziebart 2008）— データに整合する報酬の中で、最大の不確実性（最大エントロピー）を残すものを選ぶ：\(P(\tau)\propto e^{\text{報酬}}\)、ソフト最適な実演者（Block 2と同じソフトマックス）。画期点： 劣決定への原理的な解決 — データが強いる以上にコミットしない → 一つの明確な報酬を最尤で得る。IRLからソフトRL・RLHFへの橋渡し。

Frame it first: every “IRL method” learns the REWARD from behavior, and the core difficulty is that IRL is UNDERCONSTRAINED — many rewards explain the same demonstrations, so a method has to add an assumption to choose one. MaxEnt IRL (Ziebart 2008) is the foundational one: it invokes the maximum-entropy principle — among all reward-consistent trajectory distributions, pick the LEAST-committal (highest-entropy) one, P(τ) ∝ e^reward. Say WHY that’s the right move: the data underdetermines the reward, so you should hold onto as much uncertainty as the data allows rather than smuggling in extra assumptions — max entropy is exactly “commit no more than you must.” That gives a single, globally-normalized, well-defined reward fit by maximum likelihood, and crucially it’s the SAME softmax we used as the inverse-planning likelihood in Block 2 — the seam joining cog-sci ToM to machine IRL, and the launch point for deep IRL and RLHF. Next slide: GAIL, AIRL, RLHF.

…from imitation to transferable rewards to alignment…模倣 → 転移可能な報酬 → アラインメント

• GAIL (Ho & Ermon 2016) — adversarial imitation: a critic learns to tell the expert’s behavior from the imitator’s, and the imitator improves until the critic can’t tell them apart. Milestone: imitation that scales to deep, high-dim control — but it skips recovering a reward, so no transferable objective. (≈ a GAN, if you’ve met those.)
• AIRL (Fu et al. 2018) — the same critic-vs-imitator game, but the critic is structured so a reward falls out, disentangled from the dynamics. Milestone: GAIL’s scale plus a transferable reward you can re-optimize under new dynamics.
• RLHF / DPO (’22–’23) — learn the reward from human preferences (pairwise comparisons), not demonstrations; then optimize the policy (DPO folds the reward into the policy). Milestone: this is how LLMs are aligned — preference-based IRL at frontier scale.

• GAIL（Ho & Ermon 2016）— 敵対的模倣：批評器が専門家の行動と模倣者の行動を見分けようとし、模倣者は批評器が見分けられなくなるまで改善する。画期点： 深く高次元の制御へスケールする模倣 — ただし報酬は復元せず、転移可能な目的関数はない。(≈ GAN、知っていれば)
• AIRL（Fu et al. 2018）— 同じ批評器対模倣者のゲームだが、批評器を報酬が取り出せるよう構造化し、ダイナミクスから分離。画期点： GAILのスケールに加え、新しいダイナミクスで再最適化できる転移可能な報酬。
• RLHF / DPO（’22–’23）— デモではなく人間の選好（対比較）から報酬を学び、方策を最適化（DPOは報酬を方策に畳み込む）。画期点： LLMのアラインメントそのもの — 大規模な選好ベースのIRL。

The honesty note to repeat (callback to Block 2’s ill-posedness): a recovered reward is ONE explanation among many — MaxEnt’s entropy prior and AIRL’s disentanglement are regularizers that pick A reward, not THE reward. GAIL (Ho & Ermon 2016): adversarial imitation — a CRITIC tries to tell expert behavior from the imitator’s, and the imitator improves until the critic can’t; it scales to high-dim control but never returns a usable/transferable reward. (Under the hood this is a GAN, but per the course “no untaught concepts” rule we DON’T lean on that term in class — describe the critic-vs-imitator game in plain inference terms; the “≈ a GAN” aside on the slide is optional, for students who already know them.) AIRL (Fu, Luo & Levine 2018): keeps the same critic-vs-imitator setup but structures the critic so a reward function falls out, plus a disentanglement trick (state-only reward + shaping) so the reward transfers across changed dynamics. RLHF/DPO: preference-based IRL — fit a reward model to pairwise human preferences (Bradley-Terry), freeze it, optimize with PPO; DPO collapses the two stages via an implicit reward. This is the practical payoff of the whole IRL program and the bridge to the alignment thread (Weeks 11-13). Then run Widget D (reward recovery).

Recover a reward — live報酬を復元 — ライブ

Paint the true reward on the right grid (+1 / 0 / −1), add demonstrations, watch the middle panel recover it — and light up neutral cells on the way (ill-posed).右のグリッドに真の報酬を描き（+1 / 0 / −1）、デモを加え、中央が復元する様子を見る — 途中の中立セルも光る（不良設定）。

WIDGET D — reward recovery, now fully CONFIGURABLE. Demo script: the true-reward panel (right) starts with a single +1 goal. CLICK any cell on that RIGHT grid to cycle its reward +1 → 0 → −1 — set up a goal plus a −1 “hazard” or two on the way to it. Then add demonstrations: the agent value-iterates on whatever reward you painted, so it SEEKS the +1 and ROUTES AROUND the −1 (watch the paths bend). The recovered panel (middle) shows the MaxEnt-style estimate — reward ∝ where the agent is drawn (green) vs. where it avoids (red), measured against a random walker so the start doesn’t dominate. Two beats to land: (1) it recovers the structure — green at the +1 goal, red at the −1 hazard; (2) IRL is ILL-POSED — a cell merely ON THE WAY to the goal also lights up green, and it can’t cleanly tell a neutral cell from one the agent just avoided. Method note (a student asked): this is MaxEnt-IRL feature-matching (match the demonstrator’s state visitation); the repo’s GenJAX backbones do the principled Bayesian-inversion version — genjax_goal_inference.py inverts the MDP for a posterior over goals, genjax_reward_from_prefs.py recovers a reward from preferences — the widget is vanilla JS so it runs live in the browser. Fallback PNG: reward-recovery-heatmap.png. ~2-3 min.

Teaching = inverse planning, flipped教示 = 逆プランニングの反転

Section break into teaching, new to this lecture (~12 min, clock 1:26). Transition: “we just watched machines recover a reward from demonstrations — now FLIP it: if you KNOW you’ll be inverted, you choose actions that make your goal obvious. That’s teaching.”

Where we areここまでの流れ

• Run the camera backwardsカメラを逆回しする 0:00
• Goal inference as inverse planning逆プランニングとしての目標推論 0:07
• What is theory of mind?心の理論とは？ 0:37
• POMDPs — inferring beliefsPOMDP — 信念を推論する 0:45
• Break休憩 1:07
• Inverse RL — recovering rewards逆強化学習 — 報酬の復元 1:12
• Teaching, flipped教示の反転 1:26
• Alignment & LLM Theory of MindアラインメントとLLMの心の理論 1:38
• Close & Week 10まとめと第10週 1:52

Recap (~1:26). Recovering rewards is done; now the flip — teaching by demonstration (showing-vs-doing, legibility, CIRL) — then the alignment capstone. One sentence, keep moving.

Flip the inversion反転を反転する

If an observer infers your goal from your actions, you can choose actions that make your goal obvious. That’s teaching — and it’s inverse planning run the other way.

The forward planner asks “what should I do to reach my goal?” The teacher asks “what should I do so you can tell what my goal is?” — same machinery, one more level of recursion.

観察者があなたの行動から目標を推論するなら、あなたは目標を明白にする行動を選べる。それが教示であり、逆プランニングを逆向きに走らせたものだ。

順方向のプランナーは「目標に到達するには何をすべきか？」を問う。教師は「私の目標が分かるように何をすべきか？」を問う — 同じ仕組み、もう一段の再帰だ。

The conceptual flip — state it cleanly. If an observer infers your goal from your actions, you can CHOOSE actions that make your goal obvious. That’s teaching, and it’s inverse planning run the other direction. The contrast to nail: the forward planner asks “what should I do to reach my goal?”; the teacher asks “what should I do so you can TELL what my goal is?” — same machinery, one more level of recursion. Keep it to this one idea; the examples (showing-vs-doing, Dragan legibility, CIRL) make it concrete on the next slides.

Showing vs. doing見せる vs する

People doing a task and people showing it move differently — demonstrators deviate from the efficient path to make the goal unambiguous (Ho, Littman, MacGlashan, Cushman & Austerweil 2016).

Both paths here are optimal-length, but the legible one commits on the first move.

タスクをする人と見せる人は動きが違う — 実演者は効率的な経路から外れ、目標を曖昧でなくする（Ho, Littman, MacGlashan, Cushman & Austerweil 2016）。

ここの両経路は最適長だが、読み取りやすい方は最初の一手で意図を示す。

This is the instructor’s own work — Ho, Littman, MacGlashan, Cushman & Austerweil 2016, “Showing versus doing.” The empirical result: people DOING a task and people SHOWING it move differently — demonstrators deliberately deviate from the efficient path to make the goal UNAMBIGUOUS. Point at the figure: both paths here are optimal-LENGTH, but the legible (showing) one commits on the first move so the observer’s posterior jumps immediately. The Bayesian model: the teacher picks the demonstration that maximizes the observer’s posterior on the correct reward. You can mention you’re the author if it helps engagement. Then the recursive-inference model, then run the widget.

Showing as recursive inference再帰的推論としての「見せる」

Why does the legible path work? Ho et al. (2021) make it precise. Two levels:

• the observer inverts your actions into a belief about your goal (inverse planning);
• you, knowing this, pick actions to move that belief toward the truth.

Your target — the observer’s belief — is hidden to you, so the demonstration is itself a POMDP. Teaching = inverse planning, one level up.

なぜ読み取りやすい経路が効くのか？ Ho et al. (2021) が厳密にした。二層：

• 観察者はあなたの行動を反転し、目標についての信念を作る（逆プランニング）；
• あなたはそれを知り、その信念を真実へ動かすように行動を選ぶ。

あなたの狙う相手 — 観察者の信念 — はあなたに隠れているから、デモ自体がPOMDPだ。教示＝一段上の逆プランニング。

The Ho et al. 2021 formalization — the piece the first delivery missed (it had the 2016 behavioral result + CIRL→POMDP, but never named showing as recursive inference with POMDP planning). Ho, Cushman, Littman & Austerweil 2021 (JEP:General, “Communication in Action: Planning and Interpreting Communicative Demonstrations”) model showing as belief-DIRECTED planning: a level-0 observer inverts your actions to a posterior over your goal; you (level 1) plan actions to maximize that posterior on the TRUE goal. Crucially the observer’s belief is a hidden state you can’t see, so choosing the demonstration is ITSELF a POMDP whose hidden state is the observer’s belief — tie back to the master diagram (the lit box is now the OBSERVER’s belief). “Inverse planning one level up.” The legible path’s 0.61-vs-0.50 number (next, in the widget) is this model’s prediction. Instructor’s own work — say so.

Toggle legible vs. efficient — live読み取りやすさを切り替え — ライブ

Toggle efficient (doing) ↔︎ legible (showing): the observer’s P(goal) stays ~0.50 under doing but jumps to 0.61 on the first move under showing — both paths optimal-length.効率的（する）↔︎ 読み取りやすい（見せる）を切替：観察者の P(目標) は「する」では約0.50、「見せる」では最初の一手で0.61に跳ねる — 両経路とも最適長。

WIDGET C — showing-vs-doing legibility toggle. Demo script: with two candidate goals and a shared start, first run the EFFICIENT (doing) path and watch the observer’s P(goal) bar stay near 50/50 until late — the path is ambiguous early. Then toggle to LEGIBLE (showing) and re-run: the observer’s posterior commits EARLIER, jumping toward the true goal on the first move. The verified number to call out: legible 0.613 vs efficient 0.500 at step 1 — and both paths are optimal-length. That number is also the Poll 4 reveal, so plant it. Backbone: genjax_legible_teaching.py (verified). Fallback PNGs: legible-vs-efficient.png / showing-vs-doing-grid.png. ~2 min.

Legibility vs. predictability読み取りやすさ vs 予測可能性

Dragan, Lee & Srinivasa (2013): the efficient reach is ambiguous until the end; the legible reach veers toward the goal early. Two inferences in opposite directions — goal→trajectory vs trajectory→goal.

Reward can be a signal, not just a reinforcer (Ho et al. 2015 — the GardenPath paper).

Dragan, Lee & Srinivasa (2013)：効率的な到達は最後まで曖昧、読み取りやすい到達は早めに目標へ寄る。逆方向の二つの推論 — 目標→軌跡 と 軌跡→目標。

報酬は強化子だけでなく信号にもなり得る（Ho et al. 2015 — GardenPathの論文）。

Dragan, Lee & Srinivasa 2013 — the canonical robot reach-to-one-of-two-objects figure. The distinction to draw: the EFFICIENT/predictable reach hugs the ambiguous midline and resolves the goal only at the end; the LEGIBLE reach veers conspicuously toward the true goal EARLY (it’s longer, but the observer infers the goal sooner). Formalize it as two inferences in OPPOSITE directions: predictability = P(trajectory | goal) (goal→traj), legibility scored by the observer’s P(goal | trajectory-so-far) weighted to early timesteps (traj→goal) — the same axis as ToM. Tie back to the course: reward can be a SIGNAL, not just a reinforcer (Ho et al. 2015 — and that’s the GardenPath paper the RL assignment comes from, so the Week-8 feedback loop was the “communication” reading of reward). Under time pressure Dragan drops to a name-drop.

Alignment as a teaching game教示ゲームとしてのアラインメント

• Cooperative IRL (Hadfield-Menell et al. 2016): human + robot, both rewarded by the human’s reward — but only the human knows it.
• Efficient expert demonstration is provably suboptimal — you should deviate to teach.
• It reduces to a POMDP whose hidden state is the human’s reward.
• The unifying frame: every beat today is a POMDP over which-MDP-we’re-in.

• 協調的逆強化学習（Hadfield-Menell et al. 2016）：人間とロボットが共に人間の報酬で報われるが、報酬を知るのは人間だけ。
• 効率的な専門家の実演は証明可能に最適でない — 教えるにはあえて外れるべき。
• それは人間の報酬を隠れ状態とするPOMDPに帰着する。
• 統一的な枠組み：今日の全ては「どのMDPにいるか」上のPOMDPだ。

Cooperative IRL / assistance games (Hadfield-Menell, Russell, Abbeel & Dragan 2016). The setup: human + robot, both rewarded by the HUMAN’s reward, but only the human knows it. Two results to land. First, the theorem: efficient expert demonstration is provably SUBOPTIMAL — you should DEVIATE from efficient behavior to teach (formally br(br(π_E)) ≠ π_E). Second, the unifying punchline of the whole day: CIRL reduces to a POMDP whose hidden state is the human’s reward — so every beat today (goal inference, belief inference, teaching) is a POMDP over “which MDP are we in?” Say that explicitly; it closes the frame you opened in Block 2. This also seeds the alignment thread for Weeks 11-13. Then run Poll 4.

Poll — show the exitポール — 出口を示す

You want to show a friend which of two exits to take. Do you walk the shortest path, or exaggerate toward it?

友達に二つの出口のどちらを使うか示したい。最短経路を歩く？それとも誇張してそちらへ寄る？

• A. Walk the shortest, most efficient path
• B. Exaggerate toward the exit

• A. 最短で最も効率的な経路を歩く
• B. 出口に向かって誇張する

POLL 4 (commit before reveal). Source: AUTHORED (legibility / Widget C — not from the SP25 quiz). Setup: you want to show a friend which of two exits to take — walk the shortest, most efficient path, or exaggerate toward it? Have everyone commit. The likely split: a real divide — efficiency feels “correct,” but the showing-vs-doing widget they just saw argues for exaggeration. The misconception this surfaces: that the best demonstration is the most efficient ACTION, when the goal is COMMUNICATION, not task completion. Hold the reveal.

Poll — revealポール — 解答

B — exaggerate (be legible).B — 誇張する（読み取りやすく）。

The efficient path is ambiguous early; the legible path makes the observer’s posterior commit sooner — exactly what the widget showed (61% vs 50% on the first move).

効率的な経路は早期には曖昧。読み取りやすい経路は観察者の事後分布を早く確定させる — ウィジェットが示した通り（最初の一手で61% vs 50%）。

Reveal: B — exaggerate (be legible). One-line justification: the efficient path is ambiguous early; the legible path makes the observer’s posterior commit sooner — exactly what the widget showed, 61% vs 50% on the first move. Source: authored (legibility / Widget C). Connect it forward: this “communicate the objective, don’t just complete the task” idea is the seed of the alignment block coming up.

Alignment & LLM Theory of MindアラインメントとLLMの心の理論

Section break into the contemporary capstone (~14 min, clock 1:38). Transition: “the same inverse problem at frontier scale — machine theory of mind, RLHF as preference-based IRL, and the contested question of whether LLMs have a theory of mind.” This block is the modern payoff; seeds Weeks 11-13.

Where we areここまでの流れ

• Run the camera backwardsカメラを逆回しする 0:00
• Goal inference as inverse planning逆プランニングとしての目標推論 0:07
• What is theory of mind?心の理論とは？ 0:37
• POMDPs — inferring beliefsPOMDP — 信念を推論する 0:45
• Break休憩 1:07
• Inverse RL — recovering rewards逆強化学習 — 報酬の復元 1:12
• Teaching, flipped教示の反転 1:26
• Alignment & LLM Theory of MindアラインメントとLLMの心の理論 1:38
• Close & Week 10まとめと第10週 1:52

Recap (~1:38). Teaching is done; the capstone scales the inverse problem to frontier ML — machine ToM (ToMnet), RLHF/DPO as preference-based IRL, and the LLM-ToM debate. Compresses first if behind: keep the RLHF slides + the LLM-ToM bottom line + Poll 3.

Two ways to read a mind心を読む二つの方法

ToMnet (Rabinowitz et al. 2018) — a neural net watches many agents and learns to predict their behavior and false beliefs. The amortized cousin (learned in one forward pass) of Baker’s explicit Bayesian inversion: same inverse problem, learned vs. hand-built.

ToMnet（Rabinowitz et al. 2018）— ニューラルネットが多数のエージェントを観察し、行動や誤信念の予測を学習する。Bakerの明示的なベイズ反転の償却版：同じ逆問題、学習 vs 手作り。

Machine Theory of Mind — ToMnet (Rabinowitz et al. 2018, DeepMind). A meta-learned “observer” network watches many agents and LEARNS to predict their behavior and even false beliefs from behavior alone (it passes a Sally-Anne-style false-belief test in gridworlds). The contrast to draw — this is the slide’s whole point: ToMnet is the AMORTIZED/learned cousin of Baker’s explicit BAYESIAN inversion. Same inverse problem, two ways to solve it — a neural net learns the mapping in one forward pass vs. explicitly inverting a hand-specified planner. Flag that this amortized-vs-Bayesian axis is exactly the mechanism question we’ll hit with LLMs in two slides. Keep it to ~3 min.

RLHF & DPO — a reward from preferencesRLHFとDPO — 選好から報酬を

• RLHF (RL from Human Feedback): don’t hand-write a reward — show people pairs of outputs, record which they prefer, fit a reward model to the preferences, then RL the policy against it.
• DPO (Direct Preference Optimization): skip the separate reward model — optimize the policy directly on the preferences (the reward is implicit in the policy).
• The choice model is Bradley–Terry: the chance you prefer A is a logistic function of the reward gap, \(P(A \succ B) = \sigma(r_A - r_B)\) — \(\sigma\) is the S-curve squashing any real number into \((0,1)\). Fitting \(r\) from human choices is exactly preference-based inverse RL.

• RLHF（人間のフィードバックによる強化学習）：報酬を手書きせず — 出力のペアを人に見せ、どちらを好むかを記録し、選好に報酬モデルを当てはめ、それに対して方策をRLする。
• DPO（直接選好最適化）：別個の報酬モデルを省き、選好から直接方策を最適化（報酬は方策に暗黙的）。
• 選択モデルはブラッドリー–テリー：Aを好む確率は報酬差のロジスティック関数、\(P(A \succ B) = \sigma(r_A - r_B)\) — \(\sigma\) は任意の実数を \((0,1)\) に潰すS字曲線。人間の選択から \(r\) を当てはめるのが、まさに選好ベースの逆強化学習。

Explain BEFORE the code (per the lecture rule). RLHF = Reinforcement Learning from Human Feedback: rather than hand-specifying a reward, collect PAIRWISE human preferences between model outputs, fit a reward model to them, freeze it, and optimize the policy with PPO. DPO = Direct Preference Optimization: the optimal RLHF policy implies an implicit reward, so you optimize the policy directly on preferences without training a separate reward model — the reward lives inside the policy. The choice model under both is Bradley-Terry: P(A≻B) = sigmoid(r_A − r_B). Key framing: fitting r from human choices is preference-based inverse RL — recover the objective from behavior. Next slide runs it in GenJAX. (Callback: reward hacking = Week-8’s positive cycle at frontier scale.)

…recovering that reward in GenJAX…その報酬をGenJAXで復元

@gen
def prefs(pairs):                                    # the Bradley-Terry reward model
    r = jnp.array([normal(0., 2.) @ f"r_{k}" for k in range(K)])   # latent rewards
    for n, (i, j) in enumerate(pairs):
        flip(sigmoid(r[i] - r[j])) @ f"pref_{n}"                   # i ≻ j ?

Sampled data — 90 preferences from a hidden quality (A=+2, B=+0.5, C=−1): A≻B, A≻C, B≻C, A≻B, C≻B, A≻C, … Inference — condition on them, recover the reward: A=+1.36, B=0.00, C=−1.36 → A > B > C ✓ (only up to an additive constant — IRL’s ill-posedness).

サンプルデータ — 隠れた品質（A=+2, B=+0.5, C=−1）から90件の選好： A≻B, A≻C, B≻C, A≻B, C≻B, A≻C, … 推論 — それで条件付けて報酬を復元：A=+1.36, B=0.00, C=−1.36 → A > B > C ✓（定数分だけ不定 — IRLの劣決定性）。

Backbone genjax_reward_from_prefs.py (verified to run). Walk the code: latent reward r per item with a normal prior; each observed pairwise preference is a flip with probability sigmoid(r_i − r_j) — that IS the Bradley-Terry / RLHF reward-model likelihood. Then sampling-importance-resampling recovers the posterior-mean reward. The example: from a TRUE reward A=2, B=0.5, C=−1, sample 90 preferences (mostly A≻B, A≻C, B≻C with occasional flips), condition, and recover A:+1.36 / B:0 / C:−1.36 — correct ranking, identifiable only up to an additive constant (a shift cancels in every sigmoid(r_i − r_j)) — that constant ambiguity IS IRL’s ill-posedness again. This is the reward-modeling step of RLHF, in miniature. Then the LLM-ToM debate.

Do LLMs have a Theory of Mind?LLMは心の理論を持つか？

The contested capstone — present it as a LIVE debate (claim → rebuttal), not a settled answer. Walk the figure’s two poles. Pro-capability: Kosinski 2024 (GPT-4 solved ~75% of false-belief tasks — 6-year-old level — and “may have spontaneously emerged”; note the hedge, don’t strawman it as “GPT has ToM”); Strachan 2024 (at/above human on false belief, irony, hinting across 1,907 humans); Street 2024 (adult-level higher-order recursive ToM). Skeptical: Ullman 2023, Pang 2025. Set it up here; the concrete examples come next, then the skeptical verdict. ~6 min for the whole LLM segment.

What the tests actually look likeテストの実際

Standard Sally-Anne — Anne hides a toy in a box, leaves, the toy is moved. A strong model reports Anne’s false belief correctly. ✓ (Kosinski 2024: GPT-4 ~75%.)

Ullman’s perturbation (2023) — make the box transparent so Anne can see in. A belief-tracker says “Anne now knows,” yet models often still report the old false belief. ✗

Faux pas — needs the nested structure from before: the speaker’s false belief + the listener’s knowledge. Models pass some batteries, but the wins may be response bias, not the inference (Strachan 2024).

The autism lesson, live for machines: behavioral pass ≠ mechanism.

標準のサリーとアン — アンが箱に玩具を隠し、出て行き、玩具が移される。 強いモデルはアンの誤信念を正しく報告する。✓ (Kosinski 2024：GPT-4 約75%。)

Ullmanの摂動（2023） — 箱を透明にしてアンが中を見えるようにする。信念を追う心は「アンは今や知っている」と言うのに、モデルはしばしば依然として古い誤信念を報告。✗

失言 — 先の入れ子構造を要する：話し手の誤信念＋聞き手の知識。モデルは一部の課題を通過するが、勝利は推論ではなく応答バイアスかもしれない（Strachan 2024）。

まさに自閉症の教訓、機械版：行動上の通過は機構ではない。

NEW slide — the first delivery cited Ullman/Kosinski/Strachan but showed NO concrete vignette, so “does it have ToM?” stayed abstract. Make it concrete with the actual tests, leaning on the definitions we built before the break: (1) standard Sally-Anne — GPT-4 passes ~75% (Kosinski 2024); (2) Ullman 2023’s minimal perturbation — make the container TRANSPARENT so the agent can see in; the belief is preserved-as-knowledge, yet models often still report the OLD false belief, flipping success to failure; (3) the faux-pas test needs the nested structure we defined (speaker’s false belief + listener’s knowledge), and Strachan 2024’s apparent passes may be response bias / over-conservatism, not inference. This IS the “behavioral pass ≠ mechanism” point from the autism methods lesson, now live for LLMs — call back to that slide explicitly. Then the honest verdict.

The honest answer: contested正直な答え：論争中

• Pass — many false-belief, irony, and hinting vignettes, at adult level.
• Fail — minimal, belief-preserving perturbations flip success to failure (Ullman 2023): make the container transparent so the agent can see in, yet the model still reports the false belief.
• Confounds — also stumble on faux pas; the wins may be response bias or training-data contamination (Pang 2025).
• A behavioral pass is not the mechanism.

• 通過 — 誤信念・皮肉・ほのめかしの多くを、成人並みに。
• 失敗 — 信念構造を保つ些細な改変で成功が失敗に（Ullman 2023）：容器を透明にして主体が中を見えるようにしても、モデルは依然として誤信念を報告する。
• 交絡 — 失言課題でもつまずく；勝利は応答バイアスや学習データ汚染かもしれない（Pang 2025）。
• 行動上の通過は機構ではない。

Defensible 2026 position: current evidence does not show LLMs have human-like Theory of Mind.2026年の妥当な立場： 現状の証拠はLLMが人間並みの心の理論を持つことを示さない。

The skeptical landing (instructor’s lean is skeptical — present the wins, then dismantle them). The dismantling: Ullman 2023 — trivial, belief-PRESERVING perturbations flip success to failure (make the container transparent so the agent can SEE in, and the model STILL says it holds the false belief). The Strachan faux-pas result — apparent wins are response bias / hyperconservatism, not inference. Pang 2025 — Morgan’s Canon: attributing ToM is premature until memorized-template / training-data contamination is ruled out, which currently it can’t be. The through-line, repeat it: a behavioral pass is NOT the mechanism. The defensible 2026 position: current evidence does NOT show LLMs have human-like Theory of Mind — impressive ToM-shaped pattern-matching, not demonstrated mental-state representation. Then run Poll 3.

Poll — does GPT-4 have ToM?ポール — GPT-4は心の理論を持つ？

GPT-4 passes ~75% of false-belief vignettes — a 6-year-old’s level. Does it have a theory of mind?

GPT-4は誤信念課題の約75%を通過 — 6歳児の水準。それは心の理論を持つのか？

• A. Yes — it passes the tests
• B. No — passing tests isn’t having the mechanism
• C. We can’t tell yet

• A. はい — テストに通る
• B. いいえ — テスト通過は機構を持つことではない
• C. まだ判断できない

POLL 3 (commit before DISCUSS — this one is meant to open a discussion, not just a quick reveal). Source: adapts SP25 IRL quiz Q1 (the ToM functionalism question), modernized to the LLM debate. Setup: GPT-4 passes ~75% of false-belief vignettes (6-year-old level) — does it HAVE a theory of mind? Have everyone commit to A/B/C. The likely split: a genuine three-way divide — this is the point, it’s a live debate. The misconception this surfaces: functionalism — equating “passes the behavioral test” with “has the underlying capacity/mechanism.” Let it run as a short discussion before revealing.

Poll — revealポール — 解答

B / C — behavioral pass ≠ mechanism.B / C — 行動上の通過 ≠ 機構。

This is the recurring epistemics question of the course — “fits the judgments” and “this is the computation it runs” are different claims. The honest read of the current evidence is skeptical.

これは本講義で繰り返す認識論の問い — 「判断に合う」と「これがその計算だ」は別の主張。現状の証拠の正直な読みは懐疑的だ。

Reveal: B / C — behavioral pass ≠ mechanism. The one-line justification: “fits the judgments” and “this is the computation it runs” are DIFFERENT claims — that’s the recurring epistemics question of the whole course (it goes back to the model-fit-vs-mechanism distinction from earlier weeks). The honest read of the current evidence is skeptical. Source: SP25 IRL quiz Q1 (ToM functionalism), modernized. This is the intellectual high point of the back half — let it breathe before closing.

Where we landed到達点

Section break into the close (~5 min, clock 1:52). Transition: “one inversion, applied five times — let’s name what we did.”

One model, every piece inferred一つのモデル、各要素を推論

• Reward \(R\) — goal inference, invert the MDP報酬 \(R\) — 目標推論、MDPを反転
• Theory of mind — defined (false belief), then = inverse planning心の理論 — 定義（誤信念）し、＝逆プランニング
• Belief \(b\) — invert a POMDP (Sally-Anne, Tiger, food-truck)信念 \(b\) — POMDPを反転（サリーとアン、トラ、フードトラック）
• Reward at scale — MaxEnt → adversarial imitation → RLHF大規模な報酬 — MaxEnt → 敵対的模倣 → RLHF
• Teaching — steer the observer’s belief (Ho 2021; CIRL)教示 — 観察者の信念を操る（Ho 2021；CIRL）
• Alignment — RLHF as IRL; LLM theory of mind is contestedアラインメント — IRLとしてのRLHF；LLMの心の理論は論争中

The closing recap — point back at the master diagram (Block 1) one last time: ONE agent model, and today we inferred a different hidden piece each block. Reward R (invert the MDP) → theory of mind (DEFINED via false belief, then = inverse planning) → belief b (invert a POMDP: Sally-Anne, Tiger, food-truck) → reward at scale (MaxEnt → adversarial imitation → RLHF) → teaching (steer the OBSERVER’s belief — Ho 2021’s recursive POMDP; CIRL) → alignment (RLHF as preference-based IRL; LLM-ToM contested, because a behavioral pass is not the mechanism). Drive the frame home: every beat was the same inversion on a POMDP, just a different latent lit up. Keep it tight; this is the takeaway.

Next week来週

We’ve inferred discrete hidden causes — goals, beliefs, rewards. Next: Bayesian nonparametrics — inferring how many causes there are when you don’t know in advance.

⚠️ Next week (Jul 3) is async — no in-person class. Watch the recording I post; Slack discussion runs as normal. (Optional AI Safety workshop that week — in English.)

Reminders: project proposal due Sun Jun 28, 8 PM; 3 reflection chances left (Weeks 10–12); RL + Monte Carlo both due Jul 10.

ここでは離散的な隠れ原因 — 目標・信念・報酬 — を推論した。次回：ベイズ的ノンパラメトリクス — 原因がいくつあるかを事前に知らずに推論する。

⚠️ 来週（7月3日）は非同期 — 対面授業なし。投稿する録画を視聴；Slackのディスカッションは通常通り。（その週は任意のAIセーフティ・ワークショップ — 英語。）

連絡： プロジェクト提案書は 6月28日（日）20:00；リフレクション残り3回（第10〜12週）；強化学習とモンテカルロは共に 7月10日締切。

The bridge slide. Frame the through-line: all term, and especially today, we’ve inferred DISCRETE hidden causes — goals, beliefs, rewards. Next week flips a key assumption: Bayesian nonparametrics — inferring HOW MANY causes there are when you don’t know the number in advance. End with the reminders, since this is the last thing they’ll see: project proposal due Sunday Jun 28, 8 PM; reflection before Friday; RL and Monte Carlo both due Jul 10. Thank them and close.