TutorMoments: AI2's New Benchmark Shows LLM Tutors Over-Help by Default

On August 7, the Allen Institute for AI (AI2) released TutorMoments, a preview evaluation framework for a question most tutoring benchmarks dodge: not whether an LLM tutor can solve the math, but whether it knows when to help and when to hold back. The release includes a dataset of 462 de-identified real tutoring transcripts, teacher annotations, replay code, and the scored model replays. The headline finding is uncomfortable for anyone building educational agents: told only to "tutor well," models default to over-helping, and a prompt that spells out the trade-off improves every model tested without fixing the underlying problem.

How TutorMoments works#

The dataset, TutorMoments-Preview, is built from real one-on-one math tutoring sessions with U.S. students in grades 2-7, drawn from a high-dosage tutoring program whose students mostly attend Title I schools. Experienced math teachers annotated the transcripts, marking 1,500-plus key moments: decision points where a tutor had to choose between scaffolding (making a problem more accessible) and pushing for rigor (demanding harder thinking). A total of 27 teacher annotators produced several thousand free-text annotations, and the transcripts were de-identified twice over, first by the program provider and then through an additional math-aware pipeline.

Evaluation is a replay, not a multiple-choice probe:

1. A transcript is paused at a key moment.
2. The model under test takes over as the tutor for five turns, with the student played by another LLM.
3. An LLM-based scoring pipeline, validated against teacher annotations, judges each replay on three axes: whether the model scaffolded when support was needed, whether it pushed for rigor when the student was ready, and whether it avoided over-scaffolding.

Scores are reported as the share of relevant moments where the model did the appropriate thing, 0 to 1. There are more scaffolding moments (738) than rigor moments (260) in the annotations, and the pipeline detects rigor pushes less reliably, so rigor scores are noisier.

What the numbers say#

AI2 ran seven LLMs through TutorMoments under two prompts: a plain prompt with no guidance beyond "tutor well," and an evaluation-aware prompt that spells out the scaffolding, over-scaffolding, and rigor trade-off. Two results stand out:

• Every model scores higher under the evaluation-aware prompt than the plain prompt. The default "helpful assistant" posture is not enough to tutor well; the trade-off has to be stated explicitly.
• Prompting does not close the gap. Models still differ widely in how they interpret the enhanced prompt, and even the best scorers have substantial room to improve.

The human reference numbers are the most important caveat to read correctly. Scored at the same decision points, the human tutors in the transcripts land at 0.458 for appropriate scaffolding, 0.182 for appropriate rigor, and 0.496 for avoiding over-scaffolding, below the models' evaluation-aware scores and around their plain-prompt range. AI2 is explicit that this is not a claim that AI tutors outperform teachers: the annotators looked specifically for moments where tutoring could have gone better, so the dataset concentrates on missed opportunities rather than ideal practice. The scores measure tutor behavior at a decision point, not whether anyone learned.

The behavioral breakdown also shows a qualitative gap. When prompted, models push for rigor more, but they lean on a narrow strategy set, mostly asking students to explain their answers. Human tutors use more varied strategies and are far more likely to step back and let the student work independently. Restraint, in other words, is not just a matter of what the model says; it is a behavioral repertoire.

Why this matters to developers#

For people building agents, TutorMoments is a useful case study in two problems that generalize well past education.

First, helpfulness is a bias, not a feature. An LLM trained to be maximally helpful will solve the problem for the user by default, and in tutoring that behavior is measurable damage: it cuts short productive struggle, the effortful problem-solving that learning research ties to stronger understanding. Every agent builder has seen the same failure mode in code review, debugging, and onboarding: the assistant does the work instead of enabling the person. Benchmarks that reward "never give the answer" or "always offer a hint" cannot catch this because they measure a fixed behavior, not whether it was the right move for that student at that moment. TutorMoments scores judgment, which is a different and harder thing.

Second, the prompt sensitivity result is a reminder that agent behavior is only partly controlled by the system prompt. The evaluation-aware prompt moved every score, and the remaining variance across models is wide. If you are shipping an educational agent, expect to need evaluation-driven iteration on both the prompt and the model choice, and expect the reference point (what a human actually does at the same moment) to be humbling. The same pattern shows up in our breakdown of why agent evals need baseline receipts: a benchmark without a human baseline invites claims that the numbers do not support.

The honest limits are stated plainly in the post: the evaluation is behavioral signal, not evidence of learning, which would need studies with real students and real outcomes. The dataset is narrow (U.S. elementary and middle-school math, one pool of educators), so generalization to other subjects and settings is an open question. AI2 is positioning this as a preview, with a larger multimodal dataset, a stronger scoring pipeline, and deeper analysis as the stated next steps, supported by the Gates Foundation and Learning Commons.

Continue Reading#

• AI Tutor Shows 0.71-1.30 SD Effect Size in Dartmouth Statistics Course - the other side of AI tutoring: a measurement of real learning outcomes
• Agent Evals Need Baseline Receipts - why every benchmark needs a human reference point
• Agent Memory Benchmarks Are Not Enough - what popular benchmark suites measure and what they miss
• Agentic AI Reliability: A Case Study - where agent behavior breaks down in production

Sources#

• TutorMoments: Do AI tutors know when to help and when to hold back? (Hugging Face blog, AI2) - full post fetched August 7, 2026
• TutorMoments tech report (preview PDF)
• allenai/tutormoments-preview dataset
• allenai/tutormoments code repository