AI Tutor Shows 0.71-1.30 SD Effect Size in Dartmouth Statistics Course

Last updated: July 6, 2026

Researchers at Dartmouth have published results from a pilot study of an AI tutoring platform called Phosphor, deployed in an introductory statistics course. The headline numbers are striking: students who fully engaged with the platform showed a 0.71 to 1.30 standard deviation improvement in final exam performance compared to baseline expectations.

But the details matter. This was an observational study, not a randomized controlled trial, and the researchers are upfront about the limitations.

What the Study Actually Measured#

Phosphor is a practice quiz platform that uses Claude (Anthropic's model, via Dartmouth's partnership with Anthropic and AWS) to grade constructed-response questions against instructor-defined rubrics. The system provides immediate feedback on free-form answers rather than just multiple-choice questions.

Key findings:

• 90.2% voluntary adoption among enrolled students (the platform was entirely optional)
• Median engagement of 96% of lessons among users who created accounts
• 0.71-1.30 SD improvement associated with full platform engagement, after controlling for midterm performance
• No significant effect from multiple-choice-only quizzes - the constructed-response format with AI grading appeared to be the driver

The 0.71 figure is the conservative lower bound. The researchers note that only about 16 students (11% of the class) reached full engagement levels, so the statistical estimate is derived from a regression model fit across the entire dosage distribution.

What HN is Saying#

The Hacker News discussion generated over 100 comments with significant debate about methodology, implications, and the future of AI in education.

On selection bias: This was the dominant critique. Multiple commenters pointed out that students who voluntarily engage more with study materials tend to perform better regardless of the format. As one put it: "Engaged students score 0.71-1.30 SD better in tests sounds like a much simpler explanation."

The first author responded directly, noting that the dosage-performance relationship persisted across the entire range of usage, not just for full-engagement students. The R-squared values were essentially unchanged whether or not zero-completion students were included.

On the missing control group: Several commenters noted that without a randomized trial, it is impossible to isolate the AI tutoring effect from the effect of simply doing more practice problems. The platform's main contribution might be getting students to engage with material they would otherwise skip.

Interestingly, baseline reading completion for the course was estimated at 10-15% by instructors. Student responses ranged from "literally no one does that" to "is this being recorded?" So the 90% platform adoption rate represents a dramatic change in engagement patterns, whatever the cause.

On Bloom's Two Sigma: Multiple commenters referenced the famous Bloom study claiming that 1-on-1 tutoring provides a 2 standard deviation advantage over traditional classroom instruction. Some see AI tutoring as the potential solution to scaling individual attention. Others pointed to research suggesting the original 2-sigma claim was overstated - more recent replications show effect sizes closer to 0.6-0.7.

On the tutoring vs. grading distinction: One highly upvoted comment noted that Phosphor is "not an AI tutor so much as a practice quiz platform with an AI autograder." The researchers' own data showed that the RAG chat assistant component was barely used - students engaged primarily with the quiz features.

On hallucination concerns: Several educators expressed concern about AI in foundational courses where students cannot evaluate answer quality. One language learner noted: "I use it for conversations in a language I'm learning, but I quickly learned that asking it grammar questions is not a wise decision."

Why This Matters#

The study addresses a real problem in education: the gap between what works (1-on-1 tutoring) and what scales (lecture halls). If AI can provide even a fraction of the benefit of human tutoring, the implications for educational access are significant.

The engagement effect#

Perhaps the most interesting finding is not the AI tutoring itself, but the 90% voluntary adoption rate. Traditional supplementary materials see 10-15% engagement. Something about the platform's design - possibly the immediate feedback loop, possibly the novelty - got students to actually use it.

The researchers noted that engagement persisted across the full ten-week term, and two-thirds of review attempts involved retries spaced a day or more apart. This is not the pattern you would expect from pure novelty effects.

Constructed response vs. multiple choice#

When the researchers switched to multiple-choice-only quizzes mid-semester (responding to student complaints about difficulty), engagement stayed similar but the dosage-performance relationship disappeared. This suggests the AI-graded free-form responses were doing something that multiple-choice questions do not.

The cost question#

One commenter noted: "Too bad the educational use case doesn't make any money. Good LLMs are a game changer for people motivated to learn." The economics of AI tutoring remain challenging - high API costs, uncertain monetization paths, and competition with free alternatives.

Limitations#

The researchers explicitly acknowledge several:

• Selection bias: "Self-selection is the central threat: students who complete more quizzes may be more motivated or higher-performing generally"
• No randomized control: Ethical considerations prevented withholding the tool from some students
• Dartmouth-specific: These are already highly selected students; results may not generalize
• Single course: Introductory statistics has objective answers - unclear how this translates to humanities or subjective disciplines

The authors plan follow-up studies, including potentially attaching completion to course grades (which literature predicts will increase engagement) and crossover designs where different groups receive different treatments at different times.

The barely-used RAG chat assistant is a familiar pattern; see our explainer on what RAG actually is and how to add context to an LLM without retraining it for why retrieval-based chat often underperforms structured, feedback-driven interfaces. Educators weighing whether to trust AI grading against a rubric should also read our take on AI skills for knowledge work, which covers where LLM judgment is and is not reliable yet.

Practical Implications#

For educators considering AI tools:

1. The format matters more than the AI. Constructed-response questions with immediate feedback appear more effective than multiple-choice, regardless of the grading mechanism.

2. Adoption is the first hurdle. A tool that 90% of students actually use may outperform a better tool that 15% use.

3. Expect criticism. Students complained about difficulty when AI-graded questions were introduced. The researchers adjusted mid-semester, which created statistical complications.

For developers building educational tools:

1. Practice and feedback loops beat chat interfaces. The RAG chat assistant in Phosphor was barely used. The quiz features drove engagement.

2. Selection effects are real. Any voluntary educational tool will be adopted more by students who were already going to succeed. Proving causation is hard.

3. Replication will be difficult. As one commenter noted, "this is not science: science must be reproducible and this is just an historical report on artifact that will be unavailable soon."

The study is promising but preliminary. What it demonstrates most clearly is that AI can get students to engage with course material at rates far exceeding traditional methods. Whether that engagement translates to learning gains independent of selection effects remains an open question.

Continue Reading#

• Geohot on LLMs: Love the Tech, Hate the Hype
• How to Stop Claude from Saying 'Load-Bearing'
• TutorMoments: AI2's New Benchmark Shows LLM Tutors Over-Help by Default
• Vulnerability Reports Are Not Special Anymore

Sources#

• Paper: Intelligent Textbooks 2026 Workshop
• Hacker News discussion
• Phosphor platform
• Bloom's 2 Sigma Problem
• Nintil: Bloom's Two Sigma revisited