---
title: "AI Tutoring Outperforms Active Learning: Harvard RCT | Carna"
description: "AI tutoring outperforms in-class active learning in a Harvard RCT: bigger learning gains in less time. What the study found, why it worked, and its limits."
canonical: https://carna.ai/en/blog/ai-tutoring-outperforms-active-learning/
lang: en
dateModified: 2026-09-28
---

[Learning guides](https://carna.ai/en/blog/category/learning-guides/) 28 September 2026 10 min read

# AI tutoring outperforms in-class active learning: what the Harvard study found

In a randomised trial at Harvard, students learned more from a carefully designed AI tutor than from an active learning class, in less time. Here is what the study did, why the tutor worked, what it does not prove, and what it means if you are learning English.

[Eray Gündoğmuş](https://carna.ai/en/blog/author/eray-gundogmus/)

In a randomised trial at Harvard, it did. In autumn 2023, 194 students in Harvard's largest physics course learned more from a purpose-built AI tutor at home than from the same lesson taught with in-class active learning: median learning gains more than doubled, in 49 minutes instead of 60. Kestin, Miller and colleagues published it in *Scientific Reports* in June 2025.

The result often gets quoted as "AI beats teachers". It does not say that, and the details are more useful than the headline.

## What the Harvard AI tutor study did

The paper is "AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting" by Greg Kestin, Kelly Miller, Anna Klales, Timothy Milbourne and Gregorio Ponti (*Scientific Reports* 15, 17458).

The setting was Physical Sciences 2 (PS2), an introductory physics course for life-science students with 233 enrolled. The study ran in weeks nine and ten of the Fall 2023 semester and included 194 students: those who consented, took part in both conditions and completed every test.

It was a **crossover** randomised controlled trial, so every student experienced both conditions:

- **Week 1, surface tension.** Group 1 learned with the AI tutor at home; group 2 had the lesson in class.
- **Week 2, fluid flow.** The groups swapped.

Before each lesson, students took a pre-test. After it, they took a post-test and rated their experience.

The comparison class was not a lecture. In PS2 the instructor introduces an activity, students work through it in small groups with help from course staff, and the instructor then gives feedback on the difficulties that came up. 89% of students said PS2 used more active learning than their other STEM courses.

Both groups got the same worksheet and the same activity introductions: live in class, or as pre-recorded videos at home, where students then worked through the problems with the AI tutor, "PS2 Pal". The tests were written by a team member who neither designed the tutor nor taught the lessons.

## What it found

| Measure | AI tutor (at home) | Active learning (in class) |
| --- | --- | --- |
| Median post-test score | 4.5 (n = 142) | 3.5 (n = 174) |
| Median pre-test baseline, both groups | 2.75 (n = 316) | 2.75 (n = 316) |
| Time spent learning | median 49 minutes | 60 minutes |
| "I felt engaged" (1–5 scale, mean) | 4.1 | 3.6 |
| "I felt motivated" (1–5 scale, mean) | 3.4 | 3.1 |
| Enjoyment and growth mindset | no significant difference | no significant difference |

Four results stand out.

**The learning gap was large.** Median learning gains in the AI group were more than double those in the class (p < 10⁻⁸). Controlling for prior knowledge, topic, test version and time on task, a regression gave an effect size of 0.63 standard deviations. The authors call that an underestimate, because many AI-group students hit the top of the test; a quantile regression, less affected by that ceiling, put it at 0.73 to 1.3 standard deviations.

**It took less time.** 70% of the AI group finished in under an hour. Time spent did not predict the post-test score.

**Students liked it at least as much.** They felt more engaged and more motivated; enjoyment and growth mindset were about the same. 83% said the tutor's explanations were as good as, or better than, those of the human instructors.

**It held across levels.** Students with lower and higher prior physics scores both did significantly better with the AI tutor (p < 0.001).

## Why it worked: the design, not the chatbot

This is the most useful part of the paper. The authors did not hand students a general chatbot; they built a tutor around seven practices from learning research:

1. facilitating active learning
2. managing cognitive load
3. promoting a growth mindset
4. scaffolding content
5. ensuring accuracy of information and feedback
6. giving feedback that is targeted and timely
7. allowing self-pacing

The first three went into the tutor's system prompt: engage the student, do not overload them, encourage effort. The authors found that a prompt alone could not keep a multi-part problem in order, because the model sometimes jumped ahead or wandered off topic. So the platform itself walked each student through each part of each problem, one step at a time, the way the instructor does in class.

For accuracy, they did not let GPT-4 work out the answers by itself. Each prompt included a full, step-by-step solution written by the instructors, so the tutor explained from a correct answer instead of guessing.

Then there are the two practices a classroom cannot offer: feedback for each student at the moment they need it, and a pace that fits them. The authors think this explains most of the gain. Their data on pace is neat: every student who found the class "too fast" (3.8%) spent more than the median 49 minutes with the tutor, and every student who found it "too slow" (2.2%) spent less.

The effort sat in preparation: a few days to convert each lesson, several months to build the platform.

## What the study does not show

The authors are careful about their own limits, and you should be too.

- **It was physics, not language.** The lessons taught concepts and problem-solving at the understanding, applying and analysing levels. Nothing here measured speaking, listening or writing.
- **It was two lessons.** Each post-test came straight after its lesson, so the study does not tell you how much stayed weeks later. The authors list retention strategies such as spacing as future work.
- **It was first contact with new material.** The authors say this stage, where a lot of information has to be delivered, seems especially well suited to AI tutors. They "do not presume that structured AI tutoring will always outperform in-class active learning in all contexts", for example tasks that need complex synthesis and higher-order critical thinking.
- **The conditions were favourable.** The authors name them: students who needed different paces, high-quality instructional videos, a model able to follow complex prompts, question-by-question prompts written by expert instructors, and a structured platform. Take one away and the result may change.
- **It is one course at one university.** Randomisation was by peer-instruction group of two or three students, and some authors also taught the course.

The authors are also clear about the role of the teacher: "an AI tutor should not replace in-person teaching". Their proposal is to use the tutor for first exposure before class, so class time can go to problem-solving, projects and group work.

## How it fits with other tutoring research

The Harvard study makes more sense next to the evidence around it. The effect sizes below come from different tests, ages and durations, so compare the direction of the results, not the exact numbers.

**Bloom's "2 sigma problem" (1984).** Benjamin Bloom reported that students taught one-to-one by a good tutor scored about two standard deviations above students in a conventional class of about 30: the average tutored student was above 98% of the class. His "problem" was getting that effect without a tutor for every student. AI tutoring is the latest attempt at an answer.

**The tutoring meta-analysis (2020).** Nickow, Oreopoulos and Quan pooled the experimental evidence on tutoring for children from pre-kindergarten to year 12 and found an overall effect of 0.37 standard deviations. Tutoring by teachers and paraprofessionals worked better than tutoring by non-professionals or parents. Real-world tutoring works, even if it rarely reaches two sigma.

**The World Bank English study in Nigeria (2025).** This one is about English, which makes it the closest to your situation. De Simone and colleagues ran a six-week after-school programme in Benin City in June and July 2024. First-year senior secondary students, typically 15 years old, attended up to twelve 90-minute sessions and worked in pairs with Microsoft Copilot (then running on GPT-4). The prompts positioned the AI as a tutor that supports learning instead of handing out answers, and teachers guided the sessions without teaching directly. Students who were offered the programme scored 0.31 standard deviations higher on the final assessment and 0.23 standard deviations higher in English, the main outcome. The authors estimate that as 1.5 to 2 years of "business-as-usual" schooling. The test was multiple choice on paper, so it did not measure speaking.

**Tutor CoPilot (Stanford, 2024).** Wang and colleagues gave an AI assistant to human tutors in a trial with 900 tutors and 1,800 K–12 students. Students were 4 percentage points more likely to master topics, and 9 points more likely if their tutor was lower-rated. Tutors using it gave away the answer less often. AI made the human tutor better.

**"Generative AI without guardrails can harm learning" (PNAS, 2025).** Bastani and colleagues ran a trial with nearly 1,000 maths students at a high school in Turkey. One group used a plain ChatGPT-style interface ("GPT Base"); another used a tutor prompted to give teacher-designed hints instead of answers ("GPT Tutor"). During practice, grades rose 48% with GPT Base and 127% with GPT Tutor. Then the AI was taken away for an exam. The GPT Base group scored 17% lower than students who never had AI, and they did not realise they had done worse. The GPT Tutor group scored about the same as the control group.

Put together, the lesson is consistent: **a tutor designed to make you think beats an answer machine.** AI can deliver some of a tutor's personal attention at scale, when it gives hints before answers and moves at your pace. Used as a shortcut, the same technology can leave you worse off.

## What this means if you are learning English

None of these studies proves that any particular app will teach you English. They do suggest how to use AI well.

**Use AI for practice and feedback between lessons.** The Harvard tutor's edge was instant feedback at your own pace, the part a weekly lesson cannot give you. If you work with a human tutor, keep them for real conversation and judgement. Our guides to [what a personal English tutor actually does](https://carna.ai/en/blog/what-does-a-personal-english-tutor-do/) and [how an AI tutor compares with a human tutor](https://carna.ai/en/blog/ai-tutor-vs-human-tutor/) go through the split, and [what private English lessons cost](https://carna.ai/en/blog/private-english-lessons-cost/) helps you plan the budget.

**Ask for hints, not answers.** This is the Turkish maths study in one sentence. Tell the AI: "Don't correct my sentence. Tell me which part has a mistake and let me try again." You will learn more from your second attempt than from reading its version.

**Speak every day, even for ten minutes.** None of these trials measured speaking, which is where most learners get stuck. Short daily practice out loud builds the quick recall that reading does not. [Speaking practice with an AI tutor](https://carna.ai/en/ai-english-tutor/) is one way to get it; our guide to [building speaking confidence](https://carna.ai/en/blog/improve-english-speaking-confidence/) has a routine.

**Measure progress on the CEFR, skill by skill.** Research uses standard deviations; you need something you can check. The [CEFR levels](https://carna.ai/en/learn/what-is-cefr/) from A1 to C2 describe what you can do in each skill, and your speaking is often a level behind your reading. Measuring each skill separately is also the best defence against the plateau described in [why learners quit at B1](https://carna.ai/en/blog/why-english-learners-quit-at-b1/).

Carna is built on the same idea. It is [the AI English tutor that knows you](https://carna.ai/en/): it runs a private tutor's method (Learn, Practice, Review) in one app, picks your next question from your mistakes and tells you why, and scores eight skills to place you on the CEFR scale. It is in early access, and it was not part of any of the studies above. Like the tutors in these trials, it is meant to work next to teachers, not instead of them.

## FAQ

### What is the Harvard AI tutor study?

A randomised controlled trial by Greg Kestin, Kelly Miller and colleagues, published in *Scientific Reports* in June 2025. In autumn 2023, 194 students in an introductory Harvard physics course learned one topic with a custom AI tutor at home and one in an active learning class. With the tutor, median learning gains more than doubled, in less time.

### Is AI tutoring better than a teacher?

Not in general. The study tested first exposure to two physics topics. The authors say the tutor "should not replace in-person teaching" and suggest using it before class, so class time goes to harder work. Other studies show AI works best when it supports a teacher.

### Does AI tutoring work for language learning?

The early evidence is positive. In the World Bank's six-week trial in Nigeria, students offered after-school AI tutoring scored 0.23 standard deviations higher in English. The test was on paper, so the evidence for spoken English is still thin.

### Can AI tutoring hurt learning?

Yes, if it hands out answers. In a trial with nearly 1,000 high school students in Turkey, those who practised with a plain ChatGPT-style tool scored 17% lower on a later exam without AI than students who never had it. A version that gave hints instead avoided the drop.

### How long did students spend with the AI tutor?

A median of 49 minutes, against 60 minutes of learning time in class. Time spent did not predict the post-test score.

## Sources

- Springer Nature, *Scientific Reports*: Kestin, G., Miller, K., Klales, A., Milbourne, T. & Ponti, G. (2025). "AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting." *Sci Rep* 15, 17458. [nature.com/articles/s41598-025-97652-6](https://www.nature.com/articles/s41598-025-97652-6)
- American Educational Research Association, *Educational Researcher*: Bloom, B. S. (1984). "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring." 13(6), 4–16. [doi.org/10.3102/0013189X013006004](https://doi.org/10.3102/0013189X013006004)
- National Bureau of Economic Research: Nickow, A., Oreopoulos, P. & Quan, V. (2020). "The Impressive Effects of Tutoring on PreK-12 Learning: A Systematic Review and Meta-Analysis of the Experimental Evidence." Working Paper 27476. [nber.org/papers/w27476](https://www.nber.org/papers/w27476)
- World Bank: De Simone, M., Tiberti, F., Barron Rodriguez, M., Manolio, F., Mosuro, W. & Dikoru, E. J. (2025). "From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria." Policy Research Working Paper 11125. [files.eric.ed.gov/fulltext/ED676624.pdf](https://files.eric.ed.gov/fulltext/ED676624.pdf)
- arXiv: Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S. & Demszky, D. (2024). "Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise." [arxiv.org/abs/2410.03017](https://arxiv.org/abs/2410.03017)
- *Proceedings of the National Academy of Sciences*: Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö. & Mariman, R. (2025). "Generative AI without guardrails can harm learning: Evidence from high school mathematics." 122(26). [doi.org/10.1073/pnas.2422633122](https://doi.org/10.1073/pnas.2422633122)

[Eray Gündoğmuş](https://carna.ai/en/blog/author/eray-gundogmus/)

Software engineer, Carna

Eray builds the Carna platform: the AI tutor, the automated speaking and writing assessment, and the reporting that schools and teachers work from. He writes about CEFR progression, exam preparation and what AI tutoring can and cannot do, from the side that builds it.

- AI-assisted language learning
- Automated speaking & writing assessment
- CEFR framework
- IELTS preparation

- [X / Twitter](https://x.com/eraysw)
- [github.com](https://github.com/eraygundogmus)

## Keep reading

Learning guides

### [What is a personal English tutor, and what do they actually do?](https://carna.ai/en/blog/what-does-a-personal-english-tutor-do/)

A personal English tutor works with one learner and plans every lesson around that learner's level, goal and mistakes. Here is what they do, what a lesson looks like, and where an AI tutor fits.

Eray Gündoğmuş 7 min read 28 Sept 2026

Learning guides

### [How to find a private English tutor online: a 7-step checklist](https://carna.ai/en/blog/how-to-find-an-english-tutor-online/)

Where to look for an online English tutor, how to read a profile, what to test in a trial lesson, and how to tell after a month whether it is working.

Eray Gündoğmuş 6 min read 28 Sept 2026

Learning guides

### [How to improve your English speaking confidence](https://carna.ai/en/blog/improve-english-speaking-confidence/)

Speaking confidence is not the same as speaking ability. Many learners with solid grammar and vocabulary still freeze in real conversations. Here's how to fix the gap.

Eray Gündoğmuş 6 min read 20 Apr 2026

[All articles](https://carna.ai/en/blog/) [AI English tutor](https://carna.ai/en/ai-english-tutor/)

## Learning on your own?

Carna’s personal tutor — all eight skills, A1 to C2 — is still in build. Join the early-access list and we will write to you when it opens.

[Join the early-access list](https://carna.app/individual-waitlist)