EdTech Can Work. Decision Makers Have to Check the Evidence First.

By Sal Khan, Founder and CEO of Khan Academy

Almost every ed tech vendor says its research shows the product works. Some of that research is strong. Much of it isn’t. Decision makers need to know the difference, because a tool that hasn’t met a real evidence bar shouldn’t receive public money or classroom time.

I was on a panel recently where a skeptic said there’s no evidence that ed tech works. That’s wrong. There is a meaningful body of rigorous, independent evidence that some ed tech, used well, improves learning. The problem is that the evidence is uneven, and the loudest claims are often the weakest.

Start with causality. Many ed tech studies compare students who used a product heavily against students who didn’t. The most engaged students tend to grow the most, so the correlation is real but says little about the tool. A randomized controlled trial addresses that directly. Students or schools are assigned by chance, and the comparison is between the groups as a whole, including students who barely used the product. This answers “what happens if a school adopts this?” rather than “what happens to the students who loved it?”

Even correlational data can be read carefully. Suppose a study finds that students who reached recommended usage gained more than those who didn’t. One scenario is that only the most motivated 20 percent of students in each classroom reached that level, and motivation drove the gains. If so, the tool may not work for most students. But another scenario is that 20 percent of classrooms reached the recommended usage, and in those classrooms nearly every student did. The pattern then points to teacher implementation, not to which students walked in the door. That’s a more useful signal. It still isn’t proof. Teachers who implement one tool well may be the same teachers who do other things well, and those other things could be driving the gains. The classroom-level pattern is a hint worth following, but only a randomized trial can establish causation.

Independence matters as much as randomization. A trial the vendor controls, and publishes only when the results are good, is marketing. Independent researchers, ideally pre-registered, publish whether the results are good or bad. Ask who designed the study, who analyzed the data, and who decided what to publish.

What I found in a survey of the independent evidence.

I searched for independent randomized trials across the major K-12 math platforms and a couple of well-known academic interventions. I may have missed some studies, so please don’t take my word for it. Search for yourself. Here’s what I found:

Khan Academy — 6 independent RCTs Uttar Pradesh, India: 0.44–0.47 SD with dedicated staff support. Arlington, TX, at .12 SD and tutoring in Toronto at 0.06 SD. El Salvador: 0.11–0.25 SD versus traditional teaching, in schools with underqualified teachers. Silicon Valley summer program: 0.7 SD on algebra readiness (though most students still weren’t ready for algebra after). Hamilton County, TN: about 0.06–0.08 SD over two years, 0.14 SD in a subgroup. Brazil (Ferman, Finamor, and Lima, 2017 school year): no significant overall effect; students shared devices without individual practice sessions, and a later look at participant districts found 0.08 SD for students with consistent, individual practice.

ASSISTments — 2 independent RCTs An SRI International trial in Maine found a 0.18 standard deviation gain (p = 0.007), roughly a 60% improvement over the math growth a seventh grader would otherwise be expected to make in a year, with the largest gains among students who started behind. A WestEd replication in 63 North Carolina schools found a smaller but still significant 0.10 SD effect (p = 0.01), measured a full year after the program ended — one of the few ed tech trials to show a persisting effect after the intervention stopped.

Mindspark (India) — 1 independent RCT Published in the American Economic Review. A lottery-assigned after-school program gained 0.37 SD in math and 0.23 SD in Hindi in under five months, extrapolated to roughly 0.6 SD and 0.39 SD at full attendance.

Carnegie Learning — 1 to 2 independent RCTs A Department of Education-funded RCT across 147 schools in 7 states. High school effect size of +0.04 on Evidence for ESSA.

DreamBox — about 2 independent RCTs Older or small studies. Independence of the authors isn’t confirmed.

Zearn — 1 independent RCT A RAND trial in 64 Texas schools. Positive but not statistically significant on the state test.

IXL — 1 independent RCT A Johns Hopkins CRRE trial in Michigan. 10-point gain on Star Math.

i-Ready — none identified Most evaluations are quasi-experimental or affiliated with Curriculum Associates.

Of the eight platforms above, Khan Academy, ASSISTments and Mindspark seem to have quite notable results. That’s real evidence in a field where many people assume there is none. It also shows where to be skeptical. A product with only quasi-experimental or vendor-run studies hasn’t yet shown causal impact, even if the marketing says otherwise.

A quick note on effect sizes.

Numbers like 0.08 or 0.20 standard deviations can sound small without context. In education, they’re not. Researcher Matthew Kraft analyzed nearly 2,000 effects from 747 randomized trials and found a median effect of about 0.10 SD. He suggests that, for real-world education interventions, effects below 0.05 can be considered small, 0.05–0.20 medium, and 0.20+ large.

It also helps to compare those numbers with some of the strongest interventions we have outside of edtech. High-dosage tutoring, one of the most powerful interventions available, often produces effects in the 0.2–0.4 SD range, sometimes higher. Tennessee’s landmark randomized class-size reduction experiment, which gave students substantially more teacher attention throughout the school day, produced meaningful gains generally in the 0.15–0.35 SD range. And large randomized comparisons of different math curricula have found differences on the order of 0.1–0.17 SD.

These are valuable interventions, and I’m not suggesting technology should replace them. Smaller classes can give teachers more capacity to know their students and individualize instruction. Great tutoring can be transformative. And a strong core curriculum is foundational.

But the comparison does provide important context. We should think not just about effect size, but effect size per unit of scarce educational resource.

High-dosage tutoring can require several hours of additional human instruction every week and can cost thousands of dollars per student per year. Meaningfully reducing class sizes requires hiring substantially more teachers and changes the experience of essentially the entire school day. Replacing a core curriculum affects hundreds of hours of instruction and can require substantial spending, professional development, materials, and organizational change.

Edtech isn’t frictionless either. Teachers need training, the technology has to work, the product has to fit the curriculum, and schools have to establish routines that make sure students actually use it. In fact, the research I discuss below suggests that implementation may be one of the most important determinants of whether edtech works.

But consider the scale of the incremental investment. Some of the strongest edtech results listed above come from supplemental interventions involving roughly 30–60 minutes per student per week, with software costs typically measured in tens of dollars per student per year rather than thousands. Yet under strong implementation, we are seeing randomized effects in the 0.1–0.2 SD range and, in some studies, substantially higher.

These aren’t head-to-head comparisons, and we shouldn’t pretend that a 0.2 from one study is directly interchangeable with a 0.2 from another. The populations, assessments, durations, counterfactuals, and implementations are different. But when an intervention that requires a fraction of the additional time, staffing, and money is producing effects in the same broad range as interventions that can cost thousands of dollars per student and consume hours of the school week, that is something education leaders should take very seriously.

Duration matters too. A 0.20 SD effect after four months is different from a 0.20 effect after several years. A short intervention that produces a substantial gain may create an advantage that persists or potentially builds if effective use continues, although we shouldn’t assume effects simply compound linearly.

Ultimately, the question isn’t whether schools should choose technology instead of smaller classes, great teachers, strong curricula, or tutoring. In many cases, these interventions complement one another. The question is: given limited dollars, instructional time, teacher capacity, and organizational bandwidth, what combination of proven interventions produces the most learning for students?

That is why a well-implemented supplemental tool producing even a 0.10 or 0.15 SD causal effect can be a very big deal.

What to look for before you adopt anything:

• Was assignment random, and at what level: students, classrooms, or schools?
• Who designed the study, ran it, and decided what to publish?
• What was the comparison group? “No technology” and “the existing intervention” answer different questions.
• Did the analysis include students who used the product little?
• How much did students actually use the product, and what support did the school provide?
• Has the result been replicated in a different setting?
• Is my classroom/school/district ready to implement with fidelity?

I’d encourage every decision maker to do their own search. Khan Academy comes out on top of this survey, which is why I’m being transparent about how I counted.

Ed tech can help students learn. The right questions are: for which students, in which schools, with what support, compared to what? We only answer them when we demand the trials, publish the results whether they’re good or bad, and read the study designs carefully.

Reading the Khan Academy results carefully.

In case you’re interested, here is some extra nuance on how I interpret the Khan Academy independent RCTs:

• India’s effect is for implementation support. The control schools also had Khan Academy access. The 0.44–0.47 SD measures the value of dedicated staff who made sure students practiced regularly, not Khan Academy versus nothing.
• El Salvador tested Khan Academy against the realistic alternative, not against nothing. All three groups got the same additional instructional time. One got Khan Academy lessons led by a teacher, one got the same software with a non-teaching supervisor, and one got traditional teacher-led lessons with no software. Khan Academy outperformed traditional teaching in both versions, in a setting where many teachers themselves had weak math content knowledge. Attendance in the extra lessons was only about 60 percent, which likely held the measured effect below what full attendance would have produced.
• Tennessee’s 0.06–0.08 is the headline, and it measures Khanmigo usage within the Khan Academy platform, not Khan Academy’s core practice tools alone. The study, by Philip Oreopoulos (University of Toronto) and Nina Low (Charles River Associates), counts every student assigned to the program, whether or not they used it. The comparison was against the interventions the district was already running, which in most schools included other edtech products. More than a quarter of the study sample had a special education designation, and all students were below grade level in a daily math intervention block. That’s a group ed tech often struggles to move, which makes the result more notable, not less.
• Tennessee’s 0.14 is a subgroup result. It covers students who stayed in the program for the full second year. About two in five students left the program early because they had progressed enough to graduate out, and they were still counted in the main analysis even after leaving. I’ve argued elsewhere that the true effect for engaged students may be understated by this subgroup calculation, since the students who left were the ones gaining the most, and we don’t know what they would have gained had they stayed. That’s my own interpretation, not something the data proves, and I want to be upfront about that distinction.
• The Silicon Valley summer program showed a large effect on a narrow, short intervention. 0.7 standard deviations is a big number, but it measures 19 days of intensive instruction, not a typical school year. Most participants were still not ready for algebra I afterward.
• Brazil is a null tied to a specific implementation, not a verdict on the tool. This early randomized evaluation, conducted during the 2017 school year by Ferman, Finamor, and Lima, found no significant overall effect on math performance. The likely driver: students shared devices without dedicated individual practice sessions. Schools with better implementation saw gains; schools with worse implementation saw losses. A later look at participant districts found a measurable effect (0.08 SD) for students with consistent, individual practice. The lesson isn’t “Khan Academy didn’t work in Brazil” — it’s that shared-device, low-individual-practice implementations are the ones to watch for.