Your Edtech Is Getting Used. But Is It Actually Working?

The useful part

That distinction became especially clear to me while conducting a meta-analysis of Penda Learning, a computer-based science intervention. Across 13 independent school-based samples involving more than 33,000 students, we found significant gains in science achievement. It wasn't simply the amount of time students spent using the intervention that best predicted achievement.

How it works

• Schools with low, moderate, and high levels of mastered activities showed effect sizes of 0.56, 0.89, and 1.40, respectively.
• Mastery remained the strongest predictor of science outcomes even after engagement hours were considered.
• Define Successful Implementation Before Launch Too often, districts decide how they will measure implementation after a program is already underway.
• Digital programs make usage data remarkably easy to collect, but leaders should resist equating a dashboard filled with logins, minutes, and completed assignments with successful implementation.
• The relationship between mastery and outcomes followed a dose-response pattern: as students mastered more activities, achievement effects grew substantially.

What to take from it

If we want evidence-based programs to produce evidence-based results, we need to pay much closer attention to what happens after the purchase. Students were considered to have mastered an activity when they demonstrated at least 80% accuracy. As the number of activities mastered increased, so did the effects associated with the intervention.

Example or evidence

• District leaders should absolutely ask whether an instructional intervention has credible research behind it.
• Miller, Ph.D., is an education researcher studying academic achievement, economic mobility, and workforce development.
• He has served as a research consultant to universities and K-12 education companies, including Penda Learning, which funded the study described.
• Adoption and participation matter, but these are starting measures, not end goals.

Details worth keeping

How much time are they spending on the platform? Those measures can tell leaders whether a tool is being used. They cannot necessarily tell them whether students are learning.

Related coverage

• Insidehighered: Higher Ed's AI Integration Overhyped, New Data Shows kathryn.palmer… Wed, 09/30/2026 - 03:00 AM For the first time, the company that owns Canvas analyzed which external learning tools higher education users...
• Pbs: Putting technology into schools was sold as a revolution that would help kids learn.
• Techlearning: The question shouldn't be how much screen time should students have, but what they are doing, what are they learning, and is technology helping?