Study Claiming AI Helps Students Learn Retracted

In May 2025, Jin Wang & Wenxiang Fan published a study in the Springer Nature journal Humanities & Social Sciences Communications entitled “The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis.”

The meta-analysis made a very bold claim: that students who use ChatGPT (or other AI tools) perform better than students who don’t. In short, that despite the concerns that educators have about AI, that students learn faster and better when they use AI.

The study went on to become incredibly popular. Though it was cited over 500 times by other researchers, the study was especially popular on social media. By the end of its run, it was in the 99th percentile in terms of its overall attention score.

That is largely because the study became very popular in the ed tech space as technology companies pushed for more ways to integrate AI into education.

However, the study has now been retracted. It was originally retracted by the journal, in May of this year. But we now have more insight into why the study was retracted and that likely means for those who made decisions based (at least in part) on the study.

To that end, it honestly doesn’t look good for the state of research in this space.

Background on the Study

The study was conducted by Jin Wang & Wenxiang Fan, both from Hangzhou Normal University in Hangzhou, China. It was published in the journal Humanities & Social Sciences Communications, a Springer Nature journal, in May 2025.

The study is a meta-analysis, meaning that it is examining other studies in a bid to find that there is a trend in the research that had been performed up to that date. To that end, the researchers examined 51 studies that were published between November 2022 and February 2025.

The results of the analysis claimed that there is a “large positive effect” (g = 0.867) on learning performance when students use ChatGPT. There were also “moderate positive effects” on learning perception and higher-order thinking (g = 0.456/0.457).

The study further claimed that impacts required a duration of 4-8 weeks and worked best with problem-based learning and when ChatGPT was used as a “flexibly integrated into teaching as an intelligent tutor, learning partner, and educational tool.”

This outcome was immediately seized upon by the ed tech space, who shared the study widely on social media and used it to promote AI products for the classroom.

However, also almost immediately, the study came under fire. Less than a month after it was published, Ben Williamson published a blog post in which he claimed that the study (along with a similar one) was “methodologically flawed, overhyped and misleading.”

The very next month, Magnus Ingebrigtsen and Marko Lukic, both from The Arctic University of Norway, published an article entitled “Substantial Errors Invalidate The Conclusion From Wang And Fan (2025) About ChatGPT And Learning Outcomes.” (Preprint available here)

The duo highlighted a variety of issues with the study. This included dubious studies that were included in the analysis as well as problems with the analysis itself.

In the end, they requested that the study be retracted and it was, with Springer Nature confirming that it was retracted largely due to their efforts.

So why was the study retracted? Simply put, because its conclusions were not supported by the data.

The Cause of the Retraction

The retraction notice states that the study was retracted due to “discrepancies in the meta-analysis” that “undermine the Editor’s confidence in the validity of the analysis and the conclusions drawn from it.”

Those issues, according to the paper from Ingebrigtsen & Lukic, fall mostly into two categories. Issues with the analysis and issues with the studies that were included in the analysis.

According to the authors, the study was fraught with issues including discrepancies over the number of included studies, failure to weigh the studies appropriately and overreporting the found effects.

One of the most serious claims on that front was the aforementioned g=0.867 effect size for learning performance. According to Ingebrigtsen & Lukic, the study’s own forest plot shows that the effect size should have been closer to 0.5, even with all the other issues.

On the other side of the coin, the authors found that the original meta-analysis included a retracted study, several studies that were mislabeled, studies that were too small or had reported effect that were too large to be seen as reliable.

To make matters worse, many of the studies included in the analysis did not properly control for confounding variables, making them even less reliable.

In short, Ingebrigtsen & Lukic found that the study could not support its conclusions. As such, they recommended that the study be retracted. According to Springer Nature, the original authors never responded to their outreach, resulting in the retraction.

However, that retraction came almost exactly a year after it was published. That, in turn, is the biggest problem of all.

A Year of Misdirection

There’s an obvious problem here that even a non-researcher can see. ChatGPT was released in November of 2022. This study was published in May of 2025. That is less than three years between when the technology was released and when the study was published. That is not enough time for a meta-analysis to be performed.

Of course the studies up to that point were flawed. High quality research, especially when studying impacts on humans, takes time. Of course the studies that were released at that point would have small sample sizes, lack of variable control and other issues. Many were likely intended to be early studies hoping to make a case for a bigger study down the line.

Given how quickly and easily others spotted the issues with the study, it’s clear that the journal should never have published it in the first place. Instead, they did and it went on to become one of the most influential works on the topic of AI in the classroom for a full year.

Why a major Springer Nature journal allowed that is beyond me. They took a clearly problematic study and then gave it one of the most prominent platforms in the world.

It became one of the stop studies in terms of attention score and has likely shaped AI policy in ways that we are still understanding. That didn’t have to be the case. Humanities & Social Sciences Communications needs to ask some difficult questions about how this study got through the editorial and peer review process.

Much like the study linking MMR vaccines to autism and papers by Paul McCrory on concussions and sports, this paper has had impacts that can’t be erased with a retraction.

Bottom Line

The only thing that moves slower than the law is science. It will likely be years, if not decades, before we begin to have an understanding of how AI will impact learning. For better or worse, AI is here now, and, because of that, we are performing this experiment in real time in classrooms around the world.

There is a very real need for good research on the topic. And I am 100% sure that the research is being done and has been done. But there is no way to know what the long-term impacts of AI in the classroom will be because there simply has not been enough time.

In the meantime, bad research can have an outsized impact. Everyone is looking for guidance in this space and it’s only natural that they would turn to a respected journal for answers. Unfortunately, this time, the answer they got were deeply flawed.

The panic and concern over AI cannot be an excuse for publishing bad research. It is better to acknowledge what we don’t know than to rely on faulty information. Unfortunately, that is almost impossible when studies like this get such a prominent platform.

Hopefully, we will all come out of this with a bit more wisdom and caution. Unfortunately, those are two words that have not been used a lot when talking about AI.