Yves here. This study confirms anecdata from instructors and professors: that the performance and productivity boost that AI can confer comes at the expense of skill acquisition and retention.
By Naomi Hausman, Oren Rigbi, and Sarit Weisburd. Originally published at VoxEU
Student use of artificial intelligence is reshaping learning and redefining the skills of future workers. This column examines the impact of generative AI tools on academic performance at an Israeli university. Using data from more and less AI-compatible courses, before and after the introduction of ChatGPT, the authors show that AI raises grades – especially for lower-performing students – and compresses the grade distribution, eroding the signal value of grades for employers. Gains in AI-specific human capital are offset by losses in traditional human capital, highlighting AI’s possible trade-offs for productivity.
How should higher education systems respond now that every essay-writing student has a free, always-on co-author? Universities are moving in various directions: some prohibit ChatGPT outright while others try to integrate it into curricula. Evidence suggests that AI is being quickly adopted in the workforce to boost productivity, but that its effects in education are mixed. Large-scale field experiments in firms show sizeable productivity boosts, especially for less-experienced workers (Brynjolfsson et al. 2023, Noy and Zhang 2023). A recent VoxEU column estimates that between 1% and 8% of US work hours are already assisted by generative AI, with plausible annual labour-productivity gains of up to 2% (Bick et al. 2024). A recent experimental study finds that participants randomly assigned to practice writing with an AI tool that exposed learners to high-quality, personalised examples performed better on subsequent writing tasks than those not given access to the tool (Lira et al. 2025). Yet there is a real concern that easy access to generative AI crowds out the deep thinking universities are meant to cultivate (Bastani et al. 2024).
What We Ddo
Our new study (Hausman et al. 2025) tracks 36,000 students across 6,000 courses at a leading Israeli university from 2018 to 2024. We exploit the fact that some courses base most of the final mark on take‑home essays or projects (AI‑compatible), while others rely almost exclusively on in-person, supervised exams or lab work (AI‑incompatible). Because students take both types of courses, we can compare a student’s own performance in more versus less AI-compatible courses, after versus before OpenAI released ChatGPT for widespread use in November 2022.
Student and semester fixed effects absorb individual unobserved ability and time trends, and propensity-score matching analysis ensures a tightly controlled comparison of similar-on-observables courses that nevertheless differed in their compatibility with AI. Data in years leading up to the 2022 AI release confirm that grade trajectories in the two course types were parallel pre‑AI. Our design therefore isolates the causal impact of generative AI availability on student performance. We specifically focus on the impact of this tool on both basic and AI-specific human capital development and on the signal value of grades to future employers.
Key findings
- Grades rise – especially at the bottom. In 2022‑23, average grades in AI‑compatible courses rose by one point on a 0‑100 scale, growing to 1½ points in 2023‑24. The 25th‑percentile student gained two to three points. Failure rates fell by one‑third.
- The distribution is compressed. Both tails shrink: there were fewer fails, but also fewer low‑pass marks, while the share of very high grades ticks up. Compression weakens the signal value of grades.
- AI practice yields AI skill but may crowd out fundamentals. Students first exposed to AI in Year 1 perform better in later AI‑compatible courses than similar students for whom AI was not available in Year 1, suggesting genuine acquisition of AI‑specific human capital. Yet they do no better – and sometimes worse – in advanced exam‑based courses, implying substitution away from traditional learning.
- Within‑student rank predictability falls. Pre‑ChatGPT, a student’s rank in exam courses predicted her rank in take‑home courses; post‑ChatGPT, that slope flattens. Performance with AI reveals less about performance without it.
Why It Matters
Taken together, our results in the classroom echo the broader pattern of rapid, uneven benefits documented in the workplace (Bick et al. 2024). Generative AI lifts struggling students and narrows achievement gaps, but some of the apparent progress comes from outsourcing cognitive effort to the machine. Universities, employers, and policymakers therefore face a twin challenge: teaching students to use AI productively while ensuring they still master core concepts and thought processes unaided.
Policy Recommendations
- Rethink assessment design. Retain a mix of in-person, supervised exams (to measure individual mastery) and take‑home tasks redesigned to reward crafting effective prompts, critically evaluating AI output, and disclosing tool use.
- Make AI literacy part of the curriculum. Blanket bans are ineffective and risk widening inequities. Instead, teach students to verify accuracy, detect hallucinations, and attribute sources – skills already demanded in the labour market.
- Broaden the signaling bundle. As grades lose resolution, employers may rely more on structured interviews, work‑sample tests, or portfolios that demonstrate both foundational knowledge and AI fluency.
- Monitor equity effects. Because weaker students gain the most, well‑calibrated AI policies could narrow educational inequality. But if reliance on AI erodes fundamentals and is not a complete solution for all tasks, the long‑run productivity gains predicted by macro models may be overstated and effects on inequality are unclear (Acemoglu 2024).
- Increase focus on intrinsic motivation. Students may need to experience under-preparedness due to overreliance on AI. Finding ways to increase their motivation and sense of responsibility for their own learning may be the key to success.
Conclusion
Generative AI is already reshaping what – and how – students learn. Our evidence shows that easy access to ChatGPT boosts course grades but blurs the information grades convey about underlying skills. The most productive graduates of the AI era will combine robust domain knowledge and critical thinking with the ability to deploy the machine judiciously.
See original post for references
31 comments
Comments are closed.
“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”
Looks like Frank Herbert was really on to something but what are the odds we get our own Butlerian Jihad?
I kinda hesitate to comment on this because I think I’m getting a reputation as a stuck record. However, it resonates with me, but even more seriously, use of “learning for the test” and other short-cuts long pre-date AI and created a generation of clinicians who, quite frankly, I don’t trust.
You need a basic set of tools in your head, that are as intrinsic as muscle memory, in case we get an EMP or something that makes everyone stop relying on tech for months at a time.
Again, I am reminded of the store assistant (late teens in *that* major Westfield mall, Sydney) who admired my dual timezone watch but admitted the analogue one was no use to her because she couldn’t tell time. WTAF? *sigh* /oldfartmode
This fellow old fart agrees!
I was rather appalled last week when the kid came home and told me there was nothing else to do at school with a week to go before the official end of classes because her teachers had told her they’d already gone through all the “content”.
First, hearing education and learning described as “content” made me want to give the superintendent a piece of my mind, but luckily he just resigned. But also this was a clear indication they were simply teaching to a test and couldn’t be bothered to teach anything further once they finished with the prescribed curriculum.
Seconded by this South African English lecturer. I’m depressingly glad to be retiring next year. If the planet survives, that is.
The planet will survive, and some life forms on it too.
This has been discussed previously here at NC, and I agree with a number of others in the commentariat. I also teach university students. My policy on AI tools is simple: no use whatsoever is acceptable.
Even if it were acceptable, the students must cite it as a source. Failure to do so is plagiarism, which is already deemed unacceptable in higher education. Next, my policy is that AI tools are never an acceptable secondary source, just as Wikipedia is not an acceptable secondary source.
Just as there have always been problems with plagiarism, there will be students who try to cheat by using AI. If I detect this, they get an automatic “F” for the assignment and possibly for the course — again, it’s no different than plagiarism. The general university policy is that incidents of plagiarism should be reported, and repeated infractions may lead to dismissal from the academic program.
As this article suggests, there are people who try to integrate “AI skills” into their pedagogy. To this, my feeling is: what skills? Fiddling around with your prompt for an hour to get the desired results — is that a skill? Learning the syntax of an LLM prompt is far simpler than SQL, HTML, or even JSON. This is only a ‘skill’ in the most dumbed-down curriculum imaginable. Moreover, any ‘skills’ that you learn to massage a prompt will evaporate as soon as the model is changed, which is happening constantly. The people who are calling this a ‘skill’ seem to fundamentally misunderstand the technology.
My assumption is that undergraduate students are in the program to learn how to write and think critically. It won’t happen if they outsource that task to some app. If they don’t care about this, cheat their way through undergraduate study, and only see the degree as a ticket to some corporate job in the future, they will have nothing to offer and might as well be replaced by AI anyway. Graduate students should be learning how to do serious, original research. I see no reason for them to be using AI tools.
If students want to gamble with their future, that’s their choice.
I taught a large class this spring. I felt the University did not give me enough help to fight AI and cheating. I only had two proctors for the exams, so it would have been easy for students to cheat. I also think the university should consider setting faraday cages around so e exam rooms.
Yes, I hear similar from colleagues and empathize with your situation. I have a friend who quit teaching at one school over this very issue. Many universities are not implementing any policy and basically leaving everything up to the faculty. “Guidelines” are offered, but it’s often more about getting us to take responsibility for dealing with the problem. The only way we can push back is by reducing class size, but often that is out of our control. I’m afraid it’s going to take some larger crisis to compel admins to take this matter seriously.
The rising college sophomores experienced shutdown in 8th grade. It’s reasonable to surmise a high portion have simply never done the work expected of a college bound high school student, then matriculated, and wound up in environments where they don’t belong without the skills to even be there. The local colleges are holding classes for freshmen with life skills such as remember to flush.
Sorry, but I’m an absolutist on this question. AI has no place in learning, and should not be allowed. That’s it. There is no “balance to be struck,” and there is no “judicious” use of AI. In the end, AI is getting someone else to do the work for you, and claiming the result as yours. Yes, it’s tedious finding sources, weighing and judging them, assimilating them, coming to conclusions and so forth, but it’s actually a very large part of what you learn during a university course or it should be. Having an LLM make a summary for you of an academic issue with main arguments and references means that you have sub-contracted a very important part of the work to a machine, even if all of the words you use in the final essay are yours. It’s the equivalent of having a Ph D student India do the research for you and writing up the results as your own. In time, and perhaps very quickly, skills of research and analysis will simply disappear, and students will be hostage to a computer programme whose results they just have to accept because they won’t have the skills to check them. And when students brought up on AI go on to become lecturers in ten year’s time, what happens then? Does AI write their lecture notes for them?
There was an article in the Guardian in the last year talking about academics use of AI. Basically you had some using it to write course notes, some using it to write assessments and exams and others using it to mark assessments and tests. And of course some academics using it for all the above.
To me this seems to have resulted in the following:
Course notes created by AI – course notes read by student’s AI – exam/assessment set by AI -exam/assessment answered by AI – exam/assessment marked by AI
And it made me think in a few years the human being will be completely irrelevant to academia. It will basically be AI talking to AI with some humans paying fees and others earning wages. But absolutely no transfer of knowledge from human being to human being.
It seems that way to me too. AI to human to AI…. The efficient way of dealing with that would be to cut out the middleman (the human) and let AI talk to AI. But I suppose the middleman provides valuable services.
For students then it seems that universities are more and more seen as places to register at to get a degree and possibly to get valuable connections.
For people working at universities then it is about providing degrees.
For government officials then universities is about hiding unemployment and also a way of blaming individuals for government failure in dealing with changing circumstances (job-market).
Learning, teaching and research are no longer priorities at universities
Not even any point having the AI talking to the AI as it adds nothing to the dataset.
What you describe about universitiesis pretty much how I felt when I studied engineering in the late nineties. Must be even worse now.
Right on, brutha!
I’m mostly against using AI for anything where the output is relying on the AI to provide conclusions. I’d rather fact check myself than someone else and with AI you need to do that because of hallucinations.
That said, I do see places for AI when kept in a nice little box where errors are likely to be rare and likely to be more noticeable; like writing chunks of code. I work with massive amounts of data analytically and therefore have learned to code SQL and R to support my work. If I was allowed to use AI at work I’d definitely consider letting it write some of my code as a lot of time can be consumed learning new packages for a one-off data need or digging through old work to find the right way to structure code I use once every 3 years.
okay so i am an engineering student. this translates to almost none of my classes being ai-compatible. even when i do have a class that involves a project, ai writes terrible papers. trust me man. i tried using ai on a paper last semester and the only way i could get useful information from it was if i fed it papers and told it to only use the papers i fed it as reference material. even then, it failed to connect across topics. AI also cannot do my homework and a better way of cheating on homework will always and forever be getting the answers from your friends who took the class last year.
a lot of my professors are extremely pro AI. part of this definitely stems from how they use AI in their research – for predicting configurations of molecules or interpreting spectra or whatever. one time a professor told me to use chatgtp to help with a code based assignment. chatgtp was wrong but very helpful in introducing me to functions i wouldn’t have utilized myself.
im not rlly one way or the other on AI. like it does have its uses. it can be a helpful editing tool. when writing code it’s like if stack overflow and office hours had a baby. when writing emails where i need to sound fake it is also useful. and i know the main reason why i get to be more pro AI is because i am in upper level AI-incompatible courses where bespoke AI for specific tasks exists. and im sure things are different in a 100 level english course.
To be honest, AI IS extremely good at producing certain types of code: they scrape stackexchange fairly extensively, I gather, and is a lot better than I am at coming up with solutions to reasonably common problems (which is pretty much most of my issues). But this is not exactly “learning,” but drudge work, really what AI is supposed to be good at….
How do you know it was really AI that produced the code? ;)
Builder.ai faked AI with 700 engineers, now faces bankruptcy and probe
https://www.business-standard.com/companies/news/builderai-faked-ai-700-indian-engineers-files-bankruptcy-microsoft-125060401006_1.html
Agreed. If I were turn the clock back 25 years to my learning Fortran, I can see how AI could act as a glorified code dictionary for certain subroutines. That would’ve helped. However I’d still have known that I need to know what assumptions (like what type of input number and output number are acceptable) are built in, so “gaining the wider knowledge of what I must learn myself and add to the wider program” would still have been applicable.
The only good thing is that with such human checks added to the AI “starting code”, Fortran would have thrown up an error message when the output was computable but strange, compared to a lot of more black box software that won’t alert you and you’ll never know you have a nonsense answer unless you have EXPERIENCE and know the various “sniff tests”.
I won’t bore people yet again but it’s why I despise odds ratios based on human observations. I used to have a very memorable exercise in teaching where I showed output from two logit regressions in same area (could be clinical trials or non-health surveys, doesn’t matter): the odds ratios were massive in one dataset and all very clustered around 1 in other. The “trick question” I asked attendees is “do you think these reflect different underlying preferences?” (The answer is “no” since the pattern of the ratios is identical, one just came from a source with more certainty and/or fewer “external annoying factors cluttering things up”.) There’s a certain quite memorable YouTuber discussed at length here on NC recently who the NC authors said was (to paraphrase and if I understood correctly) straying beyond his lane, and I’ve tried to make this point to his latest videos……no response….the academic silo effect and I’m used to it *shrug*
I think your comment is very illustrative of why everyone needs to remember AI is NOT Intelligent!
Engineering is basically applied physics and applied chemistry. It has its own language like any specialized field. Because of this it is of no surprise that a generic AI produces invaluable papers because it DOES NOT EVEN KNOW it doesn’t speak the right language. Now an AI trained only on engineering work and literature would probably produce more useable outputs, but that would just mean it was better at faking its outputs, not that it was actually intelligently making those outputs.
As for your professors who are pro-AI for its predictive abilities… As I have pointed out before in my comments there are TWO components of AI applications that people don’t think of independently… There is the LLM part that is pure language generation via statistical language modeling and then there is the MASSIVE computing power. Your professors who like it, “for predicting configurations of molecules or interpreting spectra…” like it for its computational horsepower. In this situation it is just exponentially processing the data and providing results. There is no new THEORETICAL material science here being generated, it’s just doing the data processing way faster than a single professor or research assistant can accomplish absent access time to a super computer; HOWEVER it’s being sold as if it was just a super intelligent entity – which it is not.
Sooner or later, AI will be able to write “good” electrical engineering dissertations.
The students who rely on AI to write their dissertations for them will not be electrical engineers.
Going forward, you may end up with two types of Universities – those that allow and maybe even encourage the use of AI and those that shut it down entirely. Big businesses are already discriminating against graduates from elite schools as being sub par in performance and attitude so I would also expect big business to only hire from those universities that disallowed AI as they would be wondering just what sort of education a student in an AI-friendly university actually got.
I disagree here. I would expect big business to hire from those colleges that encourage widespread AI use. Because the purpose of education in the US isn’t to learn, it’s to produce good little workers who won’t question much. We had a superintendent who explicitly said a few years ago that the school system’s job was to train children for the jobs of the future.
Big business does not want critical thinkers – it wants drones who will press the correct button and not ask questions.
But if that is so, why would they need humans at all? AI is increasingly getting better at routine, “drudge” work. They make for much better “good little workers” than any human could be, even without much or any supervision.
They need humans to “train” the “AI” that doesn’t actually work, but that credulous upper management decided to pay big bucks for anyway. Ask me how I know.
A well-known Chemistry prof / all subjects podcast guest from upstate NY said a couple of years ago that with AI there is little point to having English composition classes anymore. It’s all going to come back as AI speak.
I remember when using Cliffs Notes was thought to be a cheat / short-changing yourself.
I have a different warning: beware of completely fake students. CA has an estimated half million attempts at fake enrollment a year. About 1/3 of community college applicants. Stealing financial aid is the primary reason, but I expect with AI attempts to get actual degrees for people who don’t exist will increase.
Overview of the current mess, https://calmatters.org/education/higher-education/2024/04/financial-aid-fraud/
So glad that I retired from academia a couple of years ago! Got out just in time!
AI is absolutely perfect for the universities.
Generally speaking, these institutions were never interested in ideas. The priorities were and are certification, status, and prestige.
Now, no one has to teach or learn and everyone can go about the much more serious business of achieving professional acclaim and notoriety as an “expert” in their respective fields.
Not “everyone”, Gulag. Just those with the right connections. And AIs can also be “experts”. I think it was Thomas Frank who remarked that it was intriguing that journalists were all enthusiastic for AI when it was obvious that they were among the first who would be replaced unless they were wholly subservient.
Journalists that are not enthusiastic for AI, are the ones to be replaced by AI first.
https://en.wikipedia.org/wiki/Kapo