ChatGPT’s responses to questions that were put to university students were as good as or better than the human answers in nine out of the 32 subjects tested
By Chris Stokel-Walker
24 August 2023
Students may be able to use ChatGPT to help with their university assignments, but the chatbot can lack critical analysis skills
Angel Garcia/Bloomberg via Getty Images
ChatGPT may be as good as or better than students at assessments in around a quarter of university courses. However, this generally only applies to questions with a clear answer that require memory recall, rather than critical analysis.
Yasir Zaki and his team at New York University Abu Dhabi in the United Arab Emirates contacted colleagues in other departments asking them to provide assessment questions from courses taught at the university, including computer science, psychology, political science and business.
These colleagues also provided real student answers to the questions. The questions were then run through the artificial intelligence chatbot ChatGPT, which supplied its own responses.
Advertisement
Next, both sets of responses were sent to a team of graders. “These graders were not made aware of the sources of these answers, nor were they aware of the purpose of the grading,” says Zaki.
In nine out of the 32 courses surveyed, ChatGPT’s answers were rated as good as or better than those of students. At times, its answers were substantially better. For example, it achieved almost double the average score of students when answering questions from a course called Introduction to Public Policy.
“ChatGPT performed much better on questions that required information recall, but performed poorly on questions which required critical analysis,” says Zaki.