Towards a research agenda for generative AI in university
mathematics education
Paola Iannone, Irene Biza, Ben Davies3, George Kinnear1 and Juuso Henrik Nieminen4
1The University of Edinburgh, School of Mathematics, Edinburgh, United Kingdom;
paola.iannone@ed.ac.uk
2University of East Anglia, School of Education and Lifelong Learning, Norwich, United Kingdom
3School of Education, Southampton, United Kingdom
4The University of Hong Kong, Faculty of Education, Hong Kong SAR
This paper reports preliminary results of a study employing the Delphi method to ascertain the consensus of stakeholders regarding the role and use of generative artificial intelligence (genAI) in teaching and learning undergraduate mathematics. Participants to the study included mathematicians, mathematics educators, and learning designers. The data consists of responses to an online survey which asked participants to state the most urgent research questions related to the use of genAI in university mathematics. We obtained 41 research questions by 34 participants from 23 institutions. We grouped these questions under two broad thematic headings: “questions concerning students’ interaction with genAI” and “questions concerning teachers’ interaction with genAI”. These themes indicate what participants deem a pressing research agenda concerning genAI in university mathematics to be. We conclude with a description of the next steps for this study.
Keywords: University mathematics, genAI and large language models, Delphi method.
Introduction
The debate about the potential of generative artificial intelligence (genAI1) and the perceived threat it poses to the teaching and learning of mathematics at university level is pervasive. Opinions are divided (Kramm & McKenna, 2023): many believe that genAI is a threat to the way in which we teach and assess academic performance – and in particular mathematics – and to the delicate relationship between students and teachers (Luo, 2024). Others argue that integration of genAI in teaching and learning at university is unavoidable and indeed desirable (Rane, 2023). In any case, genAI has and will influence teaching and learning processes in higher education at a global scale. Higher education research communities have rapidly responded to these techno-educational shifts. For example, a recent meta-review identified 66 review studies conducted about genAI in higher education alone, exemplifying the urgency of research to help communities of practice to cope with the surge of genAI technologies (Bond et al., 2024).
We see it crucial to produce disciplinary, mathematics-specific research about genAI amid such broader shifts in higher education studies. Research in mathematics education on the affordances and drawbacks of genAI for teaching mathematics has so far spanned from investigating the mathematical capabilities of a genAI (Green, 2024) to the comparison of the perceptions of teachers and students on the role of genAI in university mathematics (Fuller & Barnes, 2024). A rapid review of the higher education literature beyond mathematics (Lo, 2023) reveals how much of the attention devoted to the use of genAI in higher education concerns issues of misconduct in assessment. However, we could find no coherent research agenda in the work we reviewed, especially with focus on university mathematics, and the current research seems to remain disconnected. We, therefore, have attempted to develop a coherent research agenda regarding the role of genAI in university mathematics using the Delphi method (Hsu & Sandford, 2007). In this paper, we report results from the first phase of this study by focusing on the research questions it generated, and on what these questions tell us about the interests and priorities of stakeholders in university mathematics education.
Data collection: the Delphi method
The Delphi method (Hsu & Sandford, 2007) has recently been used productively to seek consensus on information from stakeholders from a certain community in a relatively new field. The first part of the method requires identifying who the stakeholders are in the chosen community and inviting their contributions. The Delphi method then proceeds in four stages of data collection and analysis. In the first stage, an open-ended questionnaire is sent to potential participants, to gather information about the chosen area. If an extensive review of the literature has been carried out, this stage can be replaced by a structured questionnaire. The second and the third stages of the method consist in asking stakeholders to rank the items obtained from the first stage, for example, via a questionnaire or focus group interviews, to gather consensus about the importance of the items found in stage 1. This iterative process usually produces a high level of consensus that, in the fourth stage, is refined to arrive at a final ranking of the items proposed by the participants.
In our implementation of this method, we followed the study of Kinnear et al. (2024) who, by adopting this method, outlined 55 research questions that represent the interest of mathematicians, mathematics educators, teaching designers and technicians regarding the use, design and research on e-assessment of university mathematics. As in this study, we also choose to gather research questions proposed by stakeholders who, in the case of the use of genAI in teaching and learning university mathematics, were mathematicians teaching university mathematics, mathematics educators researching the teaching and learning of mathematics at university, and/or those academics in positions related to teaching administration such as Directors of Teaching. Firstly, we contacted potential participants thought our extended professional networks; we are part both of Mathematics and Education departments, and our extended network comprises both mathematicians and mathematics educators. After the first contact we employed a snowballing recruitment strategy. We then caried out a rapid review of the literature (Wollscheid & Tripney, 2021) on the use of genAI in university mathematics. In this review we found very few mathematics education studies related to this topic; therefore, we started the Delphi process by distributing an open-ended questionnaire to those who had agreed to take part in the study. The questionnaire comprised some biographical data such as their role in the higher education institution, reason for interest in the study, and the three open-ended prompts: “(1) Please enter details about your first research question. (2) What motivates this question? (3) How would one go about researching this question?”
Participants were encouraged to enter more than one research question with justifications if they wished. These prompts resemble those used by Kinnear et al. (2024), where participants were also asked to propose research questions related to the topic of interest. Ethical approval for the study was sought and obtained at the University of Edinburgh.
We distributed the questionnaire between January and March 2024 to colleagues from 72 institutions in 15 countries. We received responses from 34 individuals from 23 institutions in 12 countries who offered 41 research questions (one verbatim example is in Figure 1). Of those, 21 participants described themselves mainly as mathematicians teaching university mathematics, 9 as mathematics educators and 4 either as learning designers or just interested in the topic.
Question
How can LLMs be used to generate bespoke automated feedback to student-produced text in tertiary mathematics?
What motivates this question?
Recent years have seen substantive efforts committed to generating such automated feedback using established platforms like STACK. Some research suggests that "elaborated feedback" is more useful than simply giving correct/incorrect feedback (Attali & van der Kleij, 2017)* but is it worth the up-front effort for the teacher to devise all this feedback?
How would one go about researching this question?
Initially, a qualitative analysis of LLM generated feedback to students’ responses can provide an initial indication of the value and scope of such feedback across different mathematical contexts. Down the line, (quasi)-experimental designs might compare the impact of different feedback types on student learning, behaviour or perceptions.
*(Attali, Y., & van der Kleij, F. (2017). Effects of feedback elaboration and feedback timing during computer-based practice in mathematics problem solving. Computers & Education,	110,	154–169. https://doi.org/10.1016/j.compedu.2017.03.012)
Figure 1: Example of one answer in its original form. Here LLM stands for Large Language Models,
one of the terms used by participants which we then included in the general ‘genAI’ category when we
made the language of the research questions proposed consistent.
After we collected the data, we worked within our team to make the language of the questions uniform so that they would create a coherent body of research questions for later consideration, and to avoid repetition between questions. We note that this is a departure from the method as described in Hsu and Sandford (2007) but given the variety of educational contexts represented by participants, we deemed it necessary to find a unifying language for the research questions. To be faithful to the intentions of the question setters, we consulted the answers to the second and third questions when editing the text. For example, we decided to use genAI as a collective name for all the large language models mentioned in the questions (LLM, ChatGPT, Claude, Gemini etc.). Once we reached a uniform language, we sought to identify themes among the questions. We held three meetings, each comprising a subset of the authors of this paper. In each of the meetings we collaboratively carried out thematic analysis (Attride-Stirling, 2001) to identify themes and sub-themes, iteratively refining the analysis in each meeting. Through this process, we identified three broad umbrella themes: ‘questions regarding students’ interaction with genAI’, ‘questions regarding teachers’ interaction with genAI’. A third theme comprising only three questions on the role of genAI in mathematics education research will not be discussed in this paper. The first and the second umbrella themes each contained a number of subthemes, as shown in tables 1 and 2. Overall, it can be said that the larger umbrella themes were rather pedagogical in nature: they concerned the day-to-day practices of teaching and learning of mathematics rather than addressing broader ethical, philosophical or technological conversations as in the bulk of general higher education studies (see Bond et al., 2023). Below we elaborate on the findings of our analysis of the research questions, and what they tell us about the community’s perceptions of the role of genAI in university mathematics education.
Theme 1: Questions about students’ interaction with genAI
We clustered 17 questions in this umbrella theme (see Table 1). The dominant concerns for participants were how critical students are in respect to genAI’s mathematical output (5 questions) and how students use these tools as a resource for their studies (6 questions).
Table 1: List of sub-themes, with examples, within the ‘students’ interaction with genAI’ group. The number in brackets indicate how many questions were found in each subtheme
Students’ interaction with genAI (17)	
Sub-theme (number of questions)	Examples of questions	
genAI as a resource for studying (6)	How do students use genAI to support their independent study?
What support do students expect with respect to the use of genAI in mathematics?	
Critical evaluation of genAI (5)	How do students evaluate the reliability of mathematical solutions produced by genAI?
Does engaging with genAI support students in developing critical evaluation of tasks and their solutions?	
Feedback (3)	How do students experience interactions with genAI-based automated systems for formative feedback?
How is feedback on a mathematics task generated from a genAI received by students compared to feedback from their teachers?	
Student-centred task design (2)	How can students develop a deeper understanding of mathematical concepts by teaching them to genAI?
In what areas of undergraduate mathematics is genAI more effective for supporting students’ learning? And why?	
Support for use of genAI (1)	What support do students expect with respect to the use of genAI in mathematics?	
Studies on the accuracy of genAI responses to advanced mathematics questions have shown that, for most topics typically included in a first-year mathematics degree and with the exclusion of proof- related questions, genAI can often provide accurate answers, but they are occasionally subject to hallucinations (Green, 2024). Hallucinations are instances when genAI’s output refers to, or directly quotes, material that is incorrect or non-existent (e.g., research papers which seem plausible by their title but have in fact not been written, or a proof following the pattern of a similar proof but in a context where such pattern is not applicable). Therefore, being able to appraise output provided by this tool becomes important for students.
It is interesting to note that participants in our study seem to recognise that genAI is a tool which must be incorporated critically in the teaching/learning interaction and is not something to be banned, as some of the papers in the review by Lo (2023) suggested. This is evidenced by the 11 questions concerning building students’ critical skills and orchestrating support for learning which constitute the largest sub-theme in this group. The need to develop students’ critical skills to appraise genAI output has been addressed in the literature before (see Drushlyak et al., 2024, for an example of a study in mathematics education concerning proof). Criticality towards the output – if not already developed – needs to be fostered with classroom activities, hence the research question regarding the evaluation of the most effective support that can be offered to students and of the activities that can be devised to this end. Related to the support for students’ use of genAI were the two questions directly concerning task design and the three questions about how students respond to automated feedback provided by genAI.
Theme 2: Questions about teachers’ interaction with genAI
The sub-themes concerning teachers’ interaction with genAI are also revealing. We found 21 questions here (see Table 2). The impact of genAI on assessment design and the threat of assessment misconduct were hardly mentioned by participants, contrary to our expectations. Only five questions were dedicated to assessment, and of those, only one (“How can educators detect AI-generated responses from students?”) concerned prevention of assessment misconduct involving genAI. This is contrary to what the general literature on the use of genAI in higher education reports, where the preoccupation with misconduct is pervasive (e.g., Lo, 2023).
Instead, the dominant concern of our participants was with determining how genAI can support the design of tasks aimed at encouraging and supporting students’ learning (10 questions). These questions detail the involvement of genAI in practices the literature recognises to be beneficial to students. For example, the question “What are the benefits of asking students to use genAI to generate their own questions on a given topic?’ refers to the idea that asking students to generate their own mathematical questions is conducive to conceptual understanding (Voica & Singer, 2013) and the need to find out whether this is the same when the questions are directed to genAI. Another interesting aspect is represented by questions about the use of genAI to support teachers’ own practice by rendering some of the tasks related to teaching more effective and less time consuming. For instance, there are questions regarding whether genAI can support lesson planning or writing lecture notes, or even support providing illustrative examples of complex mathematical concepts. Similarly, the assessment sub-theme included questions about how genAI can be used for exam marking or to help
teachers interpret the large datasets that are produced by assessment results. Such questions show again the wish to include genAI in teachers’ practice as a supporting and potentially time-saving tool, but to do so responsibly and aware of the capabilities of the tool.
Table 2: List of sub-themes, with examples, within the ‘teachers’ interaction with genAI’ group. The number in brackets indicate how many questions were found in each subtheme
Teachers’ interaction with genAI (20)	
Sub-theme (number of questions)	Examples of questions	
Student-centred task design (10)	What features of genAI-supported tasks have most potential for advancing learning?
To what extent can genAI be used to generate examples illustrative of particular topics/concepts?	
Assessment (5)	How can genAI be used to support the efficient marking of exam scripts?
How can genAI be used to interpret large assessment datasets?	
Curriculum design / statistics (2)	How can genAI be used for the teaching and learning of statistical data analysis?
What are the implications of recent advances with genAI for priorities in the statistics curriculum?	
Feedback (2)	To what extent is genAI-produced feedback reliable?
How can genAI be used to support digital scaffolding by providing them with appropriate individualised feedback and hints?	
Support for use of genAI (2)	How do we help students learn to make productive use of genAI?	

Statistics teaching has a special mention in our dataset. This may be because some of the participants may be involved in teaching statistics, but also it could be the case that our participants believed that genAI is ‘better’ at solving statistics questions than at providing proofs in pure mathematics (see for example Lo, 2023), at least for now. One surprisingly missing topic regarded the use of genAI for coding. Coding has become a more significant part of university mathematics teaching in the past 10 years (Iannone & Simpson, 2022) where the most used programming languages are Python and R for statistics (Sangwin & O’Toole, 2017). The computer science literature indicates that genAI is very efficient at producing code in these programming languages with a recent study showing that ChatGPT4 achieved an accuracy of 86.2% on the sample used for testing Python coding (Coello et al., 2024). We hypothesise that such questions, which indeed regard university mathematics teaching, are missing from our list of questions due to the composition of our group of participants.
Final remarks and next steps
The categorisation and analysis of the questions presented above shows a picture of engagement with genAI and of preoccupation with how to better shape such engagement, both for students’ learning and for teachers’ practices. Our participants recognised the potential of genAI tools and the need to explore how they can be productively used by their students. The preoccupations that are manifested by some policy makers and university administrators (see Williams, 2024, for an example in the UK sector) regarding assessment misconduct seem not to be reflected in the priorities of those who teach and research university mathematics. This finding may indicate that stakeholders believe that responsible use of a tool – in this case genAI – achieves better outcomes for teaching and learning than banning or restricting its use. One of the limitations of our study is that the questions posed depended on the expertise and interest of the participants. Although we strived to have a diverse group in our sample, both geographically and in terms of expertise, we have inevitably missed some aspect of the interaction between teaching university mathematics and genAI, as the absence of questions concerning coding suggests. For the next steps of our research, and to produce the research agenda we set out to outline, we will analyse focus group discussions with participants and move to the last step of the Delphi method. This consists in asking participants to discuss the research questions obtained in previous stages and rank them in terms of importance to them and to their community. This will allow us to finalise the research agenda both in terms of what questions will finally be included amongst the ones found, and in terms of outlining what the priorities for the community are. The final research agenda will also contain suggestions on how the research questions may be investigated. Such suggestions will be drawn both form the original questionnaire and from suggestions emerging from the focus group interviews.
Acknowledgment
We would like to thank the colleagues who contributed their time to take part in this research.
References
Attride-Stirling, J. (2001). Thematic networks: an analytic tool for qualitative research. Qualitative Research, 1(3), 385– 405. https://doi.org/10.1177/146879410100100307
Bond, M., Khosravi, H., De Laat, M., Bergdahl, N., Negrea, V., Oxley, E., Pham, P., Chong, S. W., & Siemens, G. (2024). A meta systematic review of artificial intelligence in higher education: A call for increased ethics, collaboration, and rigour. International Journal of Educational Technology in Higher Education, 21(1), Article 4. https://doi.org/10.1186/s41239-023-00436-z
Coello, C. E. A., Alimam, M. N., & Kouatly, R. (2024). Effectiveness of ChatGPT in coding: a comparative	analysis of popular large language models. Digital,	4(1),	114–125. https://doi.org/10.3390/digital4010005
Drushlyak, M., Lukashova, T., Sabadosh, Y., Melnikov, I. & Semenikhina, O. (2024). Using ChatGPT for the Development of Critical Thinking in Youth: Example of Inequality Proof. 47th MIPRO ICT and Electronics Convention	(MIPRO)	(pp. 334-339). https://doi.org/10.1109/MIPRO60963.2024.10569759
Fuller, M., & Barnes, N. (2024). The Impact of ChatGPT on teaching and learning in Higher Education: Exploring the dual perspectives of participants who were students and teachers. New Directions for Higher Education, 207, 31–46. https://doi.org/10.1002/he.20507
Green, K. H. (2024). Does ChatGPT Know Calculus? Journal of Humanistic Mathematics, 14(1), 248– 257. https://doi.org/10.5642/jhummath.ILCN1025
Hsu, C., & Sandford, B. A. (2007). The Delphi Technique: Making Sense of Consensus. Practical Assessment, Research, and Evaluation, 12(1). https://doi.org/10.7275/pdz9-th90
Iannone, P. & Simpson, A. (2022). How we assess mathematics degrees: the summative assessment diet a decade on. Teaching Mathematics and its Applications: An International Journal of the IMA, 41(1). https://doi.org/10.1093/teamat/hrab007
Kinnear, G., Jones, I., Sangwin, C., Alarfaj, M., Davies, B., Fearn, S., Foster, C., Heck, A., Henderson, K., Hunt, T., Iannone, P., Kontorovich, I., Larson, N., Lowe, T., Meyer, J. C., O’Shea, A., Rowlett, P., Sikurajapathi, I., & Wong, T. (2024). A collaboratively-derived research agenda for E-assessment in undergraduate mathematics. International Journal of Research in Undergraduate Mathematics Education, 10(1), 201–231. https://doi.org/10.1007/s40753-022- 00189-6
Kramm, N., & McKenna, S. (2023). AI amplifies the tough question: What is higher education really for? Teaching in Higher Education, 28(8), 2173–2178. https://doi.org/10.1080/13562517.2023. 2263839
Lo C. (2023). What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410
Luo, J. (2024). How does GenAI affect trust in teacher-student relationships? Insights from students’ assessment	experiences.	Teaching	in	Higher	Education. https://doi.org/10.1080/13562517.2024.2341005
Rane, N. (2023). Enhancing mathematical capabilities through ChatGPT and similar generative artificial intelligence: Roles and challenges in solving mathematical problems. SSRN. https://doi.org/10.2139/ssrn.4603237
Sangwin, C. J., & O’Toole, C. (2017). Computer programming in the UK undergraduate mathematics curriculum. International Journal of Mathematical Education in Science and Technology, 48(8), 1133–1152. https://doi.org/10.1080/0020739X.2017.1315186
Voica, C., & Singer, F. M. (2013). Problem modification as a tool for detecting cognitive flexibility in school children. ZDM, 45(2), 267–279. https://doi.org/10.1007/s11858-013-0492-8
Williams, T. (2024). Universities’ ChatGPT misconduct focus ‘panicked students’. Times Higher Education. https://www.timeshighereducation.com/news/universities-chatgpt-misconduct-focus- panicked-students
Wollscheid, S. & Tripney, J. (2021). Rapid reviews as an emerging approach to evidence synthesis in education. London Review of Education, 19(1), 1–17. https://doi.org/10.14324/LRE.19.1.32