ChatGPT in physics education: A pilot study on easy-to-implement activities

ABSTRACT

While the use of artificial intelligence (AI) and GPT models is already widespread in various industrial applications (Ahuja, 2019;Veloso et al., 2021) research investigating the implementation of chatbots into classroom practice is still in its infancy (Hwang & Chang, 2021), especially with respect to secondary schools (Adiguzel et al., 2023;Salas-Pilco & Yang, 2022;Salas-Pilco et al., 2022;Halaweh, 2023).There are specific barriers and risks associated with the use of large language models in an educational context (e.g., Farrokhnia et al., 2023;Kasneci et al., 2023).For example, Floridi and Chiriatti (2020) warn that "fake news and disinformation may [...] get a boost" (p.692) by tools like ChatGPT considering that it becomes easier to convincingly mislead with automatically generated texts (McGuffie & Newhouse, 2020).Floridi and Chiriatti (2020) conclude that "humanity will need to be even more intelligent and critical" (p.692).In the same direction, Kasneci et al. (2023) suggest integrating large language models into classroom practice "in a way that complements and enhances the learning experience" (p.7), e.g., helping students' developing critical thinking skills (Gregorcic & Pendrill, 2023).In any case, the "physics education community is well-positioned to investigate the use and capabilities of ChatGPT and other AI systems" (Wang, 2023).
One particularly relevant limitation of AI tools is the lack of higher-order thinking skills as outlined by Farrokhnia et al. (2023).AI tools are highly dependent on the data they are trained on, and often lack a deep understanding of the textual outputs produced (Bogost, 2022;Gao et al., 2023) or the content context, which is essential for higher-order thinking (Dimitrov, 2023).Additionally, ChatGPT's inability to assess the reliability of its training data (Lecler et al., 2023) may hinder its effectiveness in evaluating the accuracy of generated

Research Article
Bitzenbauer information (Sallam, 2023).For learners, it hence, seems crucial to become aware of the responsibility to critically evaluate the quality of ChatGPT outputs with regard to content and coherence.In the literature scholars even warn that for learners the use of "ChatGPT can result in simplification of the process of obtaining answers or information, which can have negative impact on students' motivation to perform independent research" (Farrokhnia et al., 2023, p. 9).However, despite and specifically because of the above raised issues, using ChatGPT in classrooms can be a valuable opportunity to promote critical thinking skills among students.Hence, we argue that educational research must bring forward suggestions for classroom practice such that students can recognize the limitations of AI tools and appreciate the importance of higher-order thinking skills that cannot be replicated by machines.
In this article, we present two easy-to-implement examples demonstrating how ChatGPT can be used in classrooms to foster students' critical thinking skills-although we contextualize these in the context of physics teaching, we believe that these examples may guide classroom practice in any subject.In the next section, we give an overview of research on critical thinking.Finally, we provide insights into the results of an initial implementation of these examples in the field, analyzing secondary students' opinions about AI in general and ChatGPT in particular, prior to and post instruction.

FOSTERING CRITICAL THINKING SKILLS USING ChatGPT IN THE PHYSICS CLASSROOM
Various definitions for critical thinking prevalent in educational contexts (Ennis, 1996;Lipman, 1988) align in that (1) "critical thinking involves drawing conclusions supported by evidence" and (2) "critical thinking involves making decisions and/or forming beliefs about a situation" (Smith & Holmes, 2020, p. 2).
Hence, fostering critical thinking skills requires deliberate practice, i.e., "doing special exercises whose main point is to improve critical thinking skills themselves" (van Gelder, 2005, p. 43).The cultivation of critical thinking abilities in students is imperative for facilitating the development of effective problem-solving skills, sound judgment, and accountable academic behavior, as noted by Hidayat et al. (2023).
At the same time, it is crucial to make secondary school students aware of the limitations of ChatGPT and other AI language models (Kasneci, 2023) because they have the potential to disseminate false information.This seems crucial far beyond the physics classroom, namely with regards to the challenges of the information age, where misinformation is prevalent and can have significant consequences.
Taken together, we argue that incorporating ChatGPT in (physics) physics classrooms allows teachers to • foster students' critical thinking skills by producing and reflecting on different ChatGPT outputs, and to • initiate reflection processes among their students about advantages and pitfalls associated with the use of ChatGPT as well as consequences for obtaining valid serious information in general.

TWO EASY-TO-IMPLEMENT EXAMPLES USING ChatGPT IN QUANTUM PHYSICS CLASSROOMS
In this section, we present two concrete examples of implementing ChatGPT1 in physics lessons on the topic of quantum physics.We have decided to stick with quantum physics-which is today part of most secondary school curricula across Europe as shown by Stadermann et al. (2019)-for these examples because • of quantum physics being highly abstract and • quantum physics is a prime example for the necessity of model descriptions (Bitzenbauer, 2021).
Thus, using ChatGPT to reflect on various texts related to quantum concepts seems valuable for engaging students in a process of critical thinking.The following examples of ChatGPT implementations in secondary education are by no means limited to quantum context, which we have used as an example here.

Production and Critical Analysis of ChatGPT Outputs
Teachers can use ChatGPT as a tool to support students' critical thinking skills by guiding them through different steps: from text production, through critical analysis, to revision, e.g., according to the following steps inspired by the think-pair-share method (Alsmadi et al., 2023;Lyman, 1981;Prahl, 2017) and exemplified for discussing wave-particle duality and photons: • First, the students can be asked to generate a text about photons using ChatGPT (part a in Figure 1).
• After the texts have been generated, the students can be asked to analyze and evaluate the accuracy and clarity of the information.For example, they can be encouraged to search for any inconsistencies or inaccuracies and compare the information provided in the text with what they had learned in class before ("think").
• The students should then exchange their texts created with ChatGPT in small groups ("pair").They will become aware that information provided by ChatGPT may differ depending on the students' prompts (part b in Figure 1).
• In a next step, the students could be encouraged to revise the text produced by ChatGPT using (and citing) additional sources such as textbooks, scientific articles, or online resources.This procedure might help students develop a habit of verifying information from multiple sources.
• Lastly, it seems sensible to facilitate a class discussion, where students can share their findings and discuss the information they have analyzed ("share").
A meaningful follow-up task might be to take advantage of ChatGPT's ability to generate reasonings.For example, in classroom practice, it might be valuable to make ChatGPT argue for a particular position, e.g., one that does not conform to the scientific view.The students can then, firstly, point out flaws in ChatGPT's The prompt used in Figure 2 is formulated very specifically in order to get ChatGPT providing the required arguments in favor of a particle notion of the photon.Therefore, it may be helpful for students to revisit the think-pair-share exercise described earlier, but with different prompts, to compare ChatGPT's responses.

Development of a Conceptual Survey Using ChatGPT for Use in the Physics Classroom
ChatGPT is able to develop a conceptual survey (or single items) to assess students' understanding (Nasution, 2023), e.g., of basic quantum concepts (part a in Figure 3).On the one hand, items created by ChatGPT may initiate classroom discussions about correct solutions.From time to time, ChatGPT will produce items (or answers to these items) that are not (entirely) correct from a scientific point of view (part b in Figure 3)-for example, Gregorcic and Pendrill (2023) have found "ChatGPT to be a reliable source of problematic and incorrect answers to conceptual physics questions" (p.8).Such invalid questions produced by ChatGPT can be used in classroom to reflect on their scientific correctness.Therefore, ChatGPT can not only assist teachers in assessing students' understanding, but also provide students with the opportunity to develop their critical thinking skills and evaluate the accuracy of information.

USE OF THE EASY-TO-IMPLEMENT ACTIVITIES IN THE FIELD: A PILOT STUDY Research Question
The examples of using ChatGPT in the physics classroom introduced above were provisionally tested in a pilot study in the field in a two-lesson intervention.In this pilot, however, it was not our goal to examine the impact of the intervention on students' critical thinking skills because 1. a two-lessons-course may not be expected to have a substantial and sustainable impact without incorporation into the previous and further lessons and 2. our focus in the pilot study was on gathering first experiences in the implementation and on getting insights into the potential values of the use of ChatGPT in the physics classroom in general.
From a scientific perspective, we were instead interested in the impact of the intervention on students' opinions about AI in general and ChatGPT in particular, as little empirical research has been published on the practical implementation of ChatGPT in the classroom so far.Hence, we addressed the following research question: To what extent are students' perceptions of the importance of AI in general, and ChatGPT in particular, influenced by their participation in the intervention?

Study Design and Sample
The pilot study was conducted in the field and employed a one-group pretest-posttest design.Two 12 th grade physics classes (n=53 students, 30 male, 23 female) of a German high school (so-called Gymnasium) were involved.The intervention was implemented by an instructed teacher during regular physics lessons on quantum physics in which students had previously learned about wave-particle duality.
In total, the intervention comprised two 45-minute lessons and involved two main activities: In the first lesson, the students critically reviewed ChatGPT outputs on the nature of photons and discussed them in a think-pair-share format as described above.In the second lesson, they created a three-item conceptual survey on wave-particle duality for a peer.This required them to find an appropriate prompt for ChatGPT, check the created items for scientific validity, and work on the items developed by another student.The two partner students later discussed their solutions and shared their results in a class discussion.

Instrument
To collect students' perceptions of the importance of AI in general, and ChatGPT in particular, we adapted items from a questionnaire on students' opinions about quantum science used in an earlier study by Moraga-Calderón et al. (2020).This questionnaire is based on ROSE (relevance of science) questionnaire originally developed by Schreiner and Sjøberg (2004).
Additionally, we used three items from Chai et al. (2020).In total, our questionnaire comprised nine 5point rating-scale items (1 corresponds to "disagree", 2 to "rather disagree", 3 to "I do not know", 4 to "rather agree", and 5 to "agree").The questionnaire was administered prior and post the intervention.The items are provided in Figure 4.

Data Analysis
We report descriptive statistics (mean value [M], standard deviation [SD]) for students' ratings in pre-and posttests, respectively.We analyze the impact of the intervention on students perceptions' by comparing the students' ratings in the pretest with the ones in the posttest.Due to the small sample size and the preliminary nature of this pilot study, we refrain from an in-depth statistical analysis.

RESULTS
In Figure 4, the students' mean ratings on the nine items of the questionnaire for the pre-and posttest points in time are shown.A clear tendency towards the middle of the scale is noticeable among the participants in the pretest: This is presumably due to the fact that the students had no or little (instructional) contact with ChatGPT or other AI tools before the intervention and thus could neither agree nor disagree with the individual statements in the questionnaire.

Contemporary Educational Technology, 2023
Contemporary Educational Technology, 15(3), ep430 7 / 10 Regarding ChatGPT-related items, an increase in the students' average agreement can be observed for all but one statement: The students' agreement with item 8 "we can use ChatGPT even if we do not understand who it works" averaged 3.32 (SD=1.32) in the pretest while the mean value decreased to 2.66 (SD=0.95) in the posttest.This can possibly be attributed to the fact that students have become aware that ChatGPT outputs may not be readily consumed-however, with our pilot study, it is not possible to uncover the reasons underlying students' ratings.
The increase in students' agreement with item 7 ("We should all learn to incorporate ChatGPT in our lives") from pretest (M=3.53,SD=1.01) to posttest (M=4.60,SD=0.77) indicates that the intervention positively influenced students' perceptions of ChatGPT.The same holds true for the students' ratings on • item 5 ("the benefits of ChatGPT are greater that the harmful effects it could have") with an increase in the students' agreement from 3.13 (SD=1.24) to 4.45 (SD=1.03)and • item 6 ("I am hopeful about my future in a world, where ChatGPT is commonly used") with an increase in the students' agreement from 3.21 (SD=0.97) to 4.36 (SD=0.83).
These findings hint to optimistic views among the students with regards to the use of ChatGPT and its impact on their lives.Lastly, the increase in students' agreement with item 9 ("using ChatGPT enables me to accomplish tasks more quickly") from pretest (M=3.42,SD=1.18) to posttest (M=4.38,SD=0.73) suggests a positive impact of the intervention on students' opinions of the benefits of ChatGPT for teaching and learning.With respect to the perceptions' of the importance of AI in general, a notable augmentation in agreement with the items can been observed as well.However, the increase in students' agreement with these items is found to be considerably lower compared to the ones that are distinctly linked to ChatGPT (Figure 4).The latter, however, was expected since AI was not a specific part of the intervention in particular.

DISCUSSION AND CONCLUSION
In this article, we provided two easy-to-implement examples for the use of ChatGPT in physics education at the secondary school level to foster students' critical thinking skills.Although we demonstrated the different procedures using the context of quantum physics, they are applicable in secondary school practice in general.
The results of a pilot study examining the application of the presented examples for the use of ChatGPT in a physics classroom indicate that the intervention had a favorable influence on students' perceptions of ChatGPT.The participants demonstrated an increase in agreement with statements related to ChatGPT's benefits and its incorporation into their daily lives.On the other hand, while there was an increase in students' agreement with items related to AI in general, the increase was considerably lower compared to ChatGPTspecific items.In contrast to earlier research on students' attitudes toward emerging fields such as quantum science and technology, where students acknowledge the societal importance of the field without necessarily finding it relevant for their own learning (Moraga-Calderón et al., 2020), our pilot study on an intervention of ChatGPT in the physics classroom revealed a different trend: We found that while students considered AI of average importance, ChatGPT was deemed a valuable tool for a variety of purposes.This observation may be attributed to the fact that students gained firsthand experience of the potential benefits of ChatGPT through the intervention, which likely contributed to their positive perceptions of the tool.
In summary, it can be concluded that the intervention seems to be effective in influencing students' perceptions of ChatGPT, specifically.However, the impact on students' perceptions of AI in general requires further investigation.In particular, future research might explore (a) further valuable ways of integrating ChatGPT into classroom practice, and in particular (b) the impact of ChatGPT enhanced teaching on students' critical thinking skills.
In the end, it is noteworthy that while ChatGPT can enrich classroom practice as shown in this paper, learning will always remain a social process that requires teachers' authority and guidance (Pavlik, 2023).
Funding: The author received no financial support for the research and/or authorship of this article.

Ethics declaration:
Author declared that the ethical review and approval were waived for the study due to the fact that the study was in accordance with Local Legislation and Institutional Requirements: Research Funding Principles (https://www.dfg.de/en/research_funding/principles_dfg_funding/research_data/index.html)& General Data Protection Regulation (https://www.datenschutz-grundverordnung.eu/wp-content/uploads/2016/04/CONSIL_ST_5419_2016_INIT_EN_TXT.pdf).

Declaration of interest:
The author declares no competing interest.
Data availability: Data generated or analyzed during this study are available from the author on request.

Figure 1 .
Figure 1.Screenshot showing different prompts a) and b) used to ask ChatGPT about the meaning of term photon led to two slightly different answers (screenshots from https://chat.openai.com/chat)

Figure 2 .Figure 3 .
Figure 2. The user enforces ChatGPT to argue in favor of a naïve imagination of the photon as a spherical particle (screenshot from https://chat.openai.com/chat)

Figure 4 .
Figure 4. Students' mean ratings on nine items of questionnaire for the pre-and posttest points in time (Source: Author)