Seamlessly Integrating and Evaluating AI Competencies for Graduate Education Students
Rob Power
Author’s Note
|
Introduction
I don’t normally write academic pieces in a casual or first-person tone. But I figured it would be appropriate for this chapter, as it is not a report on new research. Rather, it is a look at a research project and a practical course activity that I conducted in 2023 with a group of graduate-level Education students. This chapter is meant to highlight a strategy that I’ve used to help develop critical competencies for the ethical use of Artificial Intelligence (AI) tools by educators and students. That strategy represents an example of leveraging AI as a tool to facilitate a seamless learning experience. It is also an example of the seamless integration of AI skills and competencies into the curriculum based on the Seamless Learning Education Design (SLED) framework (Hambrock et al., 2020) and, subsequently, into participants’ personal and professional contexts. To prepare this chapter, I followed a similar process to that used by my students. I leveraged AI tools to generate some of the content, then discussed those outputs. The first-person tone will help to set apart the content that I’ve authored from the AI-generated components.
Background
While facilitating my Fall 2023 Critical Issues in Leadership Education course, I conducted research into the impacts of direct exposure to AI tools on the self-efficacy of participant educators to use such tools in their own classrooms (Power, 2023c). The impetus for this research was the rapid proliferation of AI tools, reports on the levels of AI use amongst students, and critiques of the preparation of teachers to tackle the practical and ethical implications of such tools. This research (Power, 2024a, b, c) noted that while large percentages of students are frequently using AI tools such as ChatGPT (OpenAI, n.d.), teacher training and professional development programs are not adequately preparing educators to understand such tools, their implications, and how to use them ethically.
Participants in the Critical Issues in Leadership Education course completed a multi-stage activity that saw them use ChatGPT to output an essay on a topic with which they were already familiar. That familiarity was essential, as they then worked in pairs to conducted a detailed formal analysis of the output paper. They examined such aspects as writing style and formatting, as well as factual accuracy, omissions, and errors. The student pairs then produced multimedia presentations on the process of producing their initial essays using ChatGPT, the findings of their critiques, and their recommendations for students and teachers looking to integrate AI tools into classroom activities. Figure 1 is an infographic provided to course participants outlining the stages of the AI investigation project.

The following video (Power, 2025) is a demonstration that I created for my students, showing how I queried ChatGPT to generate a roughly 1000-word sample paper on the topic of the importance of closed captions in instructional videos.
|
Since the completion of the study, the findings have been shared via a peer-reviewed research article (Power, 2024c), a webinar as part of the International Research Network for Innovative Sustainable Seamless Education (IRN-ISSE) 2024 webinar series (Power & Hambrock, 2024) and a presentation at the Cape Breton University 2024 Grand Meeting (Power, 2024b).
Using AI to Ethically Summarize the Research
To produce this chapter, I decided to follow a similar process to that experienced by my students. Since the study was conducted, a plethora of new AI tools targetting educational applications have emerged. Rather than using ChatGPT, I opted to explore the potential of one of these new tools, Google’s (2024a) NotebookLM. Whereas the former collates output information from the Internet at large, NotebookLM allows the user to create a folder with specific background materials that it will use to generate outputs to queries. In this case, I included a single resource in the “notebook folder” used for this chapter authoring exercise – my previous peer-reviewed journal article about the research study (Power, 2024c). NotebookLM was then asked to summarize the major concerns with rapid AI expansion for the education sector, how teacher self-efficacy with the use of AI tools could be increased, the research findings, and participants’ major recommendations for other teachers.
What Are the Major Concerns for the Education Sector?
The following is the NotebookLM response to the query, “What are the major concerns with the emergence of AI for the education sector?”.
|
How Can We Address Teacher Self-Efficacy with AI?
The following is the NotebookLM response to the query “How Can We Address Teacher Self-Efficacy with AI?”.
|
Participants’ AI Confidence and Recommendations
For this section, NotebookLM was asked to generate an audio podcast about the findings of my research (Power, 2024c). NotebookLM was instructed to “discuss what students did as part of their major course projects, the findings about the impacts on their sense of self-confidence, and their recommendations for other teachers and educational leaders.”
NotebookLM’s Output (Google, 2024a, b)
|
Impressions of the AI-Generated Content
Like my students, I’ll take a few moments to discuss the merits of the AI-generated output presented in this chapter. Participants in the Chat-T research study uncovered concerning issues as they used ChatGPT to generate essays on their chosen topics. They discussed issues with the accuracy of some content, the omission of key points or perspectives, and potential biases resulting from the ChatGPT LLM algorithms. Having used NotebookLM while preparing this chapter, I have fewer content-related concerns with the AI outputs. That’s because the design of NotebookLM allowed me to limit the source material that was queried (to a single resource that I myself had authored). Verifying the accuracy and tone of the AI-generated output was thus a straightforward exercise.
While I would continue to use a more open LLM AI tool like ChatGPT to facilitate future iterations of the student project discussed in this chapter (for the purpose of exposing participants to the strengths and limitations of such tools), I would be more inclined to recommend a tool like NotebookLM for “daily” use in the classroom. It’s design provides a much higher degree of confidence in the accuracy of the content that students will interact with or produce. I hope that this chapter has provided a working example of powerful, ethical ways that educators and their students can leverage AI tools to enrich their interactions with and production of content.
Summary and Recommendations
Unfortunately, there is no quick and easy solution to some of the concerns around the rapid proliferation of AI tools and their potential impacts on educators and students. What is needed is a deeper understanding of how AI actually works and how it can be ethically applied in real-world contexts – including as part of teaching and learning. Once educators have this understanding, it will become easier to imagine creative and impactful ways to seamlessly integrate AI tools into classroom activities. As with findings about the experiences and educators during the COVID-19 pandemic (Power & Kay, 2023; Power et al., 2023), it can be overwhelming for teachers to try to master all of the new digital tools that emerge each year. What they find most effective is targeted pre-service training and professional development that focuses on the bigger picture of pedagogy and what they are trying to achieve through the use of new tools. This pedagogical understanding makes it easier to select, learn, and integrate new tools – including AI agents – when it is appropriate for their learning outcomes and the contexts and needs of their students. This finding is supported by the recent work of Crompton et al. (2024) on the Socia Ecological Technology Integration Framework (SETI), which could be used by institutions to frame their planning for ongoing faculty professional development and support.
The case study discussed in this chapter (Power, 2024c) is an example of the potential use of AI tools to create a seamless learning experience using the SLED framework—both through leveraging AI tools to facilitate the experience and the seamless integration of AI literacies and competencies as part of the curriculum itself. The skills and understandings gained by participants are ones that they can immediately integrate into their personal and professional contexts beyond the confines of the learning scenario presented.
References
Crompton, H., Burke, D., Nickel, C., & Chigona, A. (2024). The SETI framework and technology integration in the digital age. Asian Journal of Distance Education, 19(1). https://www.asianjde.com/ojs/index.php/AsianJDE/article/view/771
Hambrock, H., de Villiers, F., Rusman, E., MacCallum, K., and Arrifin, S. A., (2020). Seamless Learning in Higher Education: Perspectives of International Educators on its Curriculum and Implementation Potential (Rob Power, Editor). [eBook]. International Association for Mobile Learning. ISBN: 978-1-7751408-1-8. https://seamlesslearning.pressbooks.com/
Google (2024a). NotebookLM. [Large language model]. https://notebooklm.google/
Google (2024b). ChatGPT and Teacher Self-Efficacy NotebookLM Podcast. [AI-generated audio]. NotebookLM. [Large language model]. https://notebooklm.google.com/notebook/5543c634-af3c-4011-b2bd-6f5d052decf8/audio
Microsoft (2024). Transcribe your recordings. Microsoft Support. https://support.microsoft.com/en-us/office/transcribe-your-recordings-7fc2efec-245e-45f0-b053-2a97531ecf57
OpenAI (n.d.). ChatGPT. [Large language model]. https://chat.openai.com
Power, R. (2024a). ChatGPT Teacher’s Sense of Efficacy Scale (Chat-T). Power Learning Solutions. https://www.powerlearningsolutions.com/chat-t.html
Power, R. (2024b). Educator Confidence with AI: A Case Study and a New Research Tool. [Presentation File]. Invited presentation at Artificial Intelligence: The Balance of Innovation and Prevention, 21 March 2024, Cape Breton University, Sydney, NS, Canada. https://docs.google.com/presentation/d/e/2PACX-1vTGV1xxKk1h1-HOmQ3lNTn7PPSWufalJ3kRNLMQd3N4N64crVFCy6rx1JDl4iShoQ/pub?start=false&loop=false&delayms=3000
Power, R. (2024c). Evaluating Graduate Education Students’ Self-Efficacy with the Use of Artificial Intelligence Agents. Journal of Educational Informatics, 5(1), 3-19. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/269
Power, R. (2025). Using ChatGPT to Generate an Academic Paper. https://youtu.be/xiD-zjOGXMk
Power, R. and Hambrock, H. (2024, April 10). Seamless teaching and learning from an AI perspective. [Webinar]. AI for Education IRN-ISSE Webinar Series 2024.
Power, R. & Kay, R. (2023). Higher Education Faculty Supports for the Transition to Online Teaching during the COVID-19 Pandemic. Journal of Educational Informatics, 4(1), 49-72. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/191
Power, R., Kay, R., & Craig, C. (2023). The Effects of COVID-19 on Higher-Education Teaching Practices. International Journal of E-Learning & Distance Education, 38(2). https://www.ijede.ca/index.php/jde/article/view/1255/1899
Stratvert, K. (2023, May 16). How to Transcribe Audio to Text in Word. [Video recording]. https://youtu.be/6dFQDIkd3r8
TechRepublic (2024, October 21). Create a PODCAST with NotebookLM in Minutes! [Video recording]. https://youtu.be/vQAmGTDsO6E
TheAIGRID (2024, October 20). How To Use NotebookLM For Beginners In 2024 (NotebookLM Tutorial). [Video recording]. https://youtu.be/1A9o-MalN0k
Appendices
|
|
|