"

Seamlessly Integrating and Evaluating AI Competencies for Graduate Education Students

Rob Power

Author’s Note

AI-Generated contentThe following chapter contains content generated using large language model (LLM) artificial intelligence (AI) tools. Specific tools used, their purpose, and the generated content are identified and cited as they are presented.

Introduction

I don’t normally write academic pieces in a casual or first-person tone. But I figured it would be appropriate for this chapter, as it is not a report on new research. Rather, it is a look at a research project and a practical course activity that I conducted in 2023 with a group of graduate-level Education students. This chapter is meant to highlight a strategy that I’ve used to help develop critical competencies for the ethical use of Artificial Intelligence (AI) tools by educators and students. That strategy represents an example of leveraging AI as a tool to facilitate a seamless learning experience. It is also an example of the seamless integration of AI skills and competencies into the curriculum based on the Seamless Learning Education Design (SLED) framework (Hambrock et al., 2020) and, subsequently, into participants’ personal and professional contexts. To prepare this chapter, I followed a similar process to that used by my students. I leveraged AI tools to generate some of the content, then discussed those outputs. The first-person tone will help to set apart the content that I’ve authored from the AI-generated components.

Background

While facilitating my Fall 2023 Critical Issues in Leadership Education course, I conducted research into the impacts of direct exposure to AI tools on the self-efficacy of participant educators to use such tools in their own classrooms (Power, 2023c). The impetus for this research was the rapid proliferation of AI tools, reports on the levels of AI use amongst students, and critiques of the preparation of teachers to tackle the practical and ethical implications of such tools. This research (Power, 2024a, b, c) noted that while large percentages of students are frequently using AI tools such as ChatGPT (OpenAI, n.d.), teacher training and professional development programs are not adequately preparing educators to understand such tools, their implications, and how to use them ethically.

Participants in the Critical Issues in Leadership Education course completed a multi-stage activity that saw them use ChatGPT to output an essay on a topic with which they were already familiar. That familiarity was essential, as they then worked in pairs to conducted a detailed formal analysis of the output paper. They examined such aspects as writing style and formatting, as well as factual accuracy, omissions, and errors. The student pairs then produced multimedia presentations on the process of producing their initial essays using ChatGPT, the findings of their critiques, and their recommendations for students and teachers looking to integrate AI tools into classroom activities. Figure 1 is an infographic provided to course participants outlining the stages of the AI investigation project.

An infographic overvivew of the major course project investigating the use of ChatGPT
Figure 1: Infographic overview of the Critical Issues in Leadership Education AI investigation project.

The following video (Power, 2025) is a demonstration that I created for my students, showing how I queried ChatGPT to generate a roughly 1000-word sample paper on the topic of the importance of closed captions in instructional videos.

Generating AI ContentView a copy of my sample ChatGPT-generated essay on “Using AI to Generate a Paper About the Importance of Closed Captions in Educational Videos.”

 

Since the completion of the study, the findings have been shared via a peer-reviewed research article (Power, 2024c), a webinar as part of the International Research Network for Innovative Sustainable Seamless Education (IRN-ISSE) 2024 webinar series (Power & Hambrock, 2024) and a presentation at the Cape Breton University 2024 Grand Meeting (Power, 2024b).

Using AI to Ethically Summarize the Research

Generating AI ContentTo produce this chapter, I decided to follow a similar process to that experienced by my students. Since the study was conducted, a plethora of new AI tools targetting educational applications have emerged. Rather than using ChatGPT, I opted to explore the potential of one of these new tools, Google’s (2024a) NotebookLM. Whereas the former collates output information from the Internet at large, NotebookLM allows the user to create a folder with specific background materials that it will use to generate outputs to queries. In this case, I included a single resource in the “notebook folder” used for this chapter authoring exercise – my previous peer-reviewed journal article about the research study (Power, 2024c). NotebookLM was then asked to summarize the major concerns with rapid AI expansion for the education sector, how teacher self-efficacy with the use of AI tools could be increased, the research findings, and participants’ major recommendations for other teachers.

What Are the Major Concerns for the Education Sector?

The following is the NotebookLM response to the query, “What are the major concerns with the emergence of AI for the education sector?”.

AI-Generated contentNotebookLM’s Output (Google, 2024a)

Here are the major concerns regarding the emergence of AI in the education sector, based on the provided excerpts from the “Power (2024c)” document:

  • The rapid growth of AI agents like ChatGPT has raised concerns in the education sector because educators are largely unprepared for their use by students. More than half of Canadian post-secondary students have reported using AI agents to complete assignments or tests. Educators need training to understand how AI agents work, the implications of their use, and the appropriate pedagogical strategies to leverage them effectively in their teaching.
  • Significant concerns exist regarding AI agents’ potential for generating inaccurate or fabricated information. AI applications like ChatGPT can create plausible-sounding results that are actually erroneous. For example, ChatGPT has a tendency to fabricate citations that appear legitimate but aren’t real. This is a major concern because students may unknowingly produce biased, inaccurate, or incorrect content based on AI-generated information.
  • The widespread use of AI by students raises ethical concerns about academic integrity. There is a risk of students using AI agents unethically, such as submitting AI-generated work as their own, which may constitute academic misconduct.
  • While institutions are developing guidelines on the ethical use of AI agents, the deeper integration of AI technologies into educational tools requires a more comprehensive approach to educator preparation. Educators need training that goes beyond the basic understanding of AI to develop pedagogical approaches for integrating AI agents effectively into teaching and learning.
  • There is a lack of research and collaboration on the effective integration of AI in education. More research is needed to determine how to best train teachers to develop the necessary digital competencies to use AI effectively and responsibly. [6] Additionally, more collaboration is needed on developing AI applications, designing AI curricula, and researching AI’s application in higher education.
  • Teacher perceptions of technology may influence their willingness to adopt AI tools and develop new digital competencies. Addressing differences in teachers’ perceptions of technology will be important for successfully integrating AI into education.

The sources highlight the need for institutions to move beyond establishing policies and towards implementing formal professional development opportunities that focus on the pedagogical use of AI. By providing educators with hands-on experience, appropriate support, and training on pedagogical practices, institutions can better prepare them for the ethical and impactful integration of AI agents in teaching and learning.

How Can We Address Teacher Self-Efficacy with AI?

The following is the NotebookLM response to the query “How Can We Address Teacher Self-Efficacy with AI?”.

AI-Generated contentNotebookLM’s Output (Google, 2024a)

The sources offer several strategies for addressing teacher self-efficacy with AI:

  • Hands-on Experience and Training: The sources emphasize providing educators with hands-on experience using AI agents in conjunction with training on their functionalities, benefits, limitations, and implications. This approach can help educators understand how AI tools like ChatGPT can be effectively incorporated into their teaching practices.6 Providing just-in-time technical resources and support during this process can further enhance educators’ comfort and confidence with AI tools.
  • Targeted Professional Development: Institutions should implement formal professional development opportunities that focus specifically on the pedagogical applications of AI.16789 This training should equip teachers with strategies for integrating AI tools into their lesson planning, instruction, and assessment methods.
  • Collaboration and Peer Support: The sources suggest that educators can benefit from collaborating with each other to design and implement AI-related curricula and share their experiences and best practices. Creating peer support networks can provide a platform for educators to learn from one another, discuss challenges, and gain confidence in using AI tools.
  • Addressing Teacher Perceptions of Technology: It’s important to acknowledge and address any existing hesitation or negative perceptions teachers may have towards technology, as this can influence their willingness to adopt AI tools. Providing clear information about AI’s potential benefits for education and addressing concerns about job displacement can help alleviate anxiety and foster a more positive attitude towards AI integration.
  • Focus on Pedagogical Approaches: Training should emphasize the complementary roles between teachers and AI tools, ensuring that educators understand that AI is a tool to enhance, not replace, their expertise.67911 This approach can help teachers gain confidence in their ability to leverage AI while still maintaining their essential role as educators. Specifically, teachers could learn to utilize AI for tasks like content creation and differentiated instruction, while focusing on fostering student engagement, critical thinking, and ethical AI use.
  • Exposure to Successful Examples: Providing educators with real-world examples of how other teachers are effectively integrating AI into their classrooms can inspire and motivate them. Showcasing successful implementations can help educators visualize the potential of AI and develop their own innovative approaches.
  • Addressing Ethical Concerns: Training should address ethical concerns related to AI use in education, such as plagiarism, data privacy, and bias. This can empower educators to guide students on the responsible and ethical use of AI agents.
  • Using Tools to Gauge and Plan Professional Development: The sources describe the “ChatGPT Teacher’s Sense of Efficacy Scale” (Chat-T), a tool designed to measure teachers’ self-efficacy with AI and identify areas requiring targeted professional development. Although further research is needed to validate this tool, it demonstrates a potential method for understanding teachers’ needs and tailoring professional development to address them.

By implementing these strategies, educational institutions can help teachers gain the confidence and skills necessary to effectively and ethically integrate AI agents into their teaching practices, ultimately improving student learning outcomes.

Participants’ AI Confidence and Recommendations

For this section, NotebookLM was asked to generate an audio podcast about the findings of my research (Power, 2024c). NotebookLM was instructed to “discuss what students did as part of their major course projects, the findings about the impacts on their sense of self-confidence, and their recommendations for other teachers and educational leaders.”

NotebookLM’s Output (Google, 2024a, b)

AI-Generated content

NotebookLM’s Podcast Summary

This case study examines the impact of a ChatGPT-focused professional development activity on graduate education students’ self-efficacy. Researchers used a modified Teacher’s Sense of Efficacy Scale (Chat-T) to measure changes in confidence before and after students used ChatGPT to generate and critique academic essays. While the small sample size limits generalizability, the study suggests that hands-on experience with AI tools, combined with critical analysis, can increase awareness of both the potential and limitations of AI in education, although further pedagogical training is needed. The study also introduces the Chat-T scale as a potential tool for evaluating AI-related professional development, though further validation is required. The findings highlight the need for educators to receive more comprehensive training to integrate AI effectively and ethically in teaching.

Audio recordingNotebookLM’s Podcast


Audio transcript availableAudio Transcript

An Audio Transcript of the NotebookLM podcast has been generated using Microsoft Office 365’s Transcribe feature (Microsoft, 2024).

Impressions of the AI-Generated Content

Investigating AI ContentLike my students, I’ll take a few moments to discuss the merits of the AI-generated output presented in this chapter. Participants in the Chat-T research study uncovered concerning issues as they used ChatGPT to generate essays on their chosen topics. They discussed issues with the accuracy of some content, the omission of key points or perspectives, and potential biases resulting from the ChatGPT LLM algorithms. Having used NotebookLM while preparing this chapter, I have fewer content-related concerns with the AI outputs. That’s because the design of NotebookLM allowed me to limit the source material that was queried (to a single resource that I myself had authored). Verifying the accuracy and tone of the AI-generated output was thus a straightforward exercise.

While I would continue to use a more open LLM AI tool like ChatGPT to facilitate future iterations of the student project discussed in this chapter (for the purpose of exposing participants to the strengths and limitations of such tools), I would be more inclined to recommend a tool like NotebookLM for “daily” use in the classroom. It’s design provides a much higher degree of confidence in the accuracy of the content that students will interact with or produce. I hope that this chapter has provided a working example of powerful, ethical ways that educators and their students can leverage AI tools to enrich their interactions with and production of content.

Summary and Recommendations

Unfortunately, there is no quick and easy solution to some of the concerns around the rapid proliferation of AI tools and their potential impacts on educators and students. What is needed is a deeper understanding of how AI actually works and how it can be ethically applied in real-world contexts – including as part of teaching and learning. Once educators have this understanding, it will become easier to imagine creative and impactful ways to seamlessly integrate AI tools into classroom activities. As with findings about the experiences and educators during the COVID-19 pandemic (Power & Kay, 2023; Power et al., 2023), it can be overwhelming for teachers to try to master all of the new digital tools that emerge each year. What they find most effective is targeted pre-service training and professional development that focuses on the bigger picture of pedagogy and what they are trying to achieve through the use of new tools. This pedagogical understanding makes it easier to select, learn, and integrate new tools – including AI agents – when it is appropriate for their learning outcomes and the contexts and needs of their students. This finding is supported by the recent work of Crompton et al. (2024) on the Socia Ecological Technology Integration Framework (SETI), which could be used by institutions to frame their planning for ongoing faculty professional development and support.

The case study discussed in this chapter (Power, 2024c) is an example of the potential use of AI tools to create a seamless learning experience using the SLED framework—both through leveraging AI tools to facilitate the experience and the seamless integration of AI literacies and competencies as part of the curriculum itself. The skills and understandings gained by participants are ones that they can immediately integrate into their personal and professional contexts beyond the confines of the learning scenario presented.

References

Crompton, H., Burke, D., Nickel, C., & Chigona, A. (2024). The SETI framework and technology integration in the digital age. Asian Journal of Distance Education, 19(1). https://www.asianjde.com/ojs/index.php/AsianJDE/article/view/771

Hambrock, H., de Villiers, F., Rusman, E., MacCallum, K., and Arrifin, S. A., (2020). Seamless Learning in Higher Education: Perspectives of International Educators on its Curriculum and Implementation Potential (Rob Power, Editor). [eBook]. International Association for Mobile Learning. ISBN: 978-1-7751408-1-8. https://seamlesslearning.pressbooks.com/

Google (2024a). NotebookLM. [Large language model]. https://notebooklm.google/

Google (2024b). ChatGPT and Teacher Self-Efficacy NotebookLM Podcast. [AI-generated audio]. NotebookLM. [Large language model]. https://notebooklm.google.com/notebook/5543c634-af3c-4011-b2bd-6f5d052decf8/audio

Microsoft (2024). Transcribe your recordings. Microsoft Support. https://support.microsoft.com/en-us/office/transcribe-your-recordings-7fc2efec-245e-45f0-b053-2a97531ecf57

OpenAI (n.d.). ChatGPT. [Large language model]. https://chat.openai.com

Power, R. (2024a). ChatGPT Teacher’s Sense of Efficacy Scale (Chat-T). Power Learning Solutions. https://www.powerlearningsolutions.com/chat-t.html

Power, R. (2024b). Educator Confidence with AI: A Case Study and a New Research Tool. [Presentation File]. Invited presentation at Artificial Intelligence: The Balance of Innovation and Prevention, 21 March 2024, Cape Breton University, Sydney, NS, Canada. https://docs.google.com/presentation/d/e/2PACX-1vTGV1xxKk1h1-HOmQ3lNTn7PPSWufalJ3kRNLMQd3N4N64crVFCy6rx1JDl4iShoQ/pub?start=false&loop=false&delayms=3000

Power, R. (2024c). Evaluating Graduate Education Students’ Self-Efficacy with the Use of Artificial Intelligence Agents. Journal of Educational Informatics, 5(1), 3-19. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/269

Power, R. (2025). Using ChatGPT to Generate an Academic Paper. https://youtu.be/xiD-zjOGXMk

Power, R. and Hambrock, H. (2024, April 10). Seamless teaching and learning from an AI perspective. [Webinar]. AI for Education IRN-ISSE Webinar Series 2024.

Power, R. & Kay, R. (2023). Higher Education Faculty Supports for the Transition to Online Teaching during the COVID-19 Pandemic. Journal of Educational Informatics, 4(1), 49-72. https://journalofeducationalinformatics.ca/index.php/JEI/article/view/191

Power, R., Kay, R., & Craig, C. (2023). The Effects of COVID-19 on Higher-Education Teaching Practices. International Journal of E-Learning & Distance Education, 38(2). https://www.ijede.ca/index.php/jde/article/view/1255/1899

Stratvert, K. (2023, May 16). How to Transcribe Audio to Text in Word. [Video recording]. https://youtu.be/6dFQDIkd3r8

TechRepublic (2024, October 21). Create a PODCAST with NotebookLM in Minutes! [Video recording]. https://youtu.be/vQAmGTDsO6E

TheAIGRID (2024, October 20). How To Use NotebookLM For Beginners In 2024 (NotebookLM Tutorial). [Video recording]. https://youtu.be/1A9o-MalN0k

Appendices

ResourcesTry it Yourself: Implementing the AI Investigation Project

As I am writing this chapter, I am preparing to facilitate another section of the Critical Issues in Leadership Education course. Here are the assignment guides and assessment rubrics I’ll be using for the AI investigation project. Feel free to adapt them for your own course!

The Chat-T surveyThe Chat-T Tool

The ChatGPT Teacher’s Sense of Efficacy Scale (Chat-T) is available for use or adaptation under a Creative Commons 4.0 license via the Power Learning Solutions Chat-T webpage.

Generating AI ContentTry it Yourself: Ethically Using AI Tools to Write a Chapter

The following resources illustrate how to use the NotebookLM (Google, n.d.) and Trascribe (Microsoft, 2024) tools that I used to create the AI-generated content included in this chapter.

Getting Started with NotebookLM

How To Use NotebookLM For Beginners In 2024 (NotebookLM Tutorial) (TheAIGRID, 2024):

Generating an Audio Podcast in NotebookLM

Create a PODCAST with NotebookLM in Minutes! (TechRepublic, 2024):

Using Microsoft’s Transcribe Tool to Generate a Podcast Transcript

How to Transcribe Audio to Text in Word (Stratvert, 2023):

License

Icon for the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License

AI for Seamless Education Copyright © 2025 by International Research Network for Innovative Sustainable Seamless Education is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted.