Models to Measure Students’ Learning in Computer Science

As computer science becomes integrated into K-12 education systems worldwide, educators and researchers continuously search for effective methods to measure and understand students’ learning levels in this field. The challenge lies in developing reliable and comprehensive assessment models that accurately and discreetly gauge student learning. Teachers must assess learning to support students’ educational needs better. Similarly, students and parents expect schools to document students’ proficiency in computing and their practical application. Unlike conventional subjects such as math and science, very few relevant assessments are available for K-12 CS education. This article explores specific models used to measure knowledge in various CS contexts and then examines several examples of student learning indicators in computer science.

Randomized Controlled Trials and Measurement Techniques

An innovative approach to measuring student performance in computer science education involves evaluating the effectiveness of teaching parallel programming concepts. Research by Daleiden et al. (2020) focuses on assessing students’ understanding and application of these concepts.

The Token Accuracy Map (TAM) technique supplements traditional empirical analysis methods, such as timings, error counting, or compiler errors, which often need more depth in analyzing the cause of errors or providing detailed insights into specific problem areas encountered by students. The study applied TAM to examine student performance across two parallel programming paradigms: threads and process-oriented programming based on Communicating Sequential Processes (CSP), measuring programming accuracy through an automated process.

The TAM approach analyzes the accuracy of student-submitted code by comparing it against a reference solution using a token-based comparison. Each element of the code, or “token,” is compared to determine its correctness, and the results are aggregated to provide an overall accuracy score ranging from 0% to 100%. This scoring system reflects the percentage of correctness, allowing for a detailed examination of which students intuitively understand specific elements of different programming paradigms or are more likely to implement them correctly.

This approach extends error counts, offering insights into students’ mistakes at a granular level. Such detailed analysis enables researchers and educators to identify specific programming concepts requiring further clarification or alternative teaching approaches. Additionally, TAM can highlight the strengths and weaknesses of different programming paradigms from a learning perspective, thereby guiding curriculum development and instructional design.

Competence Structure Models in Informatics

Torsten et al. (2015) introduced a new model in their discussion aimed at developing a competence structure model for informatics with a focus on system comprehension and object-oriented modelling. This model, part of the MoKoM project (Modeling and Measurement of Competences in Computer Science Education), seeks to create a competence structure model that is both theoretically sound and empirically validated. The project’s goals include identifying essential competencies in the field, organizing them into a coherent framework, and devising assessments to measure them accurately. The study employed the Item Response Theory (IRT) evaluation methodology to construct the test instrument and analyze survey data.

The initial foundation of the competence model was based on theoretical concepts from international syllabi and curricula, such as the ACM’s “Model Curriculum for K-12 Computer Science” and expert papers on software development. This framework encompasses cognitive and non-cognitive skills pertinent to computer science, especially emphasizing system comprehension and object-oriented modelling.

The study further included conducting expert interviews using the Critical Incident Technique to validate the model’s applicability to real-world scenarios and its empirical accuracy. This method was instrumental in pinpointing and defining the critical competencies needed to apply and understand informatics systems. It also provided a detailed insight into student learning in informatics, identifying specific strengths and areas for improvement.

Limitations

The limitation of this approach is its specificity, which may hinder scalability to broader contexts or different courses. Nonetheless, the findings indicate that detailed, granular measurements can offer valuable insights into the nature and types of students’ errors and uncover learning gaps. The resources mentioned subsequently propose a more general strategy for assessing learning in computer science.

Evidence-centred Design for High School Introductory CS Courses

Another method for evaluating student learning in computer science involves using Evidence-Centered Design (ECD). Newton et al. (2021) demonstrate the application of ECD to develop assessments that align with the curriculum of introductory high school computer science courses. ECD focuses on beginning with a clear definition of the knowledge, skills, and abilities students are expected to gain from their coursework, followed by creating assessments that directly evaluate these outcomes.

The approach entails specifying the domain-specific tasks that students should be capable of performing, identifying the evidence that would indicate their proficiency, and designing assessment tasks that would generate such evidence. The model further includes an analysis of assessment items for each instructional unit, considering their difficulty, discrimination index, and item type (e.g., multiple-choice, open-ended, etc.). This analysis aids in refining the assessments to gauge student competencies and understanding more accurately.

This model offers a more precise measurement of student learning by ensuring that assessments are closely linked to curriculum objectives and learning outcomes.

Other General Student Indicators

The Exploring Computer Science website, a premier resource for research on indicators of student learning in computer science, identifies several key metrics for understanding concepts within the field:

Student-Reported Increase in Knowledge of CS Concepts: Students are asked to self-assess their knowledge in problem-solving techniques, design, programming, data analysis, and robotics, rating their understanding before and after instruction.
Persistent Motivation in Computer Problem Solving: This self-reported measure uses a 5-point Likert scale to evaluate students’ determination to tackle computer science problems. Questions include, “Once I start working on a computer science problem or assignment, I find it hard to stop,” and “When a computer science problem arises that I can’t solve immediately, I stick with it until I find a solution.”
Student Engagement: This metric again relies on self-reporting to gauge a student’s interest in further pursuing computer science in their studies. It assesses enthusiasm and inclination towards the subject.
Use of CS Vocabulary: Through pre- and post-course surveys, students respond to the prompt: “What might it mean to think like a Computer Scientist?”. Responses are analyzed for the use of computer science-related keywords such as “analyze,” “problem-solving,” and “programming.” A positive correlation was found between CS vocabulary use and self-reported CS knowledge levels.

Comparing the Models

Each model discussed provides distinct benefits but converges on a shared objective: to gauge precisely students’ understanding of computer science. The Evidence-Centered Design (ECD) model is notable for its methodical alignment assessments with educational objectives, guaranteeing that evaluations accurately reflect the intended learning outcomes. Conversely, the randomized controlled trial and innovative measurement technique present a solid approach for empirically assessing the impact of instructional strategies on student learning achievements. Finally, the competence structure model offers an exhaustive framework for identifying and evaluating specific competencies within a particular field, like informatics, ensuring a thorough understanding of student abilities. As the field continues to evolve, so will our methods for measuring student success.

References

Daleiden, P., Stefik, A., Uesbeck, P. M., & Pedersen, J. (2020). Analysis of a Randomized Controlled Trial of Student Performance in Parallel Programming using a New Measurement Technique. ACM Transactions on Computing Education, 20(3), 1–28. https://doi.org/10.1145/3401892

Magenheim, J., Schubert, S., & Schaper, N. (2015). Modelling and measurement of competencies in computer science education. KEYCIT 2014: key competencies in informatics and ICT, 7(1), 33-57.

Newton, S., Alemdar, M., Rutstein, D., Edwards, D., Helms, M., Hernandez, D., & Usselman, M. (2021). Utilizing Evidence-Centered Design to Develop Assessments: A High School Introductory Computer Science Course. Frontiers in Education, 6. https://doi.org/10.3389/feduc.2021.695376

February 11, 2024February 13, 2024

Potential of LLMs and Automated Text Analysis in Interpreting Student Course Feedback

Integrating Large Language Models (LLMs) with automated text analysis tools offers a novel approach to interpreting student course feedback. As educators and administrators strive to refine teaching methods and enhance learning experiences, leveraging AI’s capabilities could unlock more profound insights from student feedback. Traditionally seen as a vast collection of qualitative data filled with sentiments, preferences, and suggestions, this feedback can now be more effectively analyzed. This blog will explore how LLMs can be utilized to interpret and classify student feedback, highlighting workflows that could benefit most teachers.

The Advantages of LLMs in Feedback Interpretation

Bano et al. (2023) shed light on the capabilities of LLMs, such as ChatGPT, in analyzing qualitative data, including student feedback. Their research found a significant alignment between human and LLM classifications of Alexa voice assistant app reviews, demonstrating LLMs’ ability to understand and categorize feedback effectively. This indicates that LLMs can grasp the nuances of student feedback, especially when the data is rich in specific word choices and context related to course content or teaching methodologies.

LLMs excel at processing and interpreting large volumes of text, identifying patterns, and extracting themes from qualitative feedback. Their capacity for thematic analysis at scale can assist educators in identifying common concerns, praises, or suggestions within students’ comments, tasks that might be cumbersome and time-consuming through manual efforts.

Limitations and Challenges

Despite their advantages, LLMs have limitations. Linse (2017) highlights that fully understanding the subtleties of student feedback requires more than text analysis; it demands contextual understanding and an awareness of biases. LLMs might not accurately interpret outliers and statistical anomalies, often necessitating human intervention to identify root causes.

Kastrati et al. (2021) identify several challenges in analyzing student feedback sentiment. One major challenge is accurately identifying and interpreting figurative speech, such as sarcasm and irony, which can convey sentiments opposite to their literal meanings. Additionally, many feedback analysis techniques designed for specific domains may falter when applied to the varied contexts of educational feedback. Handling complex linguistic features, such as double negatives, unknown proper names, abbreviations, and words with multiple meanings commonly found in student feedback, presents further difficulties. Lastly, there is a risk that LLMs might inadvertently reinforce biases in their training data, leading to skewed feedback interpretations.

Tools and Workflows

According to ChatGPT (OpenAI, 2024), a suggested workflow for analyzing data from course feedback forms is summarized as follows:

Data Collection: Utilize tools such as Google Forms or Microsoft Forms to design and distribute course feedback forms, emphasizing open-ended questions to gather qualitative feedback from students.
Data Aggregation: Employ automation to compile feedback data into a single repository, like a Google Sheet or Microsoft Excel spreadsheet, simplifying the analysis process.
Initial Thematic Analysis: Import the aggregated feedback into qualitative data analysis software such as NVivo or ATLAS.ti. Use the software’s coding capabilities to identify recurring themes or sentiments in the feedback.
LLM-Assisted Analysis: Engage an LLM, like OpenAI’s GPT, to further analyze the identified themes, categorize comments, and potentially uncover new themes that were not initially evident. It’s crucial to review AI-generated themes for their accuracy and relevance.
Quantitative Integration: Combine qualitative insights with quantitative data from the feedback forms (e.g., ratings) using tools like Microsoft Excel or Google Sheets. This integration offers a more holistic view of student feedback.
Visualization and Presentation: Apply data visualization tools such as Google Charts or Tableau to create interactive dashboards or charts that present the findings of the qualitative analysis. Employing visual aids like word clouds for common themes, sentiment analysis graphs, and charts showing thematic distribution can render the data more engaging and comprehensible.

Case Study: Minecraft Education Lesson

ChatGPT’s recommended workflow was used to analyze feedback from a recent lesson on teaching functions in Minecraft Education.

Step 1: Data Collection

A Google Forms survey was distributed to students, which comprised three quantitative five-point Likert scale questions and three qualitative open-ended questions to gather comprehensive feedback.

Step 2: Data Aggregation

Using Google Forms’ export to CSV feature, all survey responses were consolidated into a single file, facilitating efficient data management.

Step 3: Initial Thematic Analysis

The survey data was then imported into atlas.ti, an online thematic analysis tool with AI capabilities, to generate initial codes from the quantitative data. This process revealed several major themes, providing valuable insights from the feedback.

Step 4: Manual Verification and Analysis

Upon reviewing the survey data manually, the main themes identified by Atlas.ti were confirmed. Additionally, this manual step highlighted specific approaches students took to solve problems presented in the lesson. Generally, the AI-generated codes were quite accurate, but a closer analysis of the comments (like the ones below) shows even more insightful student suggestions.

Step 5: Quantitative Integration

With both qualitative and quantitative data at hand, we bypass the need for a separate step for quantitative integration.

Step 6: LLM-Assisted Analysis and Visualization

Next, themes were further analyzed using ChatGPT’s code interpreter feature. ChatGPT helped analyze the data and summarized the aggregated data very accurately. It even provided Python code for generating additional visualizations, enhancing the interpretation of the feedback.

ChatGPT’s guidance facilitated the creation of insightful visualizations such as bar charts and word clouds.

Python offers a wealth of data visualization libraries for even more detailed analysis (https://mode.com/blog/python-data-visualization-libraries).

Best Practices for Using LLMs in Feedback Analysis

Research by Bano et al. (2023) and insights from Linse (2017) highlight the potential of LLMs and automated text analysis tools in interpreting student course feedback. Adopting best practices for integrating these technologies is critical for educators and administrators to make informed decisions that enhance teaching quality and the student learning experience, contributing to a more responsive and dynamic educational environment. Below are several recommendations:

Educators or trained administrators must review AI-generated themes and categorizations to ensure alignment with the intended context and uncover nuances possibly missed by the AI. This step is vital for identifying subtleties and complexities that LLMs may not detect.
Utilize insights from both AI and human analyses to inform changes in teaching practices or course content. Then, assess whether subsequent feedback reflects the effects of these changes, thereby establishing an iterative loop for continuous improvement.
Offer guidance on using Student course evaluations constructively. This involves understanding the context of evaluations, looking beyond average scores to grasp the distribution, and considering student feedback as one of several measures for assessing and enhancing teaching quality.
This process should act as part of a holistic teaching evaluation system, which should also encompass peer evaluations, self-assessments, and reviews of teaching materials. A comprehensive approach offers a more precise and balanced assessment of teaching effectiveness.

References

Bano, M., Didar Zowghi, & Whittle, J. (2023). Exploring Qualitative Research Using LLMs. ArXiv (Cornell University). https://doi.org/10.48550/arxiv.2306.13298

Linse, A. R. (2017). Interpreting and using student ratings data: Guidance for faculty serving as administrators and on evaluation committees. Studies in Educational Evaluation, 54, 94–106. https://doi.org/10.1016/j.stueduc.2016.12.004

Kastrati, Z., Dalipi, F., Imran, A. S., Pireva Nuci, K., & Wani, M. A. (2021). Sentiment Analysis of Students’ Feedback with NLP and Deep Learning: A Systematic Mapping Study. Applied Sciences, 11(9), 3986. https://doi.org/10.3390/app11093986

OpenAI. (2024). ChatGPT (Feb 10, 2024) [Large language model]. https://chat.openai.com/chat

January 28, 2024January 28, 2024

Effective Technology Tools for K-12 CS Teachers

Technology plays a crucial role in teaching computer science and programming concepts to K-12 teachers. The most effective technology tools include interactive coding platforms such as Scratch, Snap! and Blockly. These tools provide a user-friendly interface and visual coding blocks, allowing students to learn programming concepts through hands-on activities and projects (Kashif Amanullah & Bell, 2020). Additionally, online learning platforms such as Code.org offer computer science platforms specifically designed for K-12 teachers. This blog examines various technologies used to teach CS in K-12 schools, drawing insights from a comprehensive study on visual programming languages (VPLs) and their suitability across different school levels.

Role of VPLs in K-12 Education:

VPLs like Scratch and ALICE have revolutionized CS education in schools. Scratch, developed by MIT, is particularly effective in elementary education due to its simplicity and interactive environment, making it an ideal tool for introducing programming concepts (Sáez-López et al., 2016). Although not web-based, ALICE has positively impacted all educational levels – elementary, high school, and undergraduate. Its ability to facilitate learning and enhance student confidence makes it an asset in the CS curriculum (Graczynska, 2010). In a 2019 study, do Nascimento et al. concluded that different visual programming languages (VPL) suit different school levels. The study focused on three VPLs: ALICE, Scratch, and iVProg. The findings indicate that Scratch is strongly suitable for elementary education, while ALICE is more appropriate for high school students. iVProg, on the other hand, has indications of suitability for high school and undergraduate levels.

Enhancing Computational Thinking with Scratch

Studies have shown that Scratch’s block-based programming approach can significantly improve students’ computational thinking skills. Its integration into various disciplines through programming games and projects encourages creative problem-solving and logical reasoning among students (Stewart & Baek, 2023). In a significant study, Scratch was also found to integrate well into other subjects in the curriculum, such as math, science, and even art and history, where students achieved comprehension and application levels in Bloom’s taxonomy (Sáez-López et al., 2016).

The advantages of using Scratch in the classroom are that its intuitive drag-and-drop interface simplifies the programming process, allowing students to focus on the logic behind their creations rather than the code syntax. Overall, the visual programming approach via Scratch was effective for developing computational thinking, improving programming skills, enabling the creation of interactive projects, and supporting active learning pedagogies (Sáez-López et al., 2016). This is significant since Sun, Hu, and Zhou (2022) found that although girls in K-12 had higher computational thinking skills, they had more negative programming attitudes, which may impact their continued development in computational thinking. Visual programming may be a good strategic approach to engage females in computer science.

ALICE for STEAM Education

ALICE (which stands for Alice Learning in a Cyberworld Environment) is a free 3D programming platform developed at Carnegie Mellon University. The visual aspect of ALICE makes programming concepts more engaging and hands-on for students. Actions like loops, methods, and events correspond to actual animated motions they can see on screen. This helps concretize abstract coding notions that beginners often struggle to grasp.

Graczyńska (2010) highlights several example uses of ALICE targeted at middle school students:

Creating videos set to music, with lyrics displayed as subtitles. This combines coding with music appreciation and language arts.
Recording narration for animations, like reciting poetry in English or other languages. This boosts public speaking and foreign language skills.
Building simple games with sound effects and animations like fire. This makes programming exciting and fun.

After testing ALICE with students, Graczyńska found increased engagement and interest in programming and academics overall. The visual nature of ALICE also helps attract female students to computer science, where they are traditionally underrepresented.

The use of 3D visual programming tools like Alice has shown positive effects on students’ performance and attitude towards computer programming. Al-Tahat (2019) found that teaching visual programming greatly improved understanding of related concepts in object-oriented programming, making it a perfect fit for the intermediate grades.

Challenges and Future Directions:

The adoption of these technologies in K-12 computer science (CS) education has shown promise, yet challenges remain to be addressed. There is substantial evidence that incorporating VPLs into the K-12 curriculum significantly boosts female engagement (Sun et al., 2022; Graczyńska, 2010). Therefore, it is important to focus on course design that appeals to diverse learners, including females and underrepresented minorities. Additionally, ongoing research and development are necessary to keep up to date with technological progress and the changing needs of education (McGill et al., 2023). Sáez-López et al. (2016) have suggested that VPLs should be implemented across various subjects, particularly in social sciences and the arts, where their visual nature can inspire creative projects. Lastly, the successful integration of new programming tools hinges on teacher training and professional development. Teachers need robust support to acquire and apply these technologies effectively.

References

Kashif Amanullah, & Bell, T. (2020). Teaching Resources for Young Programmers: the use of Patterns. https://doi.org/10.1109/fie44824.2020.9273985

Sáez-López, J.-M., Román-González, M., & Vázquez-Cano, E. (2016). Visual programming languages integrated across the curriculum in elementary school: A two year case study using “Scratch” in five schools. Computers & Education, 97, 129–141. https://doi.org/10.1016/j.compedu.2016.03.003

Graczyńska, E. (2010). ALICE as a tool for programming at schools. Natural Science, 02(02), 124–129. https://doi.org/10.4236/ns.2010.22021

do Nascimento, M. D., Felix, I. M., Ferreira, B. M., de Souza, L. M., Dantas, D. L., de Oliveira Brandao, L., & de Oliveira Brandao, A. (2019). Which visual programming language best suits each school level? A look at Alice, iVProg, and Scratch. 2019 IEEE World Conference on Engineering Education (EDUNINE). https://doi.org/10.1109/edunine.2019.8875788

Stewart, W., & Baek, K. (2023). Analyzing computational thinking studies in Scratch programming: A review of elementary education literature. International Journal of Computer Science Education in Schools, 6(1), 35–58. https://doi.org/10.21585/ijcses.v6i1.156

Sun, L., Hu, L., & Zhou, D. (2022). Programming attitudes predict computational thinking: Analysis of differences in gender and programming experience. Computers & Education, 181, 104457. https://doi.org/10.1016/j.compedu.2022.104457

Graczyńska, E. (2010). ALICE as a tool for programming at schools. Natural Science, 02(02), 124–129. https://doi.org/10.4236/ns.2010.22021

Al-Tahat, K. (2019). The Impact of a 3D Visual Programming Tool on Students’ Performance and Attitude in Computer Programming. Journal of Cases on Information Technology, 21(1), 52–64. https://doi.org/10.4018/jcit.2019010104

December 4, 2023December 4, 2023

Teaching Programming with Minecraft Education: A Reflection

Introduction

Integrating innovative tools to enhance learning is essential in the dynamic landscape of computer science education. This term, I embarked on a collaborative journey to weave Minecraft Education into a Programming 11/12 course. Our objective was to enliven the curriculum by presenting programming concepts in a more engaging and interactive manner. This reflection delves into our experiences, with a particular focus on the concept of functions.

Lesson Overview

Our lesson was carefully prepared to guide students through the fundamentals of functions in programming via the Minecraft Education platform. This approach aimed to convert abstract concepts into concrete, relatable experiences, thus making learning both enjoyable and impactful.

The session began with a simple introduction to functions in Minecraft Education using MakeCode, drawing parallels with real-life scenarios to demystify these concepts. The goal was to underscore the significance of reusing code efficiently. For instance, we showcased a function that could construct various parts of a structure, such as walls, roofs, and fences. This hands-on demonstration helped students visualize the workings of functions, deepening their comprehension.

Subsequently, we organized the students into small teams for a series of Minecraft challenges. Each group applied their newfound knowledge to construct farm elements using coded functions. Encouraging students to build barns, animal enclosures, and residential structures, this immersive experience was crucial in reinforcing the lessons imparted and empowering students to explore coding within the game environment. While the MakeCode IDE is freely available online at https://minecraft.makecode.com/, it is important to note that witnessing the code’s execution within Minecraft Education itself requires a paid subscription for each student (which we lacked for this iteration).

Following the building activities, groups presented their projects, explained their code, and engaged in Q&A sessions. This exercise culminated in the creation of a complete farm ecosystem (with a small amount of manual intervention), facilitating peer learning and evaluating their understanding of the lesson.

The lesson wrapped up with a debriefing segment, which focused the role of functions in streamlining complex coding tasks. We also distributed surveys to gauge the students’ experiences with the lesson.

Reflections and Learnings

Reflecting on the teaching process, I’ve recognized the crucial need for thorough preparation ahead of each class. Although the lesson itself was effective, there are areas where we could have utilized our time more judiciously.

Time Management:

Our planning meetings often veered towards administrative topics, detracting from the core lesson content. This experience has ingrained in me the importance of arriving at meetings well-prepared and with preliminary research completed, to maximize our collaborative efforts.

Technical Challenges:

Establishing a connection to the same Minecraft world across various platforms, such as PC and Mac, presented significant hurdles. This impacted our preparations and underscored the necessity for preemptive compatibility checks for future sessions. The tightly controlled environment of Minecraft Education by Microsoft impeded remote learning, suggesting that Minecraft Education is best suited to in-lab settings. Remote functionality was unreliable, as indicated by non-descriptive connection error messages like “timed out,” and support from Microsoft was less than helpful. The trial version of the software, supposedly available to schools with Microsoft logins, also failed to work, potentially necessitating IT intervention.

Student Engagement:

The lesson garnered positive feedback and high engagement levels, with the practical application of programming concepts within a familiar gaming environment being a key factor in its success. Nonetheless, some students noted that the inability to run the code hindered the debugging process. Ensuring every student has access to the necessary software and hardware will be a priority for future lessons.

The Power of Interactive Learning:

A major insight from this endeavour is the profound impact of interactive learning tools such as Minecraft in teaching intricate subjects like programming. Students were more engaged and assimilated the concept of functions more thoroughly compared to conventional teaching methods.

Conclusion

Incorporating Minecraft into our programming curriculum has been enlightening for students and educators. It has accentuated the significance of preparation, flexibility, and the assurance of technical compatibility to facilitate a seamless learning experience. The positive student feedback and evident boost in engagement and comprehension underscore our conviction in the power of interactive learning tools. As we progress, we are determined to refine our methods, confront the technical obstacles, and seek inventive strategies to render education more captivating and effective.

November 11, 2023

The Role of ChatGPT in Introductory Programming Courses

Introduction

Programming education is on the cusp of a major transformation with the emergence of large language models (LLMs) like ChatGPT. These AI systems have demonstrated impressive capabilities in generating, explaining, and summarizing code, leading to proposals for their integration into coding courses. Aligning with ISTE Standard 4.1e for coaches, which urges the “connection of leaders, educators, and various experts to maximize technology’s potential for learning,” this post examines how ChatGPT and similar tools can be effectively integrated into introductory programming classes. It covers the benefits of AI tutors, insights from educators on their use, and current best practices and trends for deployment in the classroom.

The Current State of AI in Computer Science Education

The current integration of AI in computer science education is showing promising results. ChatGPT excels in providing personalized and patient explanations of programming concepts, offering code examples and solutions tailored to students’ individual needs. Its interactive conversational interface encourages students to engage in a dialogue, solidifying their understanding through active participation and feedback. Students can present coding issues in simple terms and receive a comprehensive, step-by-step explanation from ChatGPT, clarifying fundamental principles throughout the process.

Such dynamic assistance clarifies misunderstandings more effectively than static textbooks or videos. ChatGPT’s round-the-clock availability as an AI tutor offers crucial support, bridging gaps when human instructors are unavailable. According to research by Kazemitabaar et al. (2023), using LLMs like ChatGPT can bolster students’ abilities to design algorithms and write code, reducing the stress often accompanying these tasks. The study also noted increased enthusiasm for learning programming among many students after exposure to LLM-based instruction.

Pros of Incorporating ChatGPT into the Classroom

The rapid advancement of AI systems such as ChatGPT offers many opportunities and poses some challenges in computing education. ChatGPT’s conversational interface and its capability to provide personalized content make it an exceptional asset for adaptive learning in AI-assisted teaching. Biswas (2023) identifies multiple applications for LLMs in educational settings, including their role in creating practice problems and code examples that enhance teaching. Furthermore, ChatGPT can anticipate and provide relevant code snippets tailored to the programming task and user preferences, accelerating development processes. It can also fill in gaps in code by analyzing the existing framework and project parameters. Additionally, LLM-facilitated platforms help with explanations, documentation, and resource location for troubleshooting and diagnosing issues from error messages, streamlining debugging and reducing the time spent on minor yet frustrating problems.

Cons of Incorporating ChatGPT in Education

Despite the advantages of ChatGPT, there is concern that its proficiency in solving basic programming tasks may lead to student overreliance on its code generation, potentially diminishing actual learning, as evidenced by Finnie-Ansley et al. (2022) and Kazemitabaar et al. (2023). Finnie-Ansley’s research indicates that, while LLMs can perform at a high level (scoring in the top quartile on CS1 exams), they are not without significant error rates. Moreover, the benefits attributed to ChatGPT, such as code completion, syntax correction, and debugging assistance, overlap with features already available in modern Integrated Development Environments (IDEs).

Concerns extend to ChatGPT facilitating ‘AI-assisted cheating,’ which threatens academic integrity and assessment validity (Finnie-Ansley et al., 2022). To counteract this, researchers suggest crafting more innovative, conceptual assignments beyond simple coding tasks (Finnie-Ansley et al., 2022; Kazemitabaar et al., 2023). Educators in computing must adopt careful strategies for integrating ChatGPT, using it as a scaffolded instructional tool rather than a crutch for solving exam problems, to maintain a focus on in-depth learning.

Instructors’ Perspectives and Experiences

In a study conducted in 2023, Lau and Guo interviewed 20 introductory programming instructors from nine countries regarding their adaptation strategies for LLMs like ChatGPT and GitHub Copilot. In the near term, most instructors intend to limit the use of LLMs to curb cheating on assignments, which they view as a potential detriment to learning. Their strategies range from emphasizing in-person examinations to scrutinizing code submissions for patterns indicative of LLM use and outright prohibiting certain tools. Some, however, are keen to explore the capabilities of ChatGPT, proposing its cautious application, such as demonstrating its limitations to students by having them assess its output against test cases.

In contemplating the future, these educators showed greater willingness to integrate LLMs as teaching tools, recognizing their congruence with real-world job skills, their potential to enhance accessibility, and their use in facilitating more innovative forms of coursework. For example, they discussed transitioning from having students write original code to evaluating and improving upon code produced by LLMs—a few envisioned LLMs functioning as custom-tailored teaching aids for individual learners.

Pedagogical Strategies and Opportunities for Future Research

Designing problems that demand a deep understanding of concepts rather than the execution of routine coding tasks, which LLMs easily handle, is a vital pedagogical shift proposed by Finnie-Ansley et al. (2022) and Kazemitabaar et al. (2023). Utilizing ChatGPT as an interactive educational tool to complement teaching—instead of as a mere solution provider—may strike an optimal balance between its advantages and potential drawbacks. Given the pace at which AI technology is being adopted in education, there’s a pressing need for further empirical research to identify the most effective ways to integrate these tools and assess their impact on student learning.

References

Biswas, S. (2023). Role of ChatGPT in Computer Programming. Mesopotamian Journal of Computer Science, 8–16. https://doi.org/10.58496/mjcsc/2023/002

Kazemitabaar, M., Chow, J., Carl, M., Ericson, B. J., Weintrop, D., & Grossman, T. (2023). Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. https://doi.org/10.1145/3544548.3580919

Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022). The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. Australasian Computing Education Conference. https://doi.org/10.1145/3511861.3511863

Lau, S., & Guo, P. (2023, August). From” Ban it till we understand it” to” Resistance is futile”: How university programming instructors plan to adapt as more students use AI code generation and explanation tools such as ChatGPT and GitHub Copilot. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 1 (pp. 106-121). https://doi.org/10.1145/3568813.3600138

October 15, 2023October 16, 2023

Teaching Computer Science with Minecraft

Introduction to Minecraft

Minecraft is currently one of the most popular games of 2023, boasting over 140 million monthly active users, according to searchlogistics.com. Despite this popularity, many players overlook that Minecraft offers an engaging and immersive environment for learning terminal commands, programming basics, computational thinking, and even artificial intelligence. ISTE standard 4.3a for coaches indicates that a successful coach should “Establish trusting and respectful coaching relationships that encourage educators to explore new instructional strategies.” So, in this blog post, I will delve into the educational benefits of Minecraft and explore the differences between the Java and Education editions.

While Minecraft is often regarded as merely a game, educators have recognized its potential as a valuable learning tool. At its core, Minecraft is built upon programming concepts. Players use blocks made of various materials to construct anything they can imagine, from simple houses to complex machines that require advanced knowledge of electronics, chemistry, and physics. This encourages computational thinking, creativity, and problem-solving as students work to bring their visions to life.

Concerning programming, Minecraft helps teach fundamental coding concepts, including commands, functions, variables, loops, and conditionals. Students can employ block-based coding or full-fledged programming languages such as Python and JavaScript to automate actions within the game. This hands-on approach to learning captivates students more effectively than traditional coding lessons, as Minecraft provides them with an imaginative space to immediately apply their newfound skills. Creating Minecraft modifications (mods) teaches students how to extend existing programs, a critical programming skill.

Minecraft Versions

Several versions of Minecraft are available for players to choose from, including Minecraft: Java Edition, Minecraft: Bedrock Edition, Minecraft: Education Edition, and Minecraft: Pocket Edition. However, for the specific purpose of our educational analysis, we will concentrate solely on the Java and Education editions. These two versions offer unique features and opportunities for learning that make them particularly relevant in an educational context.

Minecraft: Java Edition

The Java Edition is the original version of Minecraft developed in 2009 by Mojang Studios for Windows, macOS, and Linux, and maintains its popularity among long-time Minecraft players.

The Java Edition offers distinct advantages when teaching advanced computer science concepts due to its “mod-ability” and access to the source code of the game environment. The semi-open-source nature of the Java Edition allows for limitless customization through mods and plugins. Writing mods can illustrate a wide range of advanced programming concepts, including event handling, parallel programming, algorithms, data structures, debugging, and software design patterns. Developing mods not only imparts practical software development skills but also encourages students to show their creativity.

The Minecraft community has produced numerous mods that cater to various lesson plans. For instance, ComputerCraft introduces programmable turtle robots, while RedstonePlus enhances the game with advanced circuitry. The diversity of available mods supports a wide range of educational objectives, not only in CS but other disciplines.

Minecraft: Education/Bedrock Edition

Opening screen of Minecraft: Education Edition

Minecraft: Bedrock Edition was initially released in August 2011 and is particularly advantageous for classrooms with various devices. Bedrock Edition supports mobile devices such as iPads and Android tablets, which many schools already incorporate into their teaching environments. This enables students to start their Minecraft lessons on a classroom desktop computer during the day and seamlessly continue playing on their smartphones or game consoles at home.

However, Bedrock Edition offers less mod support and limited access to code customization. Minecraft Education Edition is a version of Bedrock specifically tailored for classroom use. According to Microsoft, it “typically runs about one full version behind the current Minecraft Bedrock production version” (FAQ: Game Features, 2023).

Advantages of Minecraft Education in the Classroom

One of the most significant advantages of Minecraft Education in a computer science course is its block-based CodeBuilder / MakeCode editor, similar to Scratch or Snap. This editor allows students to drag and drop commands to perform actions in the game. Younger students can learn coding logic and structure by creating houses, gardens, and machines using these visual blocks before transitioning to text-based programming languages like Python or JavaScript.

Another advantage of Education Edition is the teachers’ ability to implement special restrictions, such as limiting chat or preventing students from destroying blocks. These classroom controls create a safe environment for student exploration. Teachers can also switch to spectator mode to observe students and provide feedback; they also have the capability to build worlds and restrict access as needed. Here is a quick start guide for reference.

The Education Edition library offers hundreds of pre-made interactive worlds and lesson plans aligned with computer science curriculum standards (source: https://education.minecraft.net/en-us/resources/computer-science-subject-kit). Teachers can find lesson plans tailored to any grade level, making it much easier for educators to get started with Minecraft compared to building worlds from scratch.

According to research by Bile (2022), their study found that children aged 8 to 10 in a Minecraft education setting were able to solve abstract and complex scientific problems without prior prompting or theoretical knowledge. The game format also helped students retain knowledge better. Vostinar & Dobrota (2022) similarly found that in a primary school class, even though the majority of students had not programmed before in block or Python, they found the lesson enjoyable and easy. Furthermore, according to Nika Klimová et al. (2021), girls in grades 5-10 typically outperform boys in Minecraft education coding challenges, suggesting it may be a valuable tool for increasing diversity in computer science.

Disadvantages of Minecraft

As Vostinar & Dobrota (2022, p. 652) pointed out, there are significant disadvantages to using Minecraft in education. One such drawback is that Minecraft is not free and requires an additional cost per student, which, as mentioned in my previous post, raises ethical concerns about the practice of making students pay for educational software. Another disadvantage is that Minecraft may only appeal to a certain type of student, particularly those with a more creative inclination, potentially excluding students who do not have an affinity for the game.

Furthermore, teachers must become proficient in the game’s mechanics and capabilities to integrate it into the classroom effectively. Given the abundance of “cheats” in Minecraft, more experienced players may find trivial command-line solutions to problems if the teacher is unaware of their existence. Finally, as highlighted by Vostinar & Dobrota (2022), it’s essential to impose adequate constraints on the virtual world, especially when students collaborate, to prevent them from destroying the world with TNT blocks and other mining tools.

References:

Vostinar, P., & Dobrota, R. (2022). Minecraft as a Tool for Teaching Online Programming. 2022 45th Jubilee International Convention on Information, Communication and Electronic Technology (MIPRO). https://doi.org/10.23919/mipro55190.2022.9803384

Bile, A. (2022). Development of intellectual and scientific abilities through game-programming in Minecraft. Education and Information Technologies, 1–16. https://doi.org/10.1007/s10639-022-10894-z

Nika Klimová, Jakub Sajben, & Lovászová, G. (2021). Online Game-Based Learning through Minecraft: Education Edition Programming Contest. https://doi.org/10.1109/educon46332.2021.9453953

FAQ: Game Features. (2023, September 15). Minecraft Education. https://educommunity.minecraft.net/hc/en-us/articles/360047117692-FAQ-Game-Features

August 13, 2023

Reflecting on a Study of Competitive Programming and Cultural Inclusion

Length of Study

The study is designed to take place over two academic terms, which provides adequate time to collect meaningful data. The inclusion of an initial summer term without competitive programming establishes a baseline for comparison. The second summer term incorporates competitive programming using standardized questions, allowing assessment of this pedagogical approach. The fall term offering adds the dimension of culturally relevant questions, enabling analysis of their impact. Extending the study over multiple terms enables more robust data collection and analysis.

Promoting Active and Engaged Learning

The core content is delivered through weekly lectures focused on programming concepts. The competitive programming contests complement the lectures by providing opportunities to practice applying concepts. Weekly competitive programming contests foster active learning in several key ways. Students must apply conceptual knowledge to solve concrete programming problems. This process reinforces their understanding and helps identify knowledge gaps. The contest format adds an engaging gamification element through scoring, feedback, and peer comparison. Using standardized questions initially assesses whether baseline content needs are being met.

Introducing culturally relevant questions aims to promote better integration of concepts by relating them to students’ cultural knowledge and experiences. Having students co-create contest questions in the fall term further activates learning. They must think critically to develop culturally relevant problems that integrate with the content. This approach promotes deeper engagement with the material and encourages collaboration with classmates, allowing students to take ownership of their learning.

Addressing Teachers’ Needs

The study aims to provide teachers with insight into using competitive programming and culturally relevant pedagogy. The data collected will help determine the effectiveness of these approaches in an international educational setting. Instructors will gain an understanding of how competitive programming engages students versus standardized practice problems. They will also see whether student-created culturally relevant questions increase participation and motivation. The study addresses teachers’ needs for effective and inclusive instructional strategies. They will gain practical knowledge from the comparative data on different contest designs.

Promoting Collaborative Participation

Collaboration is encouraged through the group development of culturally relevant contest questions. Students can brainstorm and build on each other’s ideas, which fosters teamwork. Producing questions from diverse cultural perspectives requires working together. Students are also given the choice of problem-solving in teams. Students can motivate each other and strategize in groups for the competitions. Their scores are tracked on a collective leaderboard which reinforces the collaborative element. The shift from individual to team contest creation necessitates and enables productive collaboration.

The multi-term study design, interactive contest format, customized problems, and collaborative elements demonstrate an interesting pedagogical approach that promotes engaged and inclusive learning. The results should provide valuable insights for computer science educators.

July 9, 2023

Incorporating Competitive Programming into a Beginner Programming Course

Introduction

Driven by the increasing automation and digitalization of virtually every workflow, programming has become an indispensable part of our lives. As a result, introducing programming at the earliest stage of education has become a hot topic of discussion among educators and academics alike.

A particular area of interest is the concept of competitive programming (CP). Long viewed as a niche domain, a small group of enthusiasts often pursue CP to challenge their coding capabilities; many faculty have challenged the area as an unnecessary part of computer science. However, recent research underscores the potential of competitive programming as a useful pedagogical tool, especially in the context of introductory programming courses. This blog post will discuss the results of various studies that have been conducted on incorporating CP into a beginner’s programming course. I’ll review existing studies on integrating CP into intro-level programming courses, examining its effects on learning outcomes, student engagement, and skill acquisition. In addition, I will also propose some areas of CP that require further research.

Understanding Competitive Programming

Competitive programming is a mind sport, like chess and bridge, that involves participants competing to solve algorithmic problems as quickly and efficiently as possible. The ACM ICPC (Association for Computing Machinery – International Collegiate Programming Contest) is one of the world’s oldest, largest, and most prestigious programming contests, which started in the 1970s. Today, it has grown to involve tens of thousands of participants, attracting the world’s top Computer Science universities.

Several elements define each problem in the contest. First, there’s a problem statement describing the issue the team needs to solve. Next are the input and output specifications, which explain the type of data the team’s program should accept and produce. Thirdly, sample inputs and outputs are given to help the team understand the problem. Finally, constraints are provided to outline the maximum size or other limitations of the inputs and the required efficiency of the solution.

The contest is scored based on the number of problems solved and the time penalty. The number of problems solved is the most critical factor; the more problems a team solves, the higher their rank will be. Teams are primarily ranked by the number of problems they have solved. To break ties among teams who have solved the same number of problems, the ICPC uses a time penalty calculated from the beginning of the contest to the time of the first correct submission, with an additional penalty added for each incorrect submission. The team with the shortest total time is ranked highest.

The Impact of Competitive Programming on Beginners

Studies such as those conducted by Moreno et al. (2018) and Bandeira et al. (2019) employed this scoring system and contest setup to engage first-year students in programming classes. Both studies found that students introduced to competitive programming in their first year demonstrated a superior understanding of programming principles compared to those who did not. These students exhibited faster problem-solving abilities, improved code efficiency, and an increased capacity to work under pressure. Additionally, these students reported higher retention of material and reduced difficulty in grasping programming concepts.

However, not all studies concluded that CP led to improved performance. Coore and Fokum (2019), facing a lack of teaching assistants and quality feedback in first-year programming courses, employed a system of weekly competitive programming competitions to reinforce the week’s material. Their study found that while using competitive programming in assessments did increase student engagement and interest, it did not enhance the overall performance of the first-year students.

The Challenges

While CP introduces students to the rigours and excitement of coding under constraints, it’s important to recognize that CP cannot address every aspect of introductory programming. Also, certain facets of CP, such as its pace and competitive element, may only suit some learners.

Astrachan (2004) has pointed out that competitive programming only allows students to delve into key areas such as Object-Oriented Programming (OOP) design principles and enhancing code quality. CP emphasizes speed and efficiency, often overlooking the importance of well-structured, maintainable code, a crucial aspect in real-world development.

While competitive programming can inject a sense of competition into the classroom, it’s important to remember that it’s not a one-size-fits-all solution. The competitive aspect of CP may be intimidating for some students, leading to heightened anxiety and stress. This could, in turn, hinder learning and deter participation. Moreover, the pace of competitive programming, which requires swift comprehension of problem statements and speedy code implementation, may only cater to some learning styles. Some students may require more time to thoroughly grasp concepts and develop robust solutions, which could make the fast-paced environment of CP feel overwhelming.

Given these characteristics of CP, it’s clear that it should not be used as the sole determinant in course assessments. Relying too heavily on CP for grading could inadvertently favour students who possess abilities unrelated to computer science, such as high reading speed and fast typing. These intangibles can be advantageous in a competitive programming environment but have little relevance to a student’s understanding of computer science principles or their potential as a programmer.

Future of Competitive Programming in Classrooms

Although much research has been done involving introducing competitive programming into the classroom, little work explores the impact of cultural relevance in problem-setting, the role of artificial intelligence (AI) in integrating CP, and how CP interacts with various cultural and social intersections in the academic sphere.

The classroom is often characterized by a variety of cultural and social intersections. Incorporating CP in such a setting prompts us to consider how it might affect the likeability, acceptability, and academic performance across these intersections. Is CP equally appealing and accessible to students of different cultures, genders, or social backgrounds? How might the competitive nature of CP impact the dynamics of these intersections? Delving into these questions would allow us to devise strategies to ensure a more equitable and inclusive learning environment.

A unique feature of competitive programming is its creative liberty in problem-setting. This opens the possibility of integrating culturally relevant problems. Introducing programming problems referencing students’ home countries or cultures could make the learning experience more relatable and be a powerful tool to increase engagement among international students. However, the impact of such an approach is yet to be fully understood. How might culturally sensitive problems influence students’ interest and engagement? Could they enhance learning outcomes, or could they unintentionally alienate students who do not share the same cultural background?

Artificial Intelligence offers exciting possibilities in CP. For instance, large language models such as ChatGPT can assist in problem setting, which is typically a significant demand on an instructor’s time. AI-based tools could also serve as programming partners for first-year students, providing personalized assistance such as debugging help or hints for specific problems during a contest. This could supplement the responses from auto-grading judges, which is currently limited to categorized feedback that can sometimes be vague. This approach increases access to individualized learning support and mitigates common challenges associated with competitive programming, such as anxiety and intimidation. However, areas that require further exploration include the effectiveness of such tools and the best strategies for integrating them into the learning experience.

References

Moreno, J., & Pineda, A. F. (2018). Competitive programming and gamification as strategy to engage students in computer science courses. Revista ESPACIOS, 39(35).

Bandeira, I. N., Machado, T. V., Dullens, V. F., & Canedo, E. D. (2019, October 1). Competitive programming: A teaching methodology analysis applied to first-year programming classes. IEEE Xplore. https://doi.org/10.1109/FIE43999.2019.9028518

Astrachan, O. (2004). Non-competitive programming contest problems as the basis for just-in-time teaching. https://doi.org/10.1109/fie.2004.1408553

Coore, D., & Fokum, D. (2019). Facilitating Course Assessment with a Competitive Programming Platform. Proceedings of the 50th ACM Technical Symposium on Computer Science Education. https://doi.org/10.1145/3287324.3287511

January 22, 2023January 22, 2023

Are Limitations to Screen Time Necessary?

According to the American Academy of Pediatrics (2013), it is recommended that parents limit children and teens’ entertainment screen time to no more than two hours daily, stating that an increase in screen time has been linked with eye problems, violence, cyberbullying, obesity, lack of sleep, and academic decline. They quickly note that this is not a significant cause of these problems, and this information should be balanced with educating your kids about these factors. Despite this, I find this to be a gross generalization. Studies have shown that culture and class affect the amount and the type of technology children are exposed to (some are good, and some are not). Benefit from technology is greatly affected by family context (Konca, 2021).

Fraser Health, the leading health authority in B.C. Canada makes a similar recommendation of 2 hours a day (Screen Time for Children, n.d.), stating that parents should instead “Choose activities such as playing outdoors, reading or crafting over screens.” However, e-readers and tablets offer the storage of thousands of books, and web-enabled devices can give extra information on books at your fingertips. Or is reading not considered entertainment? What if I enjoy reading? Also, who does crafts without the use of a tablet? Do you have craft ideas off the top of your head?

Personally, I do not limit screen times for my six-year old son, Jacob. There are days that he spends seven hours in front of a screen, and there are days that he spends less than an hour. The important thing is that he is learning something from the experience. Screen time can be used to develop digital, creative, problem-solving, communication, social, and goal-setting skills (Using Screen Time and Digital Technology for Learning: Children and Pre-Teens, n.d.). Currently, Jacob primarily plays two games: Geometry Dash and Mario Maker. On the surface, both games provide no educational value. The former even contains many elements, such as photosensitivity and loud music, that have been known to cause seizures in players (Millichap, 1994). I have found that these games provide benefits from every category listed above, teaching children to collaborate on building levels and giving feedback to peers. Besides learning online etiquette, reading, and typing skills, Jacob has gained extensive practical knowledge about game mechanics like angle rotations, alpha transparencies, z-index, collisions, conditional structures, and counters. Some of these concepts I am teaching to my university students.

So to all health authorities: I agree with your other recommendations. Please consider deleting or altering the gross generalization of time limitations (or at least clarify it further). Otherwise, each time Jacob plays a game, I will need to set a timer on my phone, which will cut into my two hours of screen time.

References

Konca, A. S. (2021). Digital Technology Usage of Young Children: Screen Time and Families. Early Childhood Education Journal. https://doi.org/10.1007/s10643-021-01245-7

American Academy of Pediatrics. (2013). Children, Adolescents, and the Media. PEDIATRICS, 132(5), 958–961. https://doi.org/10.1542/peds.2013-2656

Screen time for children. (n.d.). Fraser Health. Retrieved January 22, 2023, from https://www.fraserhealth.ca/health-topics-a-to-z/children-and-youth/physical-activity-for-children/screen-time-for-children#.Y8zGYezMJqs

Using screen time and digital technology for learning: children and pre-teens. (n.d.). Raising Children Network. Retrieved January 22, 2023, from https://raisingchildren.net.au/school-age/school-learning/learning-ideas/screen-time-helps-children-learn#:~:text=Screen%20time%20can%20help%20children

Millichap, J. G. (1994). Video Game-Induced Seizures. Pediatric NeurologyBriefs, 8(9), 68. https://doi.org/10.15844/pedneurbriefs-8-9-5

January 21, 2023January 25, 2023

Feedback in Gamification App

Motivation

Gamification and Game-based learning are well-studied fields in education, the effects of which generally positively impact learning. From ISTE standard 1.1.c “Students use technology to seek feedback that informs and improves their practice and to demonstrate their learning in a variety of ways.” (Brooks-Young, 2017), the question that I set out to investigate is: How effective is gamification in a learning tool in terms of feedback to a student? The conclusions in this post will help improve a programming practice web application currently in development. The current iteration of the application has a series of programming challenges that a student can work through. The student can submit a program to any programming challenge and immediately receive a ‘pass’ or ‘fail’ on the problem. I aim to determine what game components and digital feedback techniques should be added to improve this online tool. The long-term use and integration of such a tool into a classroom setting to provide teachers a tools for providing practice problems for students. Under the TPACK model, the pedagogical view would be a game-based learning tool that allows students to level up as they gain more experience, the technology would be the programming platform on the web application, while the content knowledge would be gained by the programming itself.

Gamification, Game-based Learning, and Serious Game

In Teaching in a Digital Age, the author distinguishes between the following terms. Serious games are game-like interactive systems and “are referred to as entertaining tools with a purpose of education, where players cultivate their knowledge and practice their skills through overcoming numerous hindrances …” (Zhonggen, 2019, cited in Bates, 2022). Game-based learning is “the pedagogical approach of utilizing games in education.” (Anastasiadis, Lampropoulos and Siakas, 2018, cited in Bates, 2022). Gamification is the “use of game design elements in non-game contexts” (Deterding et al., 2011, cited in Bates, 2022). Game design elements for gamification include player profiles, experience points (XP), levels, badges, leaderboards, team collaboration, quests, social graphs, and certificates.

According to these definitions, my tactic is the gamification of an existing learning tool using a game-based learning pedagogical approach. I would not call the current web app a serious game since it does not possess game-like characteristics. It is widely accepted that gamification increases the app’s usability, so we seek to find the types of feedback that would prove most beneficial for the programming app.

Feedback

There are primarily two forms of feedback: explanatory and corrective. Both types of feedback inform the player of the correctness of their answer, but explanatory feedback provides the player with an additional principle-based explanation. Currently, my application provides only corrective feedback. A 1993 study using a digital game designed to teach environmental principles found “… that a version incorporating explanatory feedback significantly improved recall performances compared with a version where no feedback was available”. (Leutner, 1993, as cited in Erhel et al., 2013). Another study “… of the Design-A-Plant game showed that the use of explanatory feedback improved learners’ transfer performance more than corrective feedback did.” (Moreno and Mayer, 2004, cited in Erhel et al., 2013).

It is also essential to distinguish between generic and tailored feedback and negative and positive feedback. Generic feedback is a general statement, whereas tailored feedback is specific to the player. Negative feedback focuses on improvements that can be made, whereas positive feedback focuses on what the player did well during the last session. It is found that negative feedback is most effective when the player immediately has another turn, and positive feedback is more effective when the turn is delayed or non-existent (Burgers et al., 2015, cited in Welbers et al., 2019). Another study found that generic feedback was more effective than tailored feedback. This contradicts conventional thinking, and the authors quickly noted that it was due to corrective feedback. It is conjectured that tailored explanatory feedback, especially if accompanied by hints tailored to the player, could encourage participation (Welbers et al., 2019).

One theme prevalent in these studies is the idea of instant feedback within games. Azumi (2015) highlights the addictiveness of massive multiplayer online role-playing games (MMORPG). These games require players to work together to accomplish a goal. Including a gamified collaboration element benefits programming education students in the learning process as they attain immediate feedback from peers (Azumi, 2015). In fact, “feedback in serious games and game-based learning can also take the form of responses from dialogues with non-player characters or instant feedback messages related to game controls and challenges performed.” (Laine & Lindberg, 2020, cited in Krath, 2021). Intuitively, immediate feedback allows players to make instant adjustments to quickly correct failures and set new goals.

Other Lessons Learned

Although unrelated to feedback, it is interesting to note that gamification applications in Computer Science have mixed results. A gamification platform that contained game dynamics was used to teach a course in Software Engineering where the students did not feel interested in the gamified environment. It is conjectured that older students and those who have much experience with lecture-based learning is less receptive of game-base learning (Berkling and Thomas 2013, as cited by Nah). On the other hand, a course in Information Systems and Computer Engineering introduced multiple game design elements into the course design itself. This increased student engagement and lecture attendance but did not significantly improve student grades (Barata et al., as cited by Nah). We must ensure that online tools do not replace the programming learning experience which only comes from solving real-world problems.

Conclusions:

From the readings and conclusions above, the following actions should be taken to incorporate gamification elements in the programming application.

Since a student can debug, alter, and resubmit an incorrect program immediately, the application should give immediate negative feedback, displaying elements of the code or test cases that failed in the execution of a program.
Lessons should be drawn from the MMORPG. We should add a collaborative element, allowing students to form teams to solve problems. Students can gain peer feedback and also benefit from developing a personal learning community.
Welbers et al. (2019) noted that tailored explanatory feedback might encourage player engagement and participation. One way to do this is to give feedback by sending weekly progress reports and questions tailored to the user’s skill level.
Programming challenges beyond a student’s skill level should be hidden until the required experience points are attained. This is a form of positive feedback which is most beneficial when the game play is delayed.

References

Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Computers & Education, 67, 156–167. https://doi.org/10.1016/j.compedu.2013.02.019

Welbers, K., Konijn, E. A., Burgers, C., de Vaate, A. B., Eden, A., & Brugman, B. C. (2019). Gamification as a tool for engaging student learning: A field experiment with a gamified app. E-Learning and Digital Media, 16(2), 92–109. https://doi.org/10.1177/2042753018818342

Nah F.F., & Zeng, Q., & Telaprolu, V. R., & Ayyappa, A. P., & Eschenbrenner, B., (2014). Gamification of education: A review of literature. HCI in Business, 401-409. https://doi.org/10.1007/978-3-319-07293-7_39

Azumi, S., Iahad, N., & Ahmad, N. (2015). Gamification in online collaborative learning for programming courses: A literature review. ARPN Journal of Engineering and Applied Sciences, 10(23).

Krath, J., Schürmann, L., & von Korflesch, H. F. O. (2021). Revealing the theoretical basis of gamification: A systematic review and analysis of theory in research on gamification, serious games and game-based learning. Computers in Human Behavior, 125, 106963. https://doi.org/10.1016/j.chb.2021.106963

Bates, A. W. (Tony). (2022). Chapter 8: Pedagogical differences between media. Pressbooks.bccampus.ca. https://pressbooks.bccampus.ca/teachinginadigitalagev3m/part/chapter-8-pedagogical-differences-between-media/

Brooks-Young, S. (2017). ISTE standards for students: a practical guide for learning with technology. International Society For Technology In Education.