Category: Data-Driven Decision-Maker

Coaches use qualitative and quantitative data to guide their teaching and professional learning by:
– assisting educators in collecting and analyzing data
– helping to interpret results to inform decisions
– developing heuristics to measure their progress

Posted on

Measuring Student Contribution in a Software Engineering Team

Introduction

In software engineering, there is very little consensus on how to measure an individual developer’s contribution. Although many measures have been proposed, their usefulness in the industry lacks validation, particularly from the perspectives of team leaders and managers (Lima et al., 2015). The lack of measurement also challenges educators (Gardner et al., 2003). This post will examine student developer contributions within the context of a software engineering project.

ISTE Standard 4.6 advocates for ed tech coaches to be data-driven decision-makers using qualitative and quantitative data to inform their decisions. Standard 4.6b states, “Support educators to interpret qualitative and quantitative data to inform their decisions and support individual student learning.” Techniques discussed in this article could be used to measure student engagement and fulfillment in a team project and give insight into where instruction can be altered in a software engineering course.

I will begin by examining the use of chat platforms like Discord to track individual student contributions. Next, I’ll discuss the role of peer evaluations in assessing team member input. Lastly, I’ll introduce repository mining techniques to quantify these contributions.

Live chat Activity

We’ll start with what I consider the least effective among the three metrics. In recent years, many modern developers have adopted Discord as a tool for real-time communication and collaboration in software engineering projects. Fundamentally, Discord channels serve as dedicated spaces for text, voice, and video communication. In educational contexts, these channels can be structured to reflect the various teams within a software project, facilitating organized, topic-specific discussions. Such channels can host various activities, from casual interactions and planning sessions to problem-solving discussions and code reviews, closely mirroring a real-world software development environment. Furthermore, Discord captures all these interactions, creating a comprehensive, searchable archive of every conversation and exchange.

Moreover, thanks to its bot-integration features, Discord is increasingly seen as an innovative tool for gauging student contributions in team-based projects. Analytical bots like Statbot offer detailed statistics on individual interactions on the platform, enabling the assessment of each student’s engagement. Chat histories also supply quantitative data on the quality of contributions in software engineering team projects.

However, while bots offer valuable quantitative and analytical insights, it’s important to complement this data with qualitative evaluations. Direct observations, feedback sessions, and individual discussions remain indispensable for grasping the subtleties of each student’s input. It’s also vital to address privacy concerns and uphold ethical standards in monitoring, ensuring clear guidelines and transparency from the instructor’s side.

Peer Evaluations

Gardner et al. (2003) conducted a study exploring the use of group member ratings to gauge relative contributions among students in a software engineering team project course. At the end of the project, students rate each team member’s contributions across four criteria using a five-point scale:

  • Attendance at team meetings.
  • Volunteering for and carrying out tasks.
  • Quality of work performed.
  • Effectiveness in communicating ideas.

The findings suggest that these anonymous peer ratings are reliable for ranking team members on their contributions. While students often rate themselves higher than their teammates, the relative contributions ranking remains consistent, which aligns with previous research (West, 2018, Ch.16).

This approach quantifies peer perceptions of engagement and effort. It motivates students to interact and collaborate and allows teams to self-manage contributions. However, limitations exist. Students may not accurately judge true contributions. Dominant personalities could influence ratings. Moreover, if grades hinge directly on these ratings, it might encourage score inflation.

Despite its limitations, peer ratings offer a systematic method to encourage and gauge participation in team projects. They represent the firsthand insights of teammates into individual efforts and team dynamics. Instructors should triangulate peer evaluations with other performance indicators to mitigate potential biases. When applied thoughtfully, group member ratings can be a scalable tool to enhance accountability and ensure equitable effort distribution within student engineering teams.

Using Git Repositories

While subjective peer evaluations are commonly used, analyzing data from git repositories provides an objective lens into individual contributions, revealing insights into aspects like collaboration patterns, subsystem ownership, and consistency of participation (Lima et al., 2015). Instructors can combine these repository-based metrics with subjective evaluations to assess student effort and engagement better.

A fundamental metric is examining each student’s number of commits over time, called code contribution (Lima et al., 2015). This helps reveal whether students contribute regularly throughout the project or make concentrated commits right before deadlines. Students with relatively few commits thinly spread across the weeks likely contributed minimally, while a student with a steady stream of commits each week demonstrates consistent engagement (Glassy, 2006).

Examining the content of commits also provides insights into contribution quality. The code complexity measure is also widely accepted as a good measure of contribution. The code complexity measure considers the complexity and difficulty of the sub-problem being solved. Complexity measures were proposed by McCabe in 1976 and are still widely used today to examine git repositories. The measures analyze code complexity before and after a team member has altered it. Low commit complexity suggests weaker contributions to the team’s software development processes.

A variation of the code complexity measure is the bug-related measures, which measure the contribution to bug introductions and bug-fixing. However, this measure has limitations because some bug fixes do not require writing code, mitigating the developer’s efforts (Lima et al., 2015). Also, advanced repository analysis can reveal collaboration patterns within student teams. Tools like FRASR and ProM introduced by Poncin et al. (2011) can extract event logs from student repository data (using FRASR) and subsequently analyze the development process (with ProM). This tool also incorporates developer roles and adherence to certain development models. 

Of course, reliance solely on git metrics has limitations. First, commits mainly represent coding contributions, overlooking other forms of participation like verbal collaboration and project leadership (Lima et al., 2015). Second, students can artificially inflate their repository activity metrics if they know the algorithm being used. Despite these drawbacks, analyzing git data provides valuable insights into individual participation on student software teams. Instructors should interpret repository metrics not as absolute contribution measures but as launching points for further investigation.

Conclusion

By balancing quantitative git data with qualitative peer evaluations, product assessments, and student interviews, instructors can obtain a more equitable evaluation of individuals. Nonetheless, there is a strong correlation between subject and objective measures of contribution to a project (Hundhausen et al., 2022). Software engineering courses require team projects, but assessing individual accountability remains vital. Combining subjective reviews and objective repository analysis helps reveal a more accurate picture of each student’s contributions and commitment.

References

Lima, J., Christoph Treude, Fernando Figueira Filho, & Kulesza, U. (2015). Assessing developer contribution with repository mining-based metrics. https://doi.org/10.1109/icsm.2015.7332509

Gardner, W. (2003). Assessing individual contributions to group software projects. In 8th Western Canadian Conference on Computing Education (WCCCE’03) (pp. 33-50).

Hundhausen, C. D., Conrad, P. T., Carter, A. S., & Adesope, O. (2022). Assessing individual contributions to software engineering projects: a replication study. Computer Science Education, 32(3), 335–354. https://doi.org/10.1080/08993408.2022.2071543

West, R. E. (2018). Foundations of Learning and Instructional Design Technology. https://doi.org/10.59668/3

Glassy, L. (2006). Using version control to observe student software development processes. Journal of Computing Sciences in Colleges, 21(3), 99–106.

McCabe, T. J. (1976). A Complexity Measure. IEEE Transactions on Software Engineering, SE-2(4), 308–320. https://doi.org/10.1109/tse.1976.233837

Poncin, W., Serebrenik, A., & Mark. (2011). Mining student capstone projects with FRASR and ProM. https://doi.org/10.1145/2048147.2048181

Posted on

Computer Science Curriculum Integration for K-9 Teachers

Introduction

With a rapidly advancing world and integrating more technology into life, it has increasingly become evident to parents, teachers, and students alike that technological literacy is essential to primary education. Parents are pushing for increased computer science instruction in elementary schools as they realize how vital this knowledge will be in preparing their children for success beyond grade school.  According to code.org (Computer Science Education Stats, n.d.), 90% of parents want their children to study computer science, but only 53% of high schools offer it in their curriculum. Research has also shown that students who take computer science courses in high school are 17% more likely to pursue higher education, with even higher percentages observed among traditionally marginalized populations such as females and Black and Latino students (Brown & Brown, n.d.).

While computer science courses provide fundamental skills like coding and web design, their significance extends far beyond that. Understanding technology equips students with data analysis capabilities and problem-solving skills applicable across various fields, extending beyond digital work environments. Consequently, many believe it is crucial to prioritize teaching computer science concepts. This post outlines my efforts to understand the computer science curriculum’s current needs and promote computer science education among elementary school teachers in British Columbia.

Assessing Current Needs

According to Dr. Shannon Thissen, Regional Administrator for Educational Technology and Computer Science Teaching and Learning in Capital Region ESD 113 in Washington, integrating computer science concepts into the curriculum poses significant challenges. Dr. Thissen mentions that many untrained teachers express concerns and seek guidance. The primary obstacle is teachers’ fear of the unknown, and overcoming this fear is crucial. Several programs, including those offered by code.org, provide free resources for teachers. However, the participation of schools in these programs in British Columbia remains limited. Dr. Thissen also highlights that teachers may hesitate to incorporate such programs into their curriculum if they are not mandated.

In a previous post, I discussed the inconsistency of computer science programs in grade schools across British Columbia, particularly emphasizing the needs of interior and rural areas. While I faced difficulties connecting with schools that desired my services, I recently connected with a Programming 11/12 teacher in Kamloops. I had the opportunity to speak to her class, providing valuable insights into the needs of both teachers and students. Students showed great interest in game development and 3D animation. Despite being a beginner-level class, students had diverse backgrounds in the subject, with some having significant experience with Unity while others struggled with the early stages of block programming. However, their shared enthusiasm for the subject validated studies showing that a majority of students enjoy computer science (Computer Science Education Stats, n.d.). This highlights the need for increased promotion of computer science in the earlier grades to nurture this enjoyment.

Currently, I am collaborating with John Knox Christian School, which is revamping its computer science and technology curriculum. Working with the Director of Curriculum and Computer Science, we are developing a series of workshops to assist K-9 teachers in integrating computer science into their classrooms. This collaboration is an excellent opportunity as John Knox controls its curriculum from K-12, enabling longer-term assessment of student success through the workshops with the same group of students.

Project Design

The central portion of the project is divided into four phases: Assessing Needs, Planning and Foundations, Integration, and Reflection.

Assessing Needs:

To ensure alignment with the school’s desired outcomes and address the specific needs of K-9 teachers, a survey will be created and distributed to all K-9 teachers. The survey aims to gather information on teachers’ objectives, knowledge levels, and areas of interest in computer science. The survey includes choices for dedicated topics such as programming, computational thinking, artificial intelligence, game development, and computer hardware. The project will tailor its objectives and content to meet the needs and interests of the teachers. The survey will be sent out during the first week of summer break.

Planning and Foundations:

This phase involves conducting a workshop covering computer science fundamentals, including vocabulary, problem-solving, and demystifying computer science. The workshop will also focus on integrating hands-on activities for teachers to experience and practice computer science concepts firsthand. Teachers will be guided in designing activities that promote problem-solving, collaboration, and critical thinking skills. Additionally, resources such as coding platforms, educational apps, and lesson plans will be provided to support teachers in implementing computer science activities beyond the workshop. By the end of the workshop, teachers should have a plan to incorporate computer science concepts into an existing lesson. This workshop is scheduled for the first professional development day in September 2023.

Integration:

Teachers will execute their plans to integrate computer science concepts into the classroom during the integration stage. One strategy for this stage is to have coaches co-teach a computer science lesson alongside the classroom teacher. Initially, the director and I will act as coaches, but the plan is to train coaches for future iterations. Co-teaching allows the teacher to observe and learn from the coach’s expertise and experience, helping teachers gain confidence and deepen their understanding. The coach can help guide students through hands-on activities and will provide positive reinforcement, celebrate teachers’ achievements, and acknowledge their efforts in integrating computer science concepts. In addition to co-teaching, coaches will observe regularly and provide teachers with constructive feedback. Classroom observations, either in person or through video recordings, will be conducted to assess the implementation of computer science lessons and the effectiveness of teaching strategies. Coaches will then provide feedback, highlighting strengths and offering suggestions for improvement. This feedback loop will enable teachers to reflect on their practice, refine their instructional techniques, and make continuous progress in integrating computer science effectively.

Reflection:

Integration is ongoing, and teachers require continued support beyond the initial stages. Follow-up meetings, workshops, or online forums will be organized to facilitate knowledge sharing, questions, and further guidance. These support mechanisms will help sustain the momentum and provide teachers with ongoing professional development and collaboration opportunities. Coaches will curate and share relevant resources, including lesson plans, coding activities, and best practices, to further support teachers’ integration efforts. This stage is scheduled for the third professional development day in November 2023.

Analyzing the Design

Using Adria Steinberg’s (1998) six A’s of evaluating project design, my proposed project aligns with these principles, ensuring the successful integration of CS into K-9 classrooms.

Academic Rigor: The project emphasizes challenging and intellectually stimulating CS content. It provides opportunities for students to engage in practical problem-solving, critical thinking, and analytical reasoning. The project also encourages teachers to adopt a different mindset when teaching CS.

Authenticity: The project provides real-world contexts and experiences that connect CS concepts to students’ lives and interests. It includes relating CS to authentic scenarios, which enhances student motivation and engagement in other subjects. By establishing these connections, the project makes CS more relevant and meaningful to students.

Applied Learning: The project emphasizes the hands-on application of CS knowledge and skills. It encourages teachers to apply CS-related skills to different activities in the classroom. This approach allows students to experiment, problem-solve, and apply CS principles in real-world contexts.

Active Exploration: The project encourages students to explore CS concepts through inquiry-based and self-reflective learning. It creates an environment that nurtures teachers’ and students’ curiosity, exploration, and independent thinking; this fosters a sense of ownership in their CS learning journeys.

Adult Relationships: The project recognizes the importance of fostering supportive and meaningful connections between students and adult mentors or educators. By incorporating coaches or mentors into the framework, the project provides students (and teachers) opportunities to interact with CS professionals, experts, or mentors. These interactions offer guidance, share real-world experiences, and serve as role models for students.

Assessment: The project includes an assessment plan to evaluate student learning and program effectiveness. It incorporates post-integration surveys and other assessment strategies to measure the impact of CS integration on student learning outcomes. Gathering feedback from stakeholders informs program improvements and ensures ongoing effectiveness.

Next Steps:

To effectively integrate computer science into K-9 classrooms, conducting one or two workshop iterations is needed. This allows for refinement and improvement of the workshop content and delivery based on feedback and insights from the initial sessions. During each iteration, it is important to collect feedback from participating teachers. Assessing the teachers’ comfort levels with integrating computer science concepts and their perceived efficacy in implementing the learned strategies in their classrooms will help gauge the workshop’s effectiveness. Additionally, student learning outcomes can be evaluated through pre- and post-workshop assessments to measure students’ knowledge growth, skills development, and attitude toward computer science. Comparing these assessments will provide evidence of the workshop’s impact on student learning.

Once the initial iterations of the workshop have been conducted and evaluated, the next step is to engage rural area schools with a more concrete plan. Building on the lessons learned and feedback from the initial workshops, it is essential to develop a targeted approach specifically tailored to the needs and challenges of rural schools. Further changes to the workshop may include solutions to overcome unique barriers such as limited resources, infrastructure challenges, or teacher training opportunities. Designing the workshop to accommodate these conditions will increase its relevance and effectiveness in rural settings.

If the workshop successfully enhances teacher comfort levels and promotes student learning, developing a Professional Development Program (PDP) at the university is an excellent opportunity. A PDP would offer a more structured and comprehensive training program for teachers seeking to integrate computer science into their K-9 classrooms. The Education Department can collaborate with the School of Computing Science and instructional design experts to design a robust PDP curriculum. This curriculum should address the specific needs of teachers, providing them with theoretical knowledge, practical skills, and ongoing support to integrate computer science concepts into their classrooms effectively.

References

Computer Science Education Stats. (n.d.). Code.org. https://code.org/promote/stats

Brown, E., & Brown, R. (n.d.). The Effect of Advanced Placement Computer Science Course Taking on College Enrollment. http://www.westcoastanalytics.com/uploads/6/9/6/7/69675515/longitudinal_study_-_combined_report_final_3_10_20__jgq_.pdf

CS Journeys Resources: Mentorship and community. (n.d.). Code.org. Retrieved June 3, 2023, from https://code.org/beyond/mentors-and-community

Steinberg, A. (1998). Real learning, real work : school-to-work as high school reform. Routledge.

K–12 Computer Science Framework. (n.d.). K12cs.org. https://k12cs.org/

Posted on

Crap Detection for Node Package Manager

Open-source software allows the source code to be freely available to the public, allowing anyone to inspect, modify, use, or distribute the code. It is typically developed and maintained by a community of volunteer developers who collaborate and contribute to the project. According to the Linux Foundation, open-source code is used in 70-90% of every software found today (A Summary of Census II: Open Source Software Application Libraries the World Depends on – Linux Foundation, n.d.). Most of this code is a result of open-source development frameworks.

Open-source development frameworks are libraries of code and tools that provide a foundation for building software applications efficiently by supplying reusable components. Using open-source development frameworks can help streamline the software development process and reduce the time and effort required to build applications from scratch. Frameworks can be used for creating a wide range of software, including web, mobile, and desktop applications.

Development frameworks typically rely on package managers.  A package manager is a tool that helps manage the libraries, dependencies, and other components needed for a project. Using a package manager, developers can automate installing, updating, and tracking libraries needed for their projects. Many package managers employ a team of reviewers who review every request to push new updates to the repository. However, some frameworks may include public repositories in which anyone can upload their packages for the community. The Node Package Manager (NPM), which runs on the JavaScript open-source framework NodeJS, is one of the most popular public repositories and thus makes it a target for harmful activity. 

Types of Malicious Code

Several types of malicious code can exist on NPM, including:

  1. The most obvious is that some packages themselves may contain malicious code, such as backdoors, trojans, or malware, that can harm the software or compromise the security of the system it is installed on. In other cases, malicious packages may simply lie about what the package does, instead running damaging commands such as modifying system files, altering settings, or collecting and transmitting sensitive data without the user’s consent. For example, a data-stealing package was uncovered in late 2022 after going undetected for over a year (NPM Malware Attack Goes Unnoticed for a Year | TechTarget, n.d.). There has also been an increase in crypto mining malware on NPM in 2022. These malware packages contain code that mines cryptocurrencies on the computer of the person who installs the package, using their system resources without the user’s knowledge.
  2. Supply Chain Attacks. Packages on NPM may have dependencies on other packages that contain vulnerabilities, which can put the developed software at risk if the vulnerabilities are not detected somewhere in the chain. (Ellison et al., 2010)
  3. Malicious code could be hidden in packages that have suspiciously similar names to popular and frequently used packages. These are known as typo-squatting and combo-squatting attacks (Vu et al., 2020). For example, in 2017, a popular package called ‘cross-env’ was typo-squatted and malicious code hosted on a package called ‘crossenv’ preyed on developers who accidentally misspelled the package name. (Npm Blog: ‘Crossenv’ Malware on the Npm Registry, n.d.)
  4. Stealing credentials of legitimate NPM accounts and pushing malicious code onto its repository (Vu et al., 2020)

Information Literacy on NPM

The term crap detection is used describe a process of deciding whether something found online is true. To help analyze packages for possible malicious code, several studies have been done on lightweight NPM vulnerability auto-detection tools. They are very effective. In most cases, with over 90% success rate (Sejfia et al., 2022) and able to identify exact types of vulnerability (according to OWASP) with a high degree of accuracy (Brito et al., 2023). They are, however, not perfect, and we, as developers, must do our part to avoid falling into the pit of malicious intent. Here is a summary of recommendations from the studies. I am basing this on the C.R.A.A.P. and S.I.F.T. tests for information literacy, I will call this the F.I.S.T. test for NPM literacy:

  • Package Frequency of download and package updates: The NPM website contains the weekly downloads of every package on the repository. The Frequency of maintenance of a package also gives information about the safety of use, check the last publish date to see how active this project is. Projects that are not actively maintained or have low weekly downloads are a cause for concern.
  • Inspect the code: Before installing a package, review it to ensure it doesn’t contain any malicious code. Go to the package’s Github account and quickly scan the code for malicious behaviour. Look for things like email dependencies, non-consensual sending of HTTP requests, etc. This may be difficult for beginners of a framework.
  • Use Secure sources: Only install packages from trusted sources like NPM or verified package managers like yarn. From the NPM website, verify information about the package’s publisher, such as the publisher’s website and social media accounts. If they have very little information online, it may suggest they are trying to hide something.
  • Use auditing Tools: Use security tools like npm audit, which scans installed packages for known vulnerabilities, to help you identify and mitigate potential security risks. npm audit will look at existing packages in your system, but since you can only run this command after the package has been installed, it may already be too late. Alternatively, there are also many NPM rating tools. Spectralops.io lists the Top 5 NPM audit tools in 2022. Among them are Synk and SpectralOps. Both tools provide reviews and ratings of every NPM package. Search the packages you intend to use to check its rating and to ensure they are not on the known vulnerabilities list. A package with low ratings or negative reviews may indicate potential security issues.

NPM Improvements

Other solutions are suggested to improve the existing architecture and workflow of the package managers themselves. This includes:

  • Updating NPM policy to allow automatic updating to the latest version of a package to mitigate technical lag (the time it takes a developer to recognize a malicious package and update it)
  • securing critical accounts on NPM,
  • providing trust scores for developers, and
  • eviction of typo- and combo-squatters.

(Kaplan et el. 2021)

Conclusion

The Node Package Manager (NPM) is a powerful tool that helps software developers manage and share the packages they use in their projects. It provides an array of community packages which can be added to the software while enabling developers to write and share code with the community. However, with its growing popularity and widespread use, knowing how to perform crap detection on NPM effectively is a valuable asset for any modern developer looking to improve their information literacy skills.

References

A Summary of Census II: Open Source Software Application Libraries the World Depends On – Linux Foundation. (n.d.). Www.linuxfoundation.org. Retrieved February 5, 2023, from https://www.linuxfoundation.org/blog/blog/a-summary-of-census-ii-open-source-software-application-libraries-the-world-depends-on#:~:text=Introduction

NPM Blog: ‘crossenv’ malware on the npm registry. (n.d.). Blog.npmjs.org. Retrieved February 4, 2023, from https://blog.npmjs.org/post/163723642530/crossenv-malware-on-the-npm-registry.html

Vu, D.L., Pashchenko, I., Massacci, F., Plate, H., Sabetta, A. (2020). Typosquatting and combosquatting attacks on the python ecosystem. 2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 509–514.  

Ellison, R. J., Goodenough, J. B., Weinstock, C. B., & Woody, C. (2010). Evaluating and mitigating software supply chain security risks. Carnegie-Mellon Univ Pittsburgh PA Software Engineering Inst.

Kaplan, B., & Qian, J. (2021). A survey on common threats in npm and pypi registries. Deployable Machine Learning for Security Defense: Second International Workshop, MLHat 2021, Virtual Event, August 15, 2021, Proceedings 2 (pp. 132-156). Springer International Publishing.

NPM malware attack goes unnoticed for a year | TechTarget. (n.d.). Security. Retrieved February 4, 2023, from https://www.techtarget.com/searchsecurity/news/252525968/NPM-malware-attack-goes-unnoticed-for-a-year#:~:text=The%20researchers%20said%20that%20by

Brito, T., Ferreira, M., Monteiro, M., Lopes, P., Barros, M., Santos, J. F., & Santos, N. (2023). Study of JavaScript Static Analysis Tools for Vulnerability Detection in Node.js Packages. ArXiv. https://arxiv.org/abs/2301.05097

Sejfia, A., & Schäfer, M. (2022). Practical Automated Detection of Malicious npm Packages. ArXiv. https://doi.org/10.1145/3510003.3510104