Tag: software engineering

Posted on

Measuring Student Contribution in a Software Engineering Team

Introduction

In software engineering, there is very little consensus on how to measure an individual developer’s contribution. Although many measures have been proposed, their usefulness in the industry lacks validation, particularly from the perspectives of team leaders and managers (Lima et al., 2015). The lack of measurement also challenges educators (Gardner et al., 2003). This post will examine student developer contributions within the context of a software engineering project.

ISTE Standard 4.6 advocates for ed tech coaches to be data-driven decision-makers using qualitative and quantitative data to inform their decisions. Standard 4.6b states, “Support educators to interpret qualitative and quantitative data to inform their decisions and support individual student learning.” Techniques discussed in this article could be used to measure student engagement and fulfillment in a team project and give insight into where instruction can be altered in a software engineering course.

I will begin by examining the use of chat platforms like Discord to track individual student contributions. Next, I’ll discuss the role of peer evaluations in assessing team member input. Lastly, I’ll introduce repository mining techniques to quantify these contributions.

Live chat Activity

We’ll start with what I consider the least effective among the three metrics. In recent years, many modern developers have adopted Discord as a tool for real-time communication and collaboration in software engineering projects. Fundamentally, Discord channels serve as dedicated spaces for text, voice, and video communication. In educational contexts, these channels can be structured to reflect the various teams within a software project, facilitating organized, topic-specific discussions. Such channels can host various activities, from casual interactions and planning sessions to problem-solving discussions and code reviews, closely mirroring a real-world software development environment. Furthermore, Discord captures all these interactions, creating a comprehensive, searchable archive of every conversation and exchange.

Moreover, thanks to its bot-integration features, Discord is increasingly seen as an innovative tool for gauging student contributions in team-based projects. Analytical bots like Statbot offer detailed statistics on individual interactions on the platform, enabling the assessment of each student’s engagement. Chat histories also supply quantitative data on the quality of contributions in software engineering team projects.

However, while bots offer valuable quantitative and analytical insights, it’s important to complement this data with qualitative evaluations. Direct observations, feedback sessions, and individual discussions remain indispensable for grasping the subtleties of each student’s input. It’s also vital to address privacy concerns and uphold ethical standards in monitoring, ensuring clear guidelines and transparency from the instructor’s side.

Peer Evaluations

Gardner et al. (2003) conducted a study exploring the use of group member ratings to gauge relative contributions among students in a software engineering team project course. At the end of the project, students rate each team member’s contributions across four criteria using a five-point scale:

  • Attendance at team meetings.
  • Volunteering for and carrying out tasks.
  • Quality of work performed.
  • Effectiveness in communicating ideas.

The findings suggest that these anonymous peer ratings are reliable for ranking team members on their contributions. While students often rate themselves higher than their teammates, the relative contributions ranking remains consistent, which aligns with previous research (West, 2018, Ch.16).

This approach quantifies peer perceptions of engagement and effort. It motivates students to interact and collaborate and allows teams to self-manage contributions. However, limitations exist. Students may not accurately judge true contributions. Dominant personalities could influence ratings. Moreover, if grades hinge directly on these ratings, it might encourage score inflation.

Despite its limitations, peer ratings offer a systematic method to encourage and gauge participation in team projects. They represent the firsthand insights of teammates into individual efforts and team dynamics. Instructors should triangulate peer evaluations with other performance indicators to mitigate potential biases. When applied thoughtfully, group member ratings can be a scalable tool to enhance accountability and ensure equitable effort distribution within student engineering teams.

Using Git Repositories

While subjective peer evaluations are commonly used, analyzing data from git repositories provides an objective lens into individual contributions, revealing insights into aspects like collaboration patterns, subsystem ownership, and consistency of participation (Lima et al., 2015). Instructors can combine these repository-based metrics with subjective evaluations to assess student effort and engagement better.

A fundamental metric is examining each student’s number of commits over time, called code contribution (Lima et al., 2015). This helps reveal whether students contribute regularly throughout the project or make concentrated commits right before deadlines. Students with relatively few commits thinly spread across the weeks likely contributed minimally, while a student with a steady stream of commits each week demonstrates consistent engagement (Glassy, 2006).

Examining the content of commits also provides insights into contribution quality. The code complexity measure is also widely accepted as a good measure of contribution. The code complexity measure considers the complexity and difficulty of the sub-problem being solved. Complexity measures were proposed by McCabe in 1976 and are still widely used today to examine git repositories. The measures analyze code complexity before and after a team member has altered it. Low commit complexity suggests weaker contributions to the team’s software development processes.

A variation of the code complexity measure is the bug-related measures, which measure the contribution to bug introductions and bug-fixing. However, this measure has limitations because some bug fixes do not require writing code, mitigating the developer’s efforts (Lima et al., 2015). Also, advanced repository analysis can reveal collaboration patterns within student teams. Tools like FRASR and ProM introduced by Poncin et al. (2011) can extract event logs from student repository data (using FRASR) and subsequently analyze the development process (with ProM). This tool also incorporates developer roles and adherence to certain development models. 

Of course, reliance solely on git metrics has limitations. First, commits mainly represent coding contributions, overlooking other forms of participation like verbal collaboration and project leadership (Lima et al., 2015). Second, students can artificially inflate their repository activity metrics if they know the algorithm being used. Despite these drawbacks, analyzing git data provides valuable insights into individual participation on student software teams. Instructors should interpret repository metrics not as absolute contribution measures but as launching points for further investigation.

Conclusion

By balancing quantitative git data with qualitative peer evaluations, product assessments, and student interviews, instructors can obtain a more equitable evaluation of individuals. Nonetheless, there is a strong correlation between subject and objective measures of contribution to a project (Hundhausen et al., 2022). Software engineering courses require team projects, but assessing individual accountability remains vital. Combining subjective reviews and objective repository analysis helps reveal a more accurate picture of each student’s contributions and commitment.

References

Lima, J., Christoph Treude, Fernando Figueira Filho, & Kulesza, U. (2015). Assessing developer contribution with repository mining-based metrics. https://doi.org/10.1109/icsm.2015.7332509

Gardner, W. (2003). Assessing individual contributions to group software projects. In 8th Western Canadian Conference on Computing Education (WCCCE’03) (pp. 33-50).

Hundhausen, C. D., Conrad, P. T., Carter, A. S., & Adesope, O. (2022). Assessing individual contributions to software engineering projects: a replication study. Computer Science Education, 32(3), 335–354. https://doi.org/10.1080/08993408.2022.2071543

West, R. E. (2018). Foundations of Learning and Instructional Design Technology. https://doi.org/10.59668/3

Glassy, L. (2006). Using version control to observe student software development processes. Journal of Computing Sciences in Colleges, 21(3), 99–106.

McCabe, T. J. (1976). A Complexity Measure. IEEE Transactions on Software Engineering, SE-2(4), 308–320. https://doi.org/10.1109/tse.1976.233837

Poncin, W., Serebrenik, A., & Mark. (2011). Mining student capstone projects with FRASR and ProM. https://doi.org/10.1145/2048147.2048181

Posted on

Collaborating on Git!

Introduction

Background and Motivation

Git is the industry-leading version control system that streamlines software development and helps teams collaborate more efficiently. It provides a wide range of features to track changes, manage projects, store code, resolve conflicts, create branches/rollbacks and keep an eye on contributions from each team member. Its excellent performance, security, scalability and many implementations make it the standard version control system today. It also has powerful integrations within modern integrated development tools to help coordinate tasks such as code reviews merging or scheduling efficiently amongst multiple developers.

To my surprise, I discovered that when it comes to the fundamentals of Git, many second-year software engineering students are completely lost. They lack the collaboration skills needed for a successful project and don’t understand how crucial requesting proper merges is with this online tool. Instead of utilizing proper merge requests for documents through Git’s powerful online tooling capabilities, these same students are emailing their work in and waiting for manual merges from leaders, which disregards best practices and overlooks useful commands available at their disposal.

Overview

This project aims to equip second-year post-secondary computing science students with the basic knowledge and confidence necessary for working on team projects with an introduction to Git. Through specific workflows that reflect real-world situations, students will experience how software developers can use Git efficiently, effectively, and ethically when collaborating in a group setting. As most have no prior exposure upon starting their course in Software Engineering at my school – this proposed lesson provides them with an invaluable opportunity before participating in more extensive collaborations later in the course.

ISTE Standards

The exercise is developed to align with the International Society for Technology in Education standards for students.  These standards promote students taking charge of their learning by giving them a voice and encouraging the process to be led by them. This creates an empowering environment in which they can reach new levels of achievement.

ISTE 2 Digital Citizenship

By exploring the Git-suggested workflows, students will gain a valuable understanding of working and collaborating ethically in teams. We’ll ask them to consider their rights as content producers on this platform while respecting its privacy rules, by making sure they use public/private repositories appropriately and only grant necessary access when needed. This is in alignment with ISTE standard 2a. “Students cultivate and manage their digital identity and reputation and are aware of the permanence of their actions in the digital world.”, 2c. “Students demonstrate an understanding of and respect for the rights and obligations of using and sharing intellectual property.” and 2d “Students manage their personal data to maintain digital privacy and security and are aware of data-collection technology used to track their navigation online.”

ISTE 1 Digital Citizenship

By using Git, students have the ability to interact with one another and merge their efforts on a project. This allows them to share feedback within the team and quickly implement it into their development cycle for greater overall productivity. This is in line with ISTE standard 1b. “Students use technology to seek feedback that informs and improves their practice and to demonstrate their learning in a variety of ways,” and 1c, “Students understand the fundamental concepts of technology operations, demonstrate the ability to choose, use and troubleshoot current technologies and are able to transfer their knowledge to explore emerging technologies.”

ISTE 7 Digital Citizenship

Broadly stated, ISTE standard 7 states, “Students use digital tools to broaden their perspectives and enrich their learning by collaborating with others and working effectively in teams locally and globally.” This collaborative exercise will enable students to hone their skills remotely, allowing them to experience working with peers from different backgrounds globally, the exercise centers around this core standard.

Backwards Design

We utilize the backwards design principles to create a successful learning experience (Wiggins & McTighe, 2005). This process involves three distinct stages. Firstly, our goals are aligned with accepted content standards and curriculum expectations to identify desired results. Secondly, acceptable evidence is established in order to validate that these objectives have been achieved, emphasizing knowledge over activities or covered material alone. Finally, we plan instructional experiences and instruction based on what learners need to achieve their goals.

Stage 1 Identify Desired Results

Establish Goals:

There are two main departmental learning outcomes based on this exercise:

  1. (LO1a) Students will be introduced to team processes and cooperative programming and understand their pros and cons.
    • Students will understand the use and workflows of using Git as a collaboration tool.
    • Students will collaborate on a document.
  2. (LO2a) Students will understand how to communicate effectively within a team.
    • Students will decide on a workflow for conflict resolution.

Essential Questions:

The provocative questions that foster student inquiry, understanding and transfer of learning:

  • How can you resolve conflicts or disagreements with a spouse or significant other?
  • Why is it important to communicate in any relationship?
  • Why is it essential to collaborate instead of working alone in a software development setting?

Understandings:

Students will understand

  • The general workflow of Git.
  • Why it is essential to use a collaboration tool in software development.

Knowledge:

Students will know the following:

  • What a Git repository is, and what it’s used for
  • The basics of collaborative development
  • The Git conflict resolution process

Skills:

Students will be able to:

  • Fork a project
  • Push updates to branches
  • Merge branches
  • Create merge requests
  • Communicate with teammates over git

Stage 2 – Assessment Evidence

The following are acceptable evidence of learning. We will assess the following via the six facets of understanding outlined by Wiggins (Ch4). To demonstrate mature understanding, students: can explain, can interpret, can apply, can empathize, have perspective, and have self-knowledge of the concepts.

Performance Tasks: what would be sufficient and revealing evidence of understanding?

  • Students can create a branch for their own work (can apply)
  • Students can create a pull request and communicate via the Git interface with the team. (can apply)
  • Students can fork a project from a Git repository (can apply)
  • Students can add text to a document and push it to a repo on Git (can apply)
  • Students can create a merge request and communicate with others in a team. (perspective)
  • Students can resolve conflicts in a file to make a finalized merged document (perspective, can apply)
  • Students can remove a committed document and erase history (can apply)

Other Evidence Required: What are the types of evidence required by Stage 1 desired results?

Academic Prompts:

  • A merged finalized document on the main development branch: Students will add a short profile about themselves and their development background on a pre-existing document.  They will request to merge their part into the main document and deal with any conflicts. The result is a text document that contains background profiles about each of the members of the team. 
  • Properly merged commit tree on a new Git repository after the assignment. The commit tree on the newly created repository should resemble standard merge practices on Git and have evidence of proper communication between teammates.
  • Students can write proper Git messages to communicate a commit or a merge appropriately (empathy)

Quiz and Test Items:

  • Explain the Git conflict resolution process (can explain)
  • Explain Git branches and when to use them (can explain)
  • Describe forking, merging, rebase, revert, and reset (can explain)

Student Self-assessment

  • Students will reflect on improvements to adopt within their groups for a more complex document (self-knowledge).
  • Justify the current workflow and explain other workflows students could have taken to arrive at the same results (self-knowledge).

Stage 3 – Learning Experiences

These exercises are designed to get creative juices flowing and facilitate collaboration between team members. They provide specific workflows and reflections so students can see the bigger picture. They are assessed through the WHERETO standard (Wiggins, 2005):

  • Where is it going?
  • Hook the students
  • Explorer and Equip
  • Rethink and revise
  • E2xhibit and evaluate
  • Tailor to specific needs
  • Organize for maximum engagement and effectiveness

In the classroom (Optional)

Provide each group with a poster board and have students write a brief paragraph on the poster board highlighting their common interests. Each student should write their programming experience and name on individual pieces of paper. Then, manually combine the individual papers with the poster board to create a single poster representing the group. (W,H,O)

Reflect on the following question: What difficulties might arise if we attempt to concurrently write on the poster board? (R)

Setup

Read 1.3-1.8 in the Pro Git Book (W)

  1. The students will complete a survey about their previous exposure to Git, command line tools, and teamwork. The exercise will begin at a different stage depending on their past experience. (W)
    • Students who lack prior experience will be required to install Git. (T)
    • Students who have some familiarity will need to configure Git. (T)
  2. An experienced student will serve as the repository owner. They will fork (or download) a markdown file for a pre-existing project description template from a repository and transfer it to a new repository accessible by the entire team. (E, T)
  3. Reflect on the following question: What are other ways that step 2 could have been accomplished? What are the advantages and disadvantages of each? (R, E2, O)

Experience 1: Git Basics as an individual developer

Read 2.1-2.5 in the Pro Git Book (W)

  1. Students will create a new markdown file that includes their answers to the self-assessment survey. (H, E, T)
    • They will view the changes that they have not committed yet.
    • They will commit their changes. (E)
  2. Students will practice un-staging a commit and re-staging it. (E, E2)
  3. Students will examine their remote and push their work to the main branch. They will check the main branch on the repo and see the results. (H, O)
  4. Reflect on the following question: What will I do if I have staged a change but want to amend it? What if I have committed a change but want to amend it? (R)
Figure 1: Results of Experience 1 on Git

Experience 2: Git Branching

Read 3.1-3.4 in the Pro Git Book (W)

  1. The students will generate a new working branch and switch to it. (E)
  2. In the shared markdown file, students will insert a section introducing themselves. This section should highlight their strengths, developer background, interests, and sociocultural background. (E, T)
  3. They will then commit and push these changes to their branch. (E, O)
  4. Reflect on the following question: With respect to the classroom exercise, what issues may arise during this phase? What is the definition of “stashing”? (R)
Figure 2: Results of Experience 2 on Git

Experience 3: Merge Request

  1. Students will initiate a merge request. (E)
  2. They will then wait for feedback from other group members, with every member expected to participate. (E)
  3. Finally, they will merge their branch into the main branch. (H, O)
  4. Reflect on the following question: What should you look for when a group member creates a merge request? What type of feedback would be constructive and supportive to the group member? (R, E2)
Figure 3: Result of Experience 3 on Git

Reflection

This backwards design process was a great way to carry out this exercise. Evaluating the facets of understanding made it apparent that most assessment evidence centred around application, which makes sense since learning software is fundamental here. Incorporating an empathetic and reflective component into activities could also encourage students to think strategically about how Git can be used in collaboration and why they should use it as part of their software engineering coursework. Ultimately, my hope is that these exercises will promote better comprehension of using Git for future projects.

Posted on

Authentic PBL Experience in Software Engineering

Project-based learning (PBL) is a teaching approach that has recently gained popularity. It allows students to develop real-world problem-solving skills by working on a project in a classroom setting over an extended period. In software engineering, project-based learning has become a common approach to teaching students how to develop high-quality software. However, there has been debate over the authenticity of these projects, with some arguing that it does not accurately reflect real-world software development challenges and complexities. In this post, we will analyze the authenticity of a course-based software engineering project and how it can be improved to better prepare students for the challenges of the software development industry.

Software engineering Lifecycle and ISTE

The International Society for Technology in Education (ISTE) contains a set of standards for students to use technology effectively for learning. The 4th ISTE standard for students is:

    This standard highlights the importance of students using digital tools to manage projects, solve problems, and make informed decisions, as well as the ability to think critically and analyze risks in design, which are essential skills in modern software development.

    The software development lifecycle embodies ISTE standard 4 for students, emphasizing the importance of producing high-quality software. The lifecycle comprises four primary stages: specification, design, implementation, and maintenance. The specification stage involves gathering and documenting the software requirements, while the design stage focuses on planning and structuring the software. The implementation stage is dedicated to developing and testing the software, and the maintenance stage involves monitoring and updating the software to ensure that it remains functional and usable over time. Each stage is an iterative process that produces partial outputs of the final deliverables.

    Teaching the software engineering lifecycle in a classroom can present several challenges. One of the main issues is that software development is a highly dynamic process that is difficult to replicate in a classroom setting. Classroom projects may lack the same complexity or scope as real-world software projects, which can hinder students’ understanding of the software development lifecycle. Additionally, classroom environments may not provide students with the same level of exposure to real-world tools and technologies commonly used in software development (Kay et al., 2000). To address these challenges, the following project design outline is intended to enhance the effectiveness of a PBL software engineering project by integrating real-world elements.

    Project Design

    This project design methodology draws inspiration from the backwards design approach (Wiggins, 2005), and it has been compiled from three PBL projects by Shekar et al. (2014), Abad et al. (2019), and Spichkova et al. (2015) that were effectively carried out.

    Through the project, students will understand the fundamental software lifecycle process and the reality of rapid changes in real-world requirements. They will also learn about the critical role of effective communication in ensuring customer satisfaction and the importance of maintaining quality standards throughout a software development project.

    The software project aims to improve students’ hard and soft skills. Hard skills, which are measurable and specific, consist of software development skills in a particular language or framework, project planning, and written communication. In contrast, soft skills are personal attributes that enable students to collaborate effectively, such as teamwork, decision-making, communication, and organization.

    To demonstrate a student’s understanding, the following evidence is required: appropriately documented changes made to the software at appropriate intervals, effective communication with the customer, including asking relevant questions, and the production of working software at the end of the development cycle that adheres to proper design and testing procedures.

    Shekar et al. (2019) have proposed a set of assessments that should include the following elements:

    • A project proposal that aims to replicate the initial phase of the software development lifecycle, including customer communication.
    • A detailed design that specifies various alternatives and justifies the selection of the chosen design.
    • A demonstration of the final product that establishes the student’s ability to effectively communicate their solution and exhibit a functional end product.
    • The final project report that confirms that the software product was constructed according to the proposed design and testing procedures during the entire software development process.
    • A self-reflection component that involves maintaining a journal that records successes, failures, and evaluations of team members.

    Best Practices

    Software Engineering educators encounter a significant challenge in balancing the provision of a realistic experience and maintaining classroom control when designing a project. The guidelines and best practices presented below are once again derived from successful project designs outlined in Shekar et al. (2014), Abad et al. (2019), and Spichkova et al. (2015).

    1. Provide real-world connections and real industry experience. Providing inadequate project specifications and forcing requirements to change can effectively challenge students and prepare them for real-world scenarios. A vagueness in the problem description from the client will force students to ask questions and shape their requirements-gathering workflow. Furthermore, a requirements change later in the process can force students to adapt and develop explicit change management skills (Abad et al. 2019).
    2. Include flexibility and team-oriented decision-making. In the approach by Abad et al. (2019), students can exercise flexibility in their development process by selecting from various software process models, including Opportunistic, Waterfall, Spiral, Concurrent, or Scrum, and justifying their choice of methodology for their system development. They can also modify their chosen process model as the project advances. Similarly, in Spichkova et al.’s (2015) approach, the design architecture incorporates a team-oriented decision-making component to encourage collaborative decision-making among team members.
    3. Focus on team dynamics. Effective teamwork is a critical factor in the success of a project. It is essential to clarify that everyone has an equal role and that all team members share ownership of the code base (Sepahkar et al., 2014). According to Bates (2022), another crucial aspect is establishing an interactive and asynchronous communication medium which all team members can access and utilize. Examples of such mediums include Facebook messenger, MS teams, and Discord. This fosters a collaborative environment where all team members are accountable to each other and can respond promptly to ensure the success of the project.
    4. Include personal reflections. Shekar et al. (2014) suggest incorporating personal reflection journals at each stage of the development process. This gives students a high-level view of the entire process and allows them to reflect on their progress. In the reflection journal, students should consider what went well during each stage, any unexpected difficulties they encountered, how to improve pacing, engagement, and assessments, and how to align each stage more closely with the primary objectives of the iteration.

    Conclusion

    Providing an authentic project-based learning experience in software engineering can be a challenging but rewarding experience for software engineering educators. By designing projects that simulate real-world scenarios, students can develop both technical and soft skills that are crucial for success in the software development industry. Moreover, incorporating assessment strategies that align with the software development lifecycle and emphasizing effective communication, collaboration, and self-reflection can enhance the learning experience for students. By following the guidelines and best practices proposed, educators can strike a balance between providing students with a realistic experience, maintaining classroom control, and preparing them for the challenges and opportunities in the software engineering industry.

    Reference:

    Kay, J., Barg, M., Fekete, A., Greening, T., Hollands, O., Kingston, J. H., & Crawford, K. (2000). Problem-based learning for foundation computer science courses. Computer Science Education, 10(2), 109-128.

    Sepahkar, M., Hendessi, F., & Nabiollahi, A. (2015). Defining Project Based Learning steps and evaluation method for software engineering students. International Journal of Computer Science and Information Security, 13(10), 48.

    Spichkova, M. (2019, November). Industry-oriented project-based learning of software engineering. In 2019 24th International conference on engineering of complex computer systems (ICECCS) (pp. 51-60). IEEE.

    Shekar, A. (2014, June). Project-based learning in engineering design education: sharing best practices. In 2014 ASEE Annual Conference & Exposition (pp. 24-1016).

    Wiggins, G. P., & McTighe, J. (2005). Understanding by design (2nd ed.). Association For Supervision And Curriculum Development.

    Bates, A. W. (Tony). (2022). 7.5 Broadcast or interactive media? Pressbooks.bccampus.ca. https://pressbooks.bccampus.ca/teachinginadigitalagev3m/chapter/8-3-broadcast-vs-communicative-technologies/

    Abad, Z. S. H., Bano, M., & Zowghi, D. (2019, May). How much authenticity can be achieved in software engineering project based courses?. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) (pp. 208-219). IEEE.