GSoC 2025 Project proposal: CommitHunter: AI-Powered Commit Debugger

[llxia]

Project Idea Name

CommitHunter: AI-Powered Commit Debugger

Description  The goal of this project is to develop an automated system that identifies problematic Git commits causing test failures in both performance and non-performance test scenarios. Given a "Good" build (where tests pass) and a "Bad" build (where tests fail), the system will analyze all intermediate commits to pinpoint the problematic commit(s). The approach will evolve from rule-based methods to AI-driven models for higher accuracy. Phase 1: Rule-Based Approach Use Rule based approach and below are some examples:

String Matching: Identify test failure messages and correlate them with commit messages, logs, or diffs. Binary Search for Performance Tests: Implement a binary search approach to efficiently narrow down the commit range in performance test failures.

Phase 2: Machine Learning Model

Train an ML model using historical build data, commit logs, and failure reports. Utilize supervised learning techniques to classify commits as "Likely Problematic" or "Safe." Use natural language processing (NLP) to analyze commit messages and correlate them with test failures.

Phase 3: Automation and Integration

Integrate the approach with Git repositories (e.g., GitHub) via APIs. Develop a bot to automatically comment on Git issues with the identified problematic commit(s) if confidence levels meet a reliability threshold. Provide a dashboard for tracking identified problematic commits and their validation over time.

Links to Eclipse Project

https://github.com/eclipse/openj9 https://github.com/adoptium/aqa-tests https://github.com/adoptium/aqa-test-tools https://projects.eclipse.org/projects/technology.openj9 https://projects.eclipse.org/projects/adoptium.aqavit

Expected outcomes 

Reduction in manual effort needed to debug and triage test failures. Faster identification of problematic commits leading to improved development efficiency. A scalable system that can be expanded with improved AI models that can apply to other project and team.

Related: https://gitlab.eclipse.org/eclipsefdn/emo-team/gsoc-at-the-ef/-/issues/11