[GSoC'24] Consensus annotation

[zhiltsov-max]

Motivation and context

Depends on https://github.com/cvat-ai/cvat/pull/8283, https://github.com/cvat-ai/cvat/pull/8401

Closes https://github.com/cvat-ai/cvat/issues/7973

How has this been tested?

Checklist

• [ ] I submit my changes into the develop branch
• [ ] I have created a changelog fragment
• [ ] I have updated the documentation accordingly
• [ ] I have added tests to cover my changes
• [ ] I have linked related issues (see GitHub docs)
• [ ] I have increased versions of npm packages if it is necessary (cvat-canvas, cvat-core, cvat-data and cvat-ui)

License

• [ ] I submit my code changes under the same MIT License that covers the project. Feel free to contact the maintainers if that's a concern.

Summary by CodeRabbit

• New Features

  • Introduced a new analytics page for consensus reporting.
  • Added support for merging consensus jobs directly from the job actions menu.
  • Implemented a dedicated component for displaying assignee reports and consensus conflicts.
  • Enhanced task management with new consensus-related settings and properties.

• Bug Fixes

  • Improved error handling and state management for consensus job operations.

• Documentation

  • Updated documentation to reflect changes in consensus job handling and analytics features.

• Style

  • Added new styles for improved layout and presentation of consensus-related components.

• Tests

  • Expanded test coverage for new consensus functionalities and components.

[coderabbitai[bot]]

[!IMPORTANT]

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The changes introduce a comprehensive consensus mechanism within the CVAT application, enabling users to manage multiple job copies, aggregate annotations, and merge them into a source job. This functionality includes generating reports for the merged data and reviewing the merging process. The modifications span various components in both the core and UI layers, enhancing the overall architecture to support these new features.

Changes

Files	Change Summary
.vscode/launch.json	Added a new debug configuration for "RQ - consensus" to facilitate debugging of consensus tasks.
changelog.d/...	Multiple entries added documenting the introduction of consensus features, settings, and reports.
cvat-core/src/api-implementation.ts	Enhanced conflict management with new types and functions for merging consensus conflicts.
cvat-core/src/api.ts	Introduced a new consensus object with methods for handling consensus reports and settings.
cvat-core/src/consensus-*.ts	New classes and interfaces for managing consensus reports, conflicts, and settings were added.
cvat-ui/src/...	Multiple UI components updated to support consensus analytics, including forms, tables, and menus.
cvat-ui/src/actions/...	New Redux actions for managing consensus jobs and analytics were implemented.

Assessment against linked issues

Objective	Addressed	Explanation
Allow annotating the same data multiple times and merge them (#[7973])	✅
Aggregating their annotations (#[7973])	✅
Merging them into the source job (#[7973])	✅
Generating a report for the same (#[7973])	✅
Reviewing the merging process (#[7973])	❓	Unclear if a review mechanism is implemented.

🐇 In fields of code, we hop with glee,
New features sprout, as bright as can be!
Merging jobs, oh what a sight,
Consensus blooms, bringing delight!
With each new task, we leap and bound,
In our coding burrow, joy is found! 🌼

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[bsekachev]

I feel the code was not merged correctly

FYI: we recently removed some files from components/task-quality/* as it was moved to enterprise. If this patch also uses them, need to re-store and update our internal code to reduce code duplication

[Viditagarwal7479]

If this patch also uses them, need to re-store and update our internal code to reduce code duplication

Hello @bsekachev, how would you suggest to proceed further on this?

[Viditagarwal7479]

I have made changes to fix the rest of the reviews on which I haven't commented. Once we figure out how to fix the breaks due to sharing components from quality analytics with consensus analytics, I shall merge them!

[sonarqubecloud[bot]]

Quality Gate failed

Failed conditions
5.7% Duplication on New Code (required ≤ 3%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[Viditagarwal7479]

Hello @zhiltsov-max , Are you able to build this locally? Since, I am getting the following error while trying to build this locally.

docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d --build

216.5 gcc: internal compiler error: Illegal instruction signal terminated program as
216.5 Please submit a full bug report,
216.5 with preprocessed source if appropriate.
216.5 See <file:///usr/share/doc/gcc-11/README.Bugs> for instructions.
216.5 make: *** [ffbuild/common.mak:59: libavformat/isom.o] Error 4
[+] Running 0/1 Waiting for unfinished jobs....
 ⠧ Service cvat_server  Building                                                                                                                                                                                                  221.7s 
failed to solve: process "/bin/sh -c curl -sL https://ffmpeg.org/releases/ffmpeg-${FFMPEG_VERSION}.tar.gz --output - |     tar -zx --strip-components=1 &&     ./configure --disable-nonfree --disable-gpl --enable-libopenh264         --enable-shared --disable-static --disable-doc --disable-programs --prefix=\"${PREFIX}\" &&     make -j5 && make install && make clean" did not complete successfully: exit code: 2

[zhiltsov-max]

@Viditagarwal7479, yes, it compiles and runs without problems on my machine. The error looks like it can be something specific to your CPU or platform. Please try to launch outside of the containers or maybe change GCC version for the build.

[zhiltsov-max]

Added merge plan in the description.

[Viditagarwal7479]

@Viditagarwal7479, yes, it compiles and runs without problems on my machine. The error looks like it can be something specific to your CPU or platform. Please try to launch outside of the containers or maybe change GCC version for the build.

Thank You, removing ffmpeg and ffmpeg4 installed by brew and the related enviornment variables solved this issue.

[Viditagarwal7479]

Added merge plan in the description.

Aren't this 2 already implemented? [ ] Implement consensus task creation (API and UI) [ ] Implement consensus job merging and settings (API and UI)

What more params are you looking for related to the merge conflicts? [ ] Implement simple analytics for merge conflicts

[Viditagarwal7479]

https://github.com/cvat-ai/cvat/pull/8895 in this PR I have updated the schema.yml because of which CI / build (pull request) build is failing and updated this branch with develop

[sonarqubecloud[bot]]

Quality Gate failed

Failed conditions
5.3% Duplication on New Code (required ≤ 3%)
D Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[zhiltsov-max]

Added merge plan in the description.

Aren't this 2 already implemented? [ ] Implement consensus task creation (API and UI) [ ] Implement consensus job merging and settings (API and UI)

What more params are you looking for related to the merge conflicts? [ ] Implement simple analytics for merge conflicts

There is an implementation for these points in the current PR. The description says about in which parts this PR will be merged.

[SpecLad]

@zhiltsov-max Is this PR still relevant?

[zhiltsov-max]

It has a PoC for consensus analytics, but I think we can return to this PR later, if needed.