argilla icon indicating copy to clipboard operation
argilla copied to clipboard

[FEATURE] `disagreement` for `unification` can either deduplicate or duplicate responses that are agread on

Open davidberenstein1957 opened this issue 1 year ago • 1 comments

Is your feature request related to a problem? Please describe. Let's consider two questions, A and B, with the following responses:

Question A: [1, 1, 1] Question B: [2, 1, 2]

When unifying responses based on disagreement, there are three possible scenarios:

Combine all individual responses:
    Result: [1, 1, 1, 2, 1, 2]
    Explanation: Each response is included, resulting in [1, 1, 1, 2, 1, 2]. When comparing with the original responses for A and B, [1, 1, 1] remains unchanged, and [2, 1, 2] also remains unchanged.

Combine a set of all responses only if all agree:
    Result: [1, 2]
    Explanation: Only the agreed-upon responses are included. In this case, the agreed-upon elements are [1]. Therefore, the result is [1] from the common agreement between [1, 1, 1] and [2, 1, 2].

Combine a set of all responses per record:
    Result: [1, 2, 1]
    Explanation: Combine the unique responses for each record. For question A, [1, 1, 1] yields [1]. For question B, [2, 1, 2] yields [1, 2]. Combining these sets results in [1, 2, 1].

These three scenarios represent different ways of unifying responses based on disagreement between question A and question B.

Describe the solution you'd like N.A.

Describe alternatives you've considered N.A.

Additional context Not sure if it is the most reasonable and if we should change/differentiate between those but It is good to investigate what is commonly done and if our current implementation is correct

davidberenstein1957 avatar Nov 20 '23 11:11 davidberenstein1957

I'll let the paper for reference

plaguss avatar Nov 21 '23 15:11 plaguss

This issue is stale because it has been open for 90 days with no activity.

github-actions[bot] avatar Mar 19 '24 01:03 github-actions[bot]

This issue was closed because it has been inactive for 30 days since being marked as stale.

github-actions[bot] avatar Apr 18 '24 01:04 github-actions[bot]