argilla
argilla copied to clipboard
[FEATURE] `disagreement` for `unification` can either deduplicate or duplicate responses that are agread on
Is your feature request related to a problem? Please describe. Let's consider two questions, A and B, with the following responses:
Question A: [1, 1, 1] Question B: [2, 1, 2]
When unifying responses based on disagreement, there are three possible scenarios:
Combine all individual responses:
Result: [1, 1, 1, 2, 1, 2]
Explanation: Each response is included, resulting in [1, 1, 1, 2, 1, 2]. When comparing with the original responses for A and B, [1, 1, 1] remains unchanged, and [2, 1, 2] also remains unchanged.
Combine a set of all responses only if all agree:
Result: [1, 2]
Explanation: Only the agreed-upon responses are included. In this case, the agreed-upon elements are [1]. Therefore, the result is [1] from the common agreement between [1, 1, 1] and [2, 1, 2].
Combine a set of all responses per record:
Result: [1, 2, 1]
Explanation: Combine the unique responses for each record. For question A, [1, 1, 1] yields [1]. For question B, [2, 1, 2] yields [1, 2]. Combining these sets results in [1, 2, 1].
These three scenarios represent different ways of unifying responses based on disagreement between question A and question B.
Describe the solution you'd like N.A.
Describe alternatives you've considered N.A.
Additional context Not sure if it is the most reasonable and if we should change/differentiate between those but It is good to investigate what is commonly done and if our current implementation is correct
I'll let the paper for reference
This issue is stale because it has been open for 90 days with no activity.
This issue was closed because it has been inactive for 30 days since being marked as stale.