open-metric-learning
open-metric-learning copied to clipboard
OML-Team
Feature/metrics/stable metrics
Introduce permutation stable version of the retrieval metrics. The idea is to sort gt_tops for which distances are the same, i.e. if distances_tops and gt_tops are
distances_tops = [1, 2, 2, 2, 3]
gt_tops = [0, 1, 0, 1, 0]
then we need to obtain
gt_tops = [0, 0, 1, 1, 0]
# -------
# sort gt_tops where there are multiple
# distances of the same value
The code for the retrieval metrics itself doesn't change.
In the failed test called test_metrics_is_similar_in_ddp it is said that
Metrics shouldn't be equal, but very similar
If I change the atol for the test from 5e-3 to 8e-3 the test passes. Is this workaround ok?
@AlekseySh @DaloroAT hey guys. It looks like this PR is ready to be reviewed.
As I described above the main idea is to sort gt_tops values corresponding to the same distance values. In order to do it properly, first we need to find new value of max_k such that torch.any(distances[:, max_k] == distances[:, max_k + 1] is False (function _estimate_suitable_max_k). Next, we need to make the sorting (function _sort_gt_tops). And all the steps for getting the gt_tops is now encapsulated in function _get_gt_tops.
@DaloroAT What do you think? Does bringing the additional logic worth the benefits of metrics' stability? PS. Initially, I thought that the required extra logic would be smaller and simpler...
@dapladoc let's implement what we've decided offline:
- measure overhead
- introduce a flag for this functionality available for developers only (so, we don't expose is to config)
- depends on the first step, set this flag as True or False by default
Let me close it for now due to inactivity.
Metrics code has also been significantly modified.
The order of predictions is determined in RetrievalResults, so, I'm not sure this problem is even actual now.
So, anyway, the PR has to be reworked.