Stone-Soup icon indicating copy to clipboard operation
Stone-Soup copied to clipboard

Issues with the GOSPA metric

Open narykov opened this issue 1 year ago • 4 comments

I ran into a couple of problems trying to use GOSPA in place of the OSPA metric:

  1. While compute_OSPA_distance returns a scalar, compute_gospa_metric is designed to return a dictionary of values (including that of distance). This leads to problems with visualising the GOSPA errors using built-in tools, as scalar values are anticipated.
  2. In the Metrics example, it is not possible to replace OSPA with GOSPA metric (e.g., via from stonesoup.metricgenerator.ospametric import GOSPAMetric as OSPAMetric) as it leads to issues when trying to extract a timestamp from a numpy array of Particle elements.

Perhaps, some of these issues are known (at least I heard about the second one from @A-acuto), but I couldn't find them explicitly outlined here.

narykov avatar Apr 29 '24 11:04 narykov

I think issue 2. was fixed in #967

sdhiscocks avatar Apr 30 '24 09:04 sdhiscocks

@sdhiscocks Thanks for pointing out that! I was not aware, I found that issue some time ago and I forgot about mentioning before. Glad it is fixed now

A-acuto avatar Apr 30 '24 09:04 A-acuto

If we want a common visualisation process that is agnostic to whether we use GOSPA or OSPA, I think there is still an issue here. Do we want the OSPA output to look like GOSPA's or vice versa? I don't think we want to persist the current status quo

simonrmaskell avatar May 21 '24 15:05 simonrmaskell

While compute_OSPA_distance returns a scalar, compute_gospa_metric is designed to return a dictionary of values (including that of distance). This leads to problems with visualising the GOSPA errors using built-in tools, as scalar values are anticipated.

To get any visualisation, I 'hacked' computed_over_time() in GOSPA by putting metric.value = metric.value['distance'] above line 196.

https://github.com/dstl/Stone-Soup/blob/13f7578f7a5d4238b00c52a07fc0b184506687ec/stonesoup/metricgenerator/ospametric.py#L192-L196

This way I only access 'distance' metric, which then can be visualised. By replacing 'distance' with 'localisation', 'missed', or 'false', it is (potentially) possible to get access to other information that is particular to GOSPA.

P.S. Visualisation of the GOSPA metric is accessed via title='GOSPA Metric', and of the OSPA metric via title='OSPA distances', which is not very intuitive.

narykov avatar May 29 '24 15:05 narykov