About VOT setup issue (#2)

[MARMOTatZJU]

The former issue has been created here: (https://github.com/xl-sr/THOR/issues/6) and has been closed by the repo collaborator.

The original code, which uses the frame number (f) that is not provided to tracker's init function according to VOT official toolkit, is pasted as follows

        # initialize the long term module
        if not f or self._cfg.vanilla:
            self.lt_module = LT_Module(K=self._cfg.K_lt, template_keys=self.template_keys,
                                       lb=self._cfg.lb, lb_type=self._cfg.lb_type,
                                       verbose=self._cfg.verbose, viz=self._cfg.viz)
            self.lt_module.fill(temp)
        else:
            # reinitialize long term only at the beginning of the episode
            self.lt_module.update(temp, div_scale=0)

With this version of code, I successfully reproduced the reported result(EAO=0.416).

{'acc': 0.5818219747855319, 'robustness': 0.2341262408690766, 'eao': 0.4160014947564685, 'mean_fps': 94.913035446227}

However, as I commented the conditional branch which uses f:

        # initialize the long term module
        # if not f or self._cfg.vanilla:
        self.lt_module = LT_Module(K=self._cfg.K_lt, template_keys=self.template_keys,
                                   lb=self._cfg.lb, lb_type=self._cfg.lb_type,
                                   verbose=self._cfg.verbose, viz=self._cfg.viz)
        self.lt_module.fill(temp)
        # else:
            # reinitialize long term only at the beginning of the episode
            # self.lt_module.update(temp, div_scale=0)

, I got this result:

{'acc': 0.5797615533198762, 'robustness': 0.28563401386027343, 'eao': 0.368673347545505, 'mean_fps': 102.34960671562767}

Honestly, with an EAO of 0.369, the method did bring improvement w.r.t. to SiamRPN baseline(EAO=0.322), but not as much as reported while being removed the conditional branch that uses the frame number information. The latter indicates if an initialization is due to the beginning of new test video or a reinitialization after a drift. This information should not been provided to trackers according to VOT official toolkit.

I wish that a reasonable justification can be provided w.r.t. to this question before this issue is closed. Thanks.

[xl-sr]

The reason was already given in #6, resetting templates during a sequence destroys the purpose of a long-term module. In the evaluation procedure of VOT, the resetting is done by giving the ground truth position to the tracker for reinitialization, then the tracking is continued. However, it is not stated that the tracker also needs to reset its template. I agree that the current implementation is potentially confusing, in a future commit the current structure will be adapted to make it more conform to the standard VOT API.

Obviously, you cannot just comment out parts of the code and expect it to still reproduce reported results. The hyperparameters for THOR were searched for this setting. If you want to run VOT with a full reset after every drift (which makes the task easier, not harder) you can use the following hyperparameters:

 {
    "tracker": {
      "penalty_k": 0.078722,
      "window_influence": 0.318922,
      "lr": 0.448255
    },
    "THOR": {
      "K_st": 6,
      "K_lt": 4,
      "iou_tresh": 0.91256,
      "lb": 0.841342,
      "tukey_alpha": 0.881996,
      "lb_type": "dynamic",
      "modulate": true,
      "dilation": 10,
      "context_temp": 0.5,
      "viz": false,
      "verbose": false,
      "vanilla": false
    }
}

This yields the following performance:

{'acc': 0.5640, 'robustness': 0.2107,  'eao': 0.4174, 'mean_fps':  107.0628}

[MARMOTatZJU]

All right thanks for your reply. However, among the released part of code, I didn't find the code for the hyper-parameter search. Would you mind specify with which technique you performed the hyper-parameter search?