How to run the online tracker for every frame?

[mizeller]

In the online demo, the tracker runs on every 4th step because of this if-clause in online_demo.py:

if i % model.step == 0 and i != 0:
       _process_step(...)

Is it possible to run the tracker on every frame somehow? I tried adjusting the model.window_len which ultimately determines model.step, but could not figure out a working solution somehow.

[nikitakaraevv]

Hi @mizeller, yes, the current online model indeed only works with 4-frame windows. There is going to be a lag of 3 frames, and we don't have a solution for this yet.

[mizeller]

@nikitakaraevv Thank you for the speedy reply! I guess I'll continue with the regular CoTrackerPredictor if there is no easy "hacky" solution in that case :-)

[mizeller]

(BTW: the variables W and H are not initialised in the case of is_first_step=True and queries is not None in the online predictor --> small bug)

if is_first_step:
      self.model.init_video_online_processing()
      if queries is not None:
          B, N, D = queries.shape
          assert D == 3
          queries = queries.clone()
          queries[:, :, 1:] *= queries.new_tensor(
          [
                (self.interp_shape[1] - 1) / (W - 1), # here
                (self.interp_shape[0] - 1) / (H - 1), # and here
                ....

[nikitakaraevv]

Thanks! Fixed it: https://github.com/facebookresearch/co-tracker/commit/fac27989b31568a6a874f2e0ee00a44361a503c5