Removing Queries as the tracking progresses

[Clemens-E]

Hey, I'm amazed at how well this works, however I have an issue with implementing this for my use case.

In my scenario, both the camera and the points to track are moving, this makes it fairly hard, I do realize that, but its still works well.

My issue is, that as the video progresses, points that have long been out of frame get "re-detected" in absurd or weird locations because they roughly match the color of the last tracked point.

In my opinion, it would be helpful to have a way to remove or mark queries as "do not track further", or any other way to stop tracking them. I tried adjusting the queries mid-processing, but this did not work, made obvious by this code: predictor.py showing it's only set on the first step.

Any ideas? Could I just modify the code to set queries even if it's not the first step, or will this break things?