talmolab
Predicting on single frames mis-assigns predictions in multi-video projects
A user reported a potential bug when running single frame inference in a multi-video project where predicting on the current frame in the first video works fine, but predicting on the current frame in the second video of the project results in the labeled frame being added to the first video.
See CLI in the photo:
Relevant CLI:
sleap-track labels.v002.slp --video.index 1 --frames 1 -m path/to/model
We might be doing something funky with the indexing or when merging back in?
A little more context and testing - the same things happens when I do prediction on 20 frames from each movie (the first movie has 20 that predict fine, the next 20 correspond to positions of the animal in the second movie but are written over the first movie file in the GUI), as well as when predicting on a whole movie (again first movie works fine, second overwrites everything on the first except the hand-labeled frames).
I've included what could be relevant readouts below (although to me it seems like the movie indices do line up with what should be happening). One other thing to mention is that the videos have the same filename (although different full paths).
sleap-track E:/three_chamber/sleap/labels.v002.slp --video.index 0 --frames 2896,4527,5064,6762,7967,8410,11375,13353,17055,17158,18411,18434,20349,21273,21509,22036,23965,24600,30652,36535 -m E:/three_chamber/sleap\models\threechamber4_230810_154919.single_instance.n=206\training_config.json --tracking.tracker none -o E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205450.predictions.slp --verbosity json --no-empty-frames
Started inference at: 2023-08-10 20:54:57.018911 Args: { 'data_path': 'E:/three_chamber/sleap/labels.v002.slp', 'models': [ 'E:/three_chamber/sleap\models\threechamber4_230810_154919.single_instance.n=206\training_config.json' ], 'frames': '2896,4527,5064,6762,7967,8410,11375,13353,17055,17158,18411,18434,20349,21273,21509,22036,23965,24600,30652,36535', 'only_labeled_frames': False, 'only_suggested_frames': False, 'output': 'E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205450.predictions.slp', 'no_empty_frames': True, 'verbosity': 'json', 'video.dataset': None, 'video.input_format': 'channels_last', 'video.index': '0', 'cpu': False, 'first_gpu': False, 'last_gpu': False, 'gpu': 'auto', 'max_edge_length_ratio': 0.25, 'dist_penalty_weight': 1.0, 'batch_size': 4, 'open_in_gui': False, 'peak_threshold': 0.2, 'max_instances': None, 'tracking.tracker': 'none', 'tracking.target_instance_count': None, 'tracking.pre_cull_to_target': None, 2023-08-10 20:54:58.507694: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 'tracking.pre_cull_iou_threshold': None, 'tracking.post_connect_single_breaks': None, 'tracking.clean_instance_count': None, 'tracking.clean_iou_threshold': None, 'tracking.similarity': None, 'tracking.match': None, 'tracking.robust': None, 'tracking.track_window': None, 'tracking.min_new_track_points': None, 'tracking.min_match_points': None, 2023-08-10 20:54:59.027776: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3991 MB memory: -> device: 0, name: GeForce GTX 1660 SUPER, pci bus id: 0000:65:00.0, compute capability: 7.5 'tracking.img_scale': None, 'tracking.of_window_size': None, 'tracking.of_max_levels': None, 'tracking.save_shifted_instances': None, 'tracking.kf_node_indices': None, 'tracking.kf_init_frame_count': None }
INFO:sleap.nn.inference:Auto-selected GPU 0 with 5355 MiB of free memory. 2023-08-10 20:55:03.832582: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201 Versions: SLEAP: 1.3.1 TensorFlow: 2.7.0 Numpy: 1.21.5 Python: 3.7.12 OS: Windows-10-10.0.19041-SP0
System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initalized: False Memory growth: True
Finished inference at: 2023-08-10 20:55:06.438046 Total runtime: 9.419135808944702 secs Predicted frames: 20/20 Provenance: { 'model_paths': [ 'E:/three_chamber/sleap\models\threechamber4_230810_154919.single_instance.n=206\training_config.json' ], 'predictor': 'SingleInstancePredictor', 'sleap_version': '1.3.1', 'platform': 'Windows-10-10.0.19041-SP0', 'command': 'C:\Users\jakoa\miniconda3\envs\sleap\Scripts\sleap-track E:/three_chamber/sleap/labels.v002.slp --video.index 0 --frames 2896,4527,5064,6762,7967,8410,11375,13353,17055,17158,18411,18434,20349,21273,21509,22036,23965,24600,30652,36535 -m E:/three_chamber/sleap\models\threechamber4_230810_154919.single_instance.n=206\training_config.json --tracking.tracker none -o E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205450.predictions.slp --verbosity json --no-empty-frames', 'data_path': 'E:/three_chamber/sleap/labels.v002.slp', 'output_path': 'E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205450.predictions.slp', 'total_elapsed': 9.419135808944702, 'start_timestamp': '2023-08-10 20:54:57.018911', 'finish_timestamp': '2023-08-10 20:55:06.438046' }
Saved output: E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205450.predictions.slp Process return code: 0 Command line call: sleap-track E:/three_chamber/sleap/labels.v002.slp --video.index 1 --frames 1934,8071,8494,13088,16105,17350,20777,21491,22107,22694,23412,23415,25806,29294,29717,29756,32272,32981,36745,37418 -m E:/three_chamber/sleap\models\threechamber4_230810_154919.single_instance.n=206\training_config.json --tracking.tracker none -o E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205508.predictions.slp --verbosity json --no-empty-frames
Started inference at: 2023-08-10 20:55:14.323040 Args: { 'data_path': 'E:/three_chamber/sleap/labels.v002.slp', 'models': [ 'E:/three_chamber/sleap\models\threechamber4_230810_154919.single_instance.n=206\training_config.json' ], 'frames': '1934,8071,8494,13088,16105,17350,20777,21491,22107,22694,23412,23415,25806,29294,29717,29756,32272,32981,36745,37418', 'only_labeled_frames': False, 'only_suggested_frames': False, 'output': 'E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205508.predictions.slp', 'no_empty_frames': True, 'verbosity': 'json', 'video.dataset': None, 'video.input_format': 'channels_last', 'video.index': '1', 'cpu': False, 'first_gpu': False, 'last_gpu': False, 'gpu': 'auto', 'max_edge_length_ratio': 0.25, 'dist_penalty_weight': 1.0, 'batch_size': 4, 'open_in_gui': False, 'peak_threshold': 0.2, 'max_instances': None, 'tracking.tracker': 'none', 'tracking.target_instance_count': None, 2023-08-10 20:55:15.793491: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 'tracking.pre_cull_to_target': None, 'tracking.pre_cull_iou_threshold': None, 'tracking.post_connect_single_breaks': None, 'tracking.clean_instance_count': None, 'tracking.clean_iou_threshold': None, 'tracking.similarity': None, 'tracking.match': None, 'tracking.robust': None, 'tracking.track_window': None, 'tracking.min_new_track_points': None, 2023-08-10 20:55:16.293716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3991 MB memory: -> device: 0, name: GeForce GTX 1660 SUPER, pci bus id: 0000:65:00.0, compute capability: 7.5 'tracking.min_match_points': None, 'tracking.img_scale': None, 'tracking.of_window_size': None, 'tracking.of_max_levels': None, 'tracking.save_shifted_instances': None, 'tracking.kf_node_indices': None, 'tracking.kf_init_frame_count': None }
INFO:sleap.nn.inference:Auto-selected GPU 0 with 5361 MiB of free memory. 2023-08-10 20:55:21.015130: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201 Versions: SLEAP: 1.3.1 TensorFlow: 2.7.0 Numpy: 1.21.5 Python: 3.7.12 OS: Windows-10-10.0.19041-SP0
System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initalized: False Memory growth: True
Finished inference at: 2023-08-10 20:55:23.567285 Total runtime: 9.244246006011963 secs Predicted frames: 20/20 Provenance: { 'model_paths': [ 'E:/three_chamber/sleap\models\threechamber4_230810_154919.single_instance.n=206\training_config.json' ], 'predictor': 'SingleInstancePredictor', 'sleap_version': '1.3.1', 'platform': 'Windows-10-10.0.19041-SP0', 'command': 'C:\Users\jakoa\miniconda3\envs\sleap\Scripts\sleap-track E:/three_chamber/sleap/labels.v002.slp --video.index 1 --frames 1934,8071,8494,13088,16105,17350,20777,21491,22107,22694,23412,23415,25806,29294,29717,29756,32272,32981,36745,37418 -m E:/three_chamber/sleap\models\threechamber4_230810_154919.single_instance.n=206\training_config.json --tracking.tracker none -o E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205508.predictions.slp --verbosity json --no-empty-frames', 'data_path': 'E:/three_chamber/sleap/labels.v002.slp', 'output_path': 'E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205508.predictions.slp', 'total_elapsed': 9.244246006011963, 'start_timestamp': '2023-08-10 20:55:14.323040', 'finish_timestamp': '2023-08-10 20:55:23.567285' }
Saved output: E:/three_chamber/sleap\predictions\labels.v002.slp.230810_205508.predictions.slp Process return code: 0
This could be due to having the same video name - suspicious point of interest
