Bagfix all

[marinagmoreira]

• Fix all does entire pipeline
• Uniform bag_processing package

Still a WIP, needs tons of testing and more flexibility

[trey0]

Hi, Marina-

Big picture thought: I designed rosbag_fix_all.py with a particular use case in mind, namely that it could be something the Astrobee Facility team could run on raw bag files before they go through the public release process. Since the raw files are often already many GB in size and a big pain to download, I think for that use case it's important not to bloat the "fixed" bag output beyond the size of the raw bag.

Some reprocessing steps in your WIP script would increase the bag file size, so they might be better packaged separately as something the user could run after they download the fixed bag.

Also, the way this script currently works is that it takes an all-topics bag as input and produces an all-topics bag as output (possibly adding one new post-processed topic, or deleting one topic and replacing it with another). But depending on the use case, it can be nicer to output smaller separate bags for each new processed topic that take up less space on disk (not duplicating all the other topics) and can be processed more efficiently by downstream scripts.

In fact, from an ease-of-use and efficiency perspective, the best approach is often to have on-demand processing... like if a script expects RGB images as input, you just tell it the image topic to use, and if the images it receives happen to have a bayer encoding, the script itself can detect this and knows how to apply the necessary debayering, so the user running the script doesn't need to explicitly specify that, and you never need to write a huge bag of 3x-size debayered RGB images to disk. (Of course, automatic on-demand isn't always practical, but it sure can be nice.)

Two other minor comments:

• There is a minor conflict between this PR and #496, which ironically was an attempt to address one of your concerns.
• The Queen calibration changes seem unrelated, might better put them in a separate PR that could go right through.

[marinagmoreira]

Hi, Marina-

Big picture thought: I designed rosbag_fix_all.py with a particular use case in mind, namely that it could be something the Astrobee Facility team could run on raw bag files before they go through the public release process. Since the raw files are often already many GB in size and a big pain to download, I think for that use case it's important not to bloat the "fixed" bag output beyond the size of the raw bag.

Some reprocessing steps in your WIP script would increase the bag file size, so they might be better packaged separately as something the user could run after they download the fixed bag.

Also, the way this script currently works is that it takes an all-topics bag as input and produces an all-topics bag as output (possibly adding one new post-processed topic, or deleting one topic and replacing it with another). But depending on the use case, it can be nicer to output smaller separate bags for each new processed topic that take up less space on disk (not duplicating all the other topics) and can be processed more efficiently by downstream scripts.

In fact, from an ease-of-use and efficiency perspective, the best approach is often to have on-demand processing... like if a script expects RGB images as input, you just tell it the image topic to use, and if the images it receives happen to have a bayer encoding, the script itself can detect this and knows how to apply the necessary debayering, so the user running the script doesn't need to explicitly specify that, and you never need to write a huge bag of 3x-size debayered RGB images to disk. (Of course, automatic on-demand isn't always practical, but it sure can be nice.)

Two other minor comments:

• There is a minor conflict between this PR and rosbag_fix_all.py: make it easier to clean up failed output bags #496, which ironically was an attempt to address one of your concerns.
• The Queen calibration changes seem unrelated, might better put them in a separate PR that could go right through.

Please note that this PR is a Draft and not suitable for review. My idea for the final form would address your comments. I still want to make it possible to skip any processing pipeline and have specific profiles for specific needs (e.g. for map building we want to filter unnecessary topics at the end). I am aware of the conflicting PR, I can merge it on this PR no problem

[trey0]

Please note that this PR is a Draft and not suitable for review. My idea for the final form would address your comments. I still want to make it possible to skip any processing pipeline and have specific profiles for specific needs (e.g. for map building we want to filter unnecessary topics at the end). I am aware of the conflicting PR, I can merge it on this PR no problem

Well, sure, I didn't try to do a complete review, but I thought that the main reason to file a Draft PR is for team awareness and feedback, right?

Maybe I didn't convey properly the spirit of my comments. It's more that I'm not super-satisfied with the way rosbag_fix_all.py works already. It's kind of a brittle mess that can be inconvenient to use, so I'm kind of hunting for different ideas on how to structure new bag processing features so they don't have to become part of that mess. But if you're convinced this is the right approach, don't let me stop you.

[bcoltin]

Maybe we could add a bit more explanation? Especially I noticed there are these calibration(?) coefficients for the haz cam on different robots. How did we get these, how do we recalibrate? What if we want to do this for perch cam or different robots?