mcap icon indicating copy to clipboard operation
mcap copied to clipboard

MCAP filter cli

Open amacneil opened this issue 3 years ago • 2 comments
trafficstars

Ability to filter mcap files, similar to rosbag filter:

  • include/exclude topics (glob/regex or just static list?)
  • start/end time

amacneil avatar Mar 29 '22 20:03 amacneil

What if instead we implemented a filter-friendly API in Python, and delivered a mcap_filter.py wrapper for simple CLI jobs?

Here's my thought process:

  1. robot data manipulation tasks exist on a smooth complexity continuum, from: a. simple filtering: select all messages by channel or time range b. more complex filtering: filter by a field in a message, like frame_id in a sensor msg or hardware_id in /diagnostics c. transformation: reassign all images to one topic, fix timestamps, etc. ideally users can use tools from the same ecosystem for all of the above.
  2. Python is a super popular language for data wrangling, in the ROS ecosystem especially but also more generally.
  3. Building a nice python API for mcap manipulation will probably be really nice for users.
  4. Once that's done, it's easy to implement a command-line filter tool in Python, which would also double as an example script for users who want to dip their toes into mcap manipulation in Python.

james-rms avatar Jun 24 '22 10:06 james-rms

I agree with the use cases you outlined. However, our CLI is very easy to install (brew install mcap), and at the very least there should be an easy way to split out a single topic or time range (since we already have the ability to merge files from the CLI - seems weird if I can't do the reverse).

Landing more advanced filter features in the python library might be a better place to do it (and improving our python library in general is good). But honestly I think we should ignore more advanced filtering for now until there are requests for it - plenty of other things to work on.

amacneil avatar Jun 24 '22 22:06 amacneil

This was completed in https://github.com/foxglove/mcap/pull/445

james-rms avatar Aug 29 '22 23:08 james-rms