Feature Request: Writing EVT3 (& EVT2) RAW format

[mandulaj]

When processing recorded event videos, it is often useful to write the processed events into a file. Naturally, there is a plethora of formats available, ranging from CSV, HDF5, Numpy, etc... to custom solutions.

Nevertheless, most of these formats have significant shortcomings:

• They lack the ability to store additional information (such as external trigger events)
• The storage density is low resulting in huge files
• Parsing these formats is slow resulting in lengthy reading/writing
• Non-standard, leading to many tools not supporting the reading/writing

Despite being a proprietary Raw format and coming with its own shortcomings, the EVT3 seems to cover a lot of the points above, including being tightly compatible with the Metavision SDK tools (in terms of reading at least)

However, there doesn't seem to be any way of taking a processed event stream and writing it back into an EVT3 format.

It would be nice to provide this functionality, similar to the the H5EventsWriter or DatWriter as a Evt3Writer API.

[lbristiel-psee]

Thanks for your feedback. We are currently considering adding an EVT3 encoder in OpenEB. Though, I notice that you mention the H5EventsWriter Python class in your message. This class is writing HDF5 compressed event file. This feature and format are a first version of an attempt to support compressed event file in HDF5 format in our SDK. Since then, we developed a more efficient compression algorithm used in a new HFD5 file format: ECF. This file format is explained in this page: https://docs.prophesee.ai/stable/data/file_formats/hdf5.html. As mentioned in there, those HDF5 event files can be played with Metavision Studio, Metavision Viewer and most of our samples (including Python ones). The main missing piece currently is that we don't yet provide an HDF5ECFEventsWriter in Python, so to write those files you have to use C++.

We continue working on this topic, improving our SDK and its documentation. Check our future release notes and come back to us if you have other questions or suggestions.

[mandulaj]

Is the HDF5 ECF format suitable for random Seeks into the files? Similar to how the index file can be used with the RAW files.

Also, we have written our own rudimentary EVT2 and EVT3 encoder that generates valid raw formats, however, it doesn't perform any encoding of Vecotred events, and thus can't exploit the compression to its fullest.

[lbristiel-psee]

Is the HDF5 ECF format suitable for random Seeks into the files? Similar to how the index file can be used with the RAW files.

yes, HDF5 ECF contains an index of the CD events and hence is seekable. You can test it out by opening an HDF5 ECF file in Studio (HDF5 ECF files can be found in here: https://docs.prophesee.ai/stable/datasets.html). You can also check the HDF5 reader code in here: https://github.com/prophesee-ai/openeb/blob/main/sdk/modules/driver/cpp/src/hdf5_event_file_reader.cpp

Also, we have written our own rudimentary EVT2 and EVT3 encoder that generates valid raw formats, however, it doesn't perform any encoding of Vecotred events, and thus can't exploit the compression to its fullest.

I understand, this is the tricky part. We hope to have something to share soon on that matter.

[berndpfrommer]

I wrote a simple EVT2 -> EVT3 conversion tool. It's using some vect primitives but not all of them. https://github.com/berndpfrommer/evt_conversion