Problems creating a custom sensor

[EltonCN]

Hi! I'm developing some custom sensors to capture data, but I ran into some problems when trying to develop:

1. In the SensorDefinition class, the default simulationDeltaTime value (0.0f) causes the Editor to stuck on the "Application.EnterPlayMode" operation when entering player mode. The stuck makes sense, but I think that checking and raising some warning when the value is 0 would help to debug the code.

2. Searching in the repository files (including the documentation), the only place I found that had an understandable explanation of what the modality parameter of the SensorDefinition class is in the SOLO Dataset Schema documentation (Definitions->sensor_definitions.json) The actual documentation, "The mode of the sensor", doesn't seem to me enough to understand what information should be entered in it.

3. In the "Custom sensors" documentation, the SensorHandle.ReportCapture does not exist (should be ReportSensor).

Overall, I think some simple example of how to create a sensor, as there is for creating annotations, would help a lot to create one, as I had to keep switching between the necessary classes and their implementations for the camera to understand what I should do.

Thanks!

[StevenBorkman]

Thanks @EltonCN for the feedback and the heads up on some problem documentation. I agree that the Sensor documentation is pretty lacking and is something that needs to be enhanced in the future. Truthfully, we really didn't expect for a lot of work on custom sensors, so this is pretty cool that you are going down that route. If you don't mind me asking, what type of sensor are you implementing?

[nacorti]

I'm also interested in creating custom sensors, though I'll admit I'm extremely new to Unity and still getting my feet wet.

My use case is that I'd like to be able to capture twin synchronized RGB inputs as well as a depth map in order to generate training data and ground truth for a deep learning model.

[EltonCN]

Hi! Thank you for the response.

I'm working at a research group that aims to create cognitive architectures for mobile devices, so my project goal is to create a "user simulator", modeling complex behaviors and incrementally adding realistic models of real and virtual sensors that cell phones usually have, like GPS, IMUs, light intensity, proximity and so on.

I haven't started implementing any specific sensors yet, I was just trying to understand how the toolkit works for implement new sensors. I'm also thinking about how to minimize rework to implement sensors that can be used in different scenes to capture data (Perception) and train agents (ML Agents).

And @nacorti , I think you can get what you need using two PerceptionCamera? Two for the color data and one also for the depth info.

[StevenBorkman]

@nacorti You should be able to achieve what you are looking for out of the box. We can run with multiple perception cameras and have the ability to capture depth maps with the DepthLabeler.

[nacorti]

@EltonCN @StevenBorkman Can confirm that I got it working! I actually used 3 PerceptionCamera, two recording only RGB images and a middle one recording depth, but no RGB.

I'm not sure if it was intentional, but I do appreciate that even though I didn't explicitly "connect" them, all 3 cameras stay in sync and output to the same folder. This makes generating a training set much easier. Thanks gents!