Feature Caching fails

[mh0797]

Hi, I ran the preprocessing using python nuplan/planning/script/run_training.py +training=training_vector_model py_func=cache cache.force_feature_computation=true data_augmentation="[]" which returned Completed dataset caching! Failed samples: 6770 out of -188520323.

First, i am wondering how the total number of samples can be negative ?

Second, i found out that in all of the 6770 failed samples, the agents feature is missing. By inspecting a single scenario token I found out that if the scene is empty (no actors), then the feature computation works fine but feature.is_valid returns false so that the created feature will not be stored [See here]. Is there a reason for this?

Third, the output at the end of the caching is misleading. I did the caching for another model (4 features and 1 target). The final output was Completed dataset caching! Failed samples: 0 out of 2556075. However, I scanned the cache dir and found out that in total 511215 scenarios were processed (the entire mini dataset). I guess the output refers to the number of features / targets instead of the number of samples/scenarios.

[patk-motional]

Hi @mh0797,

1. The total number of samples should not be negative. Something is very wrong there. Let me try to replicate your issue
2. We can re-evaluate if we should change the validity condition for the Agents feature. @dimitris-motional can elaborate on this.
3. ~~Scenario does not equate to samples. When training the data loader operates on samples, not scenarios. If we fail to compute a sample there can be other samples from the same scenario that succeeds.~~

[mh0797]

Hi @patk-motional, 3. Can you elaborate more precisely, what exactly a scenario consists of and how it can be distinguished from a sample? As far as I can see, for every scenario (i.e. every token) exactly one set of features and targets is computed.

3. The total number in the output refers to the number of features and targets. Assuming a sample consists of 4 features and 1 target (like in my case), then processing 511215 scenarios results in 511215*5=2556075 features/targets which is different to having 2556075 distinct samples.

[patk-motional]

Hey,

I'd like to take back what I said. For training you are correct, sample == scenario. For each sample, multiple features and targets can be computed. What is confusing here is the logging statement. Where it says Failed samples: it should say Failed features and targets:.

With that said, we can update the logging to be clearer.

Thanks for pointing this out

[patk-motional]

Hi, with the new release the logging should be accurate now.

[mh0797]

Thank you.

How about the validity condition of the agents feature? As far as I can see, this was not updated so far.