Lack of Spark Support for On-Demand Feature Views

[FabioTomaz]

Currently, stream feature views support both spark and spark_sql execution modes, which enables them to scale efficiently with large datasets. However, on-demand feature views are limited to pandas or python modes only.

This limitation becomes a bottleneck when trying to perform transformations on features that haven't been precomputed, especially in cases where those transformations need to scale to large data volumes. Without Spark support, users are forced to rely on local computation, which significantly hinders scalability and performance.

[HaoXuAI]

Team is working on it 👍

[franciscojavierarceo]

A temporary hack is to use the ODFV as a UDF. That's ultimately what the real implementation will look like.