Hudi Job on Lambda - Directory /var/task/metastore_db cannot be created.

[sha12br]

Hi Team,

I have been trying to run Hudi job on Lambda, the function is getting properly invoked and the files are read fine as well. But the issue seems to be while writing i guess where its trying to create derby.log under /var/task (which is readonly).

I have followed through this repo guide, from CloudWatch Logs :

ERROR XBM0H: Directory /var/task/metastore_db cannot be created.

I have tried changing the config for "spark.sql.warehouse.dir" to local and S3 as well, Still for some reason its trying to create under /var/task (where the spark-submit was invoked).

Could someone please help me out here. Any suggestions/help would be great.

Thanks.

[JohnChe88]

Have tried adding HUDI configuration "hoodie.write.markers.type":"direct", # It's not advisable to use this configuration. Working on workaround without using this config. "hoodie.embed.timeline.server":"false" # It's not advisable to use this configuration. Working on workaround without using this config.

[JohnChe88]

Closing it because of no response