Serge Smertin
Serge Smertin
i'll look into it
Implementing that in #265, @rtyler please link the PR also to this issue.
@rtyler what you're mentioning about kafka ingest is called "userMetadata" and it is a string. We can't get a struct there.
@houqp we'd need to check to what extent is it arbitrary, as delta history technically returns commit infos.
https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ManifestFileCommitProtocol.scala#L115-L120 here's good enough reference on conventional filename formatting.
@dispanser Azure Databricks has "autoloader" feature, that resembles inotify behavior. Did you already try that?
@houqp there's autoloader on aws as well :)
Usage should be like https://github.com/nfx/AsyncBundle/blob/master/DependencyInjection/Compiler/CollectAsyncTasksPass.php#L72
so, what should i change?