rportilla-databricks

Hi thanks for the question! We do not yet support structured streaming - but do you have a use case you're interested in? For example streaming AS OF joins? On...

Thanks for the detailed explanation! We can help with this - would you mind sending over your email so I can set up a meeting. On Tue, Jun 8, 2021...

@Sonali-guleria , just for reference.

Thanks for the note @rzsquirrel , this is actually due to catalyst not being enabled with there are 100+ columns and is a Spark-related effect. Putting the values in a...

@rzsquirrel , as a workaround, can you try updating this parameter to a column count higher than 100? spark.sql.codegen.maxFields This is why codegen fails to be enabled - I'm wondering...

@rzsquirrel , coming back to this issue, can you provide the timing on your tests above? When joining 100 cols to 100 columns, I'm getting a 20s runtime for the...

@BenLBurke , this is definitely an interesting request. Can you explain what functions you are looking for in particular? The reason we ask is because applyInPandas is the go-to method...

Hi Martin, we will take a look at this in a meeting next week to prioritize. Thanks for sending!

@sim-san , we will be supporting streaming for select operations. The first function which will support streaming is the asofJoin. Resample/interpolation will come later and are not scoped for the...

We should be able to remove partition columns. On Mon, Aug 15, 2022 at 11:41 AM Lorin Dawson ***@***.***> wrote: > In io.py we Z ORDER on partitionCols + optimizationCols...