Josh Rosen
Josh Rosen
Happy to accept PRs for this. Do you have any idea how hard this would be? Is it simply a matter of getting this library to not reject `.xz` files?...
Nope, it looks like we don't support this currently. I'd be happy to review a PR to add support for this, though.
To clarify, it looks like #121 is concerned with `logicalType`, whereas it sounds like this issue is discussing ReflectData / the `java-class` attribute. These are related but slightly orthogonal features,...
I agree that implicit conversion is bad in this context. This is a behavioral change that we should consider making in a 4.x release. For now, though, I think that...
I'm totally happy to help debug this, but it would be useful to have a standalone reproduction or an updated stacktrace. Could you post either here?
/cc @marmbrus, how should we handle this? Simply omitting the check probably isn't sufficient in case there are Hadoop temp files in the same directory, right?
Actually, this issue looks like a possible duplicate of #40.
Do we have a test case for this scenario? Even if we believe that it has been fixed, we should probably add a test to make sure it doesn't break...
So my hunch here is that the problem lies somewhere in our schema conversion of Spark SQL structs to avro schemas: https://github.com/databricks/spark-avro/blob/branch-3.1/src/main/scala/com/databricks/spark/avro/SchemaConverters.scala#L113 I don't have time to debug this at...
This turns out to be pretty easy to support now that we we honor user-provided Avro schemas during writes. I've opened #197 to fix this for 3.2.0.