Ryan Blue

It looks like we may be able to add another rule to Spark that automatically finds the outer scope based on the case class. This shows a rules that does...

Here's what worked: ```scala case AmmoniteREPLClass(cellClassName) => () => { val objClass = Utils.classForName(cellClassName) val objInstance = objClass.getField("MODULE$").get(null) objClass.getMethod("instance").invoke(objInstance) } ``` ```scala private[this] val AmmoniteREPLClass = """^(ammonite\.\$sess\.cmd(?:\d+)\$).*""".r ```

@alexarchambault, the PR is https://github.com/apache/spark/pull/23607. Feel free to review it or test it out in your environment.

This changes the notebook language to Python and duplicates a cell. I don't think it is something that's ready to merge. There's also a ton of extra metadata. Also, there's...

I thought Glue exposed the same Thrift API that Hive uses. If that's the case, then we should be able to use the same lock API and code.

I think that Glue should implement locking as required by the interface it exposes. I'd be fine adding a solution specific to Glue in Iceberg as well, but I'm not...

@Liorba, thanks for working on this! This is a good start. To answer your questions: 1. Integrating engines to use startsWith can be done in a follow-up. 2. I think...

@Liorba, thanks for updating, but this still needs to update the `project` methods in each transform so Iceberg can convert these into partition predicates.

@Liorba, if you want to continue working on this, please re-open it in the apache/incubator-iceberg repository. That's the project's new home. Thanks!

I think we're talking about different use cases that are both idempotent: * **Incremental processing** - I wouldn't want to build incremental processing on top of the `overwrite` you're referring...