[WIP] Add support for `userMetadata` argument to delta update, delete, and merge

[PeterDowdy]

Refers to https://github.com/delta-io/delta/issues/857

I used the spark .scalafmt for this code but it made formatting changes that don't seem like they should have happened; what's the right .scalafmt for this project?

Can someone point me in the right direction on this? The update and delete paths hit spark code pretty much right away, so there isn't a way to set the userMetadata on just that command that I can tell - am I looking in the wrong place? I tried setting and unsetting the userMetadata in config before and after execution, and it works, but it's ugly (and maybe unstable)?

Is there a way to embed the userMetadata into the query plan somehow? Is there a way to get the userMetadata set in the Operation that gets run?

Description

This PR adds userMetadata as an argument to Delta update, delete, and merge commands.

How was this patch tested?

Tests were added to Delete, Update, MergeInto, and HadoopOptions suites

Does this PR introduce any user-facing changes?

Yes, it updates the contracts of the update, delete, and merge methods to these:

DeltaTable.delete(condition, userMetadata)
DeltaTable.update(condition, set, userMetadata)
DeltaTable.merge(source, condition, userMetadata)

[PeterDowdy]

Can someone help me understand this error?

[error] delta-core: Failed binary compatibility check against io.delta:delta-core_2.13:2.2.0! Found 3 potential problems (filtered 299)
[error]  * method this(io.delta.tables.DeltaTable,org.apache.spark.sql.Dataset,org.apache.spark.sql.Column,scala.collection.immutable.Seq)Unit in class io.delta.tables.DeltaMergeBuilder does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("io.delta.tables.DeltaMergeBuilder.this")
[error]  * method executeDelete(scala.Option)Unit in class io.delta.tables.DeltaTable does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("io.delta.tables.DeltaTable.executeDelete")
[error]  * method executeUpdate(scala.collection.Map,scala.Option)Unit in class io.delta.tables.DeltaTable does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("io.delta.tables.DeltaTable.executeUpdate")

[scottsand-db]

Can someone help me understand this error?

Take a look at https://github.com/delta-io/delta/blob/master/project/MimaExcludes.scala.

MIMA is a tool to check for binary incompatibilities between APIs in different release versions.

[PeterDowdy]

This almost works, except the new java APIs don't exist in 2.12. How do I backport these?