USE reference@hash is always setting reference head

[ajantha-bhat]

In nessie spark sql extensions, use reference main at $someHash

is always setting the head reference instead of specified reference.

cause: At nessie side we are only using the refName here and ignoring the ref hash https://github.com/projectnessie/nessie/blob/main/clients/spark-extensions-base/src/main/scala/org/apache/spark/sql/execution/datasources/v2/NessieUtils.scala#L193

At iceberg side, we need a to introduce a new spark configuration (to accept refHash along with refName) https://github.com/apache/iceberg/blob/master/nessie/src/main/java/org/apache/iceberg/nessie/NessieCatalog.java#L129 and the set this hash in NessieCatalog.initialize().

Note So, it needs 2 PR one at iceberg and one at nessie. I will only work on this as I have analyzed. PR will be up tomorrow as it is bit late in my timezone.

Issue capture: use ref at hash.txt

[ajantha-bhat]

Raised #3440 from Nessie side. Once this is merged and release is made, need to fix at Iceberg side using this configuration. Later end to end test has to be updated in ITNessieStatements once Iceberg releases it.

[ajantha-bhat]

Raised https://github.com/apache/iceberg/pull/4330 at Iceberg side.

[nastra]

https://github.com/apache/iceberg/pull/4700 should fix this on the Iceberg side

[ajantha-bhat]

Once Nessie depends on Iceberg 0.14.0 (once, it is released) we need to add/update the test case at the Nessie side before closing this issue.

[snazy]

What's the status of this issue?

[ajantha-bhat]

What's the status of this issue?

we need to update the testcase on Nessie side. I will track it to closure this week

[ajantha-bhat]

Whenever the SQL "USE REFERENCE AT HASH" is used, defined hash will not be updated in the backend. Only client uses that information. Nessie integration testcases are independent of NessieClient.

Also, we may need to update the "SHOW REFERENCE" command similar to this (#5081). But that is an independent issue. Hence, closing this ticket as completed.