Bug - Athena ingestion fail to patch the tags on database schemas if the database schema already exists

[nicor88]

Affected module Athena ingestion

Describe the bug Athena ingestion fails to patch the database schema tags if the schema already exists.

Full error log:

PATCH /api/v1/databaseSchemas/026d5a97-0326-4c40-a8fc-e4e4abb43656] o.o.s.e.CatalogGenericExceptionMapper - exception 
javax.json.JsonException: '{"id":"026d5a97-0326-4c40-a8fc-e4e4abb43656","name":"gold","fullyQualifiedName":"athena.awsdatacatalog. gold","dataProducts":[],"version":0.1,"updatedAt":1713524086664,"updatedBy":"ingestion-bot","service":{"id":"c8edce0d-c9ae-4920-833e-5fecae78e887","type":"databaseService","name":"athena","fullyQualifiedName":"athena","deleted":false},"serviceType":"Athena","database":{"id":"23de4186-bd83-477c-bf69-4ae948c9adbf","type":"database","name":"awsdatacatalog","fullyQualifiedName":"athena.awsdatacatalog","deleted":false},"tags":[],"deleted":false,"votes":{"upVotes":0,"downVotes":0,"upVoters":[],"downVoters":[]}}' contains no value for name 'sourceHash'
	at org.apache.johnzon.core.JsonPointerImpl.getValue(JsonPointerImpl.java:338)
	at org.apache.johnzon.core.JsonPointerImpl.validateJsonPointer(JsonPointerImpl.java:513)
	at org.apache.johnzon.core.JsonPointerImpl.validateRemove(JsonPointerImpl.java:320)
	at org.apache.johnzon.core.JsonPointerImpl.remove(JsonPointerImpl.java:282)
	at org.apache.johnzon.core.JsonPointerImpl.remove(JsonPointerImpl.java:266)
	at org.apache.johnzon.core.JsonPatchImpl.apply(JsonPatchImpl.java:75)
	at org.openmetadata.service.util.JsonUtils.applyPatch(JsonUtils.java:296)
	at org.openmetadata.service.util.JsonUtils.applyPatch(JsonUtils.java:300)
	at org.openmetadata.service.jdbi3.EntityRepository.patch(EntityRepository.java:838)
	at org.openmetadata.service.resources.EntityResource.patchInternal(EntityResource.java:273)
	at org.openmetadata.service.resources.databases.DatabaseSchemaResource.patch(DatabaseSchemaResource.java:339)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:292)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlets.HeaderFilter.doFilter(HeaderFilter.java:117)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:318)
	at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722)
	at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26)
	at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:840)

To Reproduce

• Athena ingestion with includeTags: false
• Run Athena ingestion with includeTags: true

Of course you need to have a lakeformation tagging for the glue catalog.

Expected behavior Tags for database schemas are patched even if the schema was already existing. Specifically patching the tags for an existing schema works fine using APIs:

import requests

schema_id='xxxxx'

patches = [
    {
        "op": "add",
        "path": "/tags/0",
        "value": {
            "tagFQN": "dataTier.bronze",
            "source": "Classification",
        }
    },
]

# example on how to patch a databaseSchemas
patch_header = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json-patch+json"}
r = requests.patch(f'http://{os.environ.get("OM_HOST")}:8585/api/v1/databaseSchemas/{schema_id}',
                   data=json.dumps(patches),
                   headers=patch_header)

Version:

• OS: Linux
• Python version: 3.11
• OpenMetadata version: 1.3.2
• OpenMetadata Ingestion package version: openmetadata-ingestion[athena]~=1.3.2.0

Additional context Add any other context about the problem here.