dsp-api
dsp-api copied to clipboard
dasch-swiss
InconsistentTriplestoreDataException/has no creator error
I get this error on a gravsearch query:
webapi_1 | 2019-10-23 09:12:05 | ERROR | Dispatcher | Entity http://rdfh.ch/0113/9h2huqLgS4aG9d3xLgXcIQ/values/-szOyfUkRnGYbF0UULjnUg has no creator
webapi_1 | org.knora.webapi.InconsistentTriplestoreDataException: Entity http://rdfh.ch/0113/9h2huqLgS4aG9d3xLgXcIQ/values/-szOyfUkRnGYbF0UULjnUg has no creator
webapi_1 | at org.knora.webapi.util.PermissionUtilADM$.$anonfun$getUserPermissionFromAssertionsADM$1(PermissionUtilADM.scala:414)
webapi_1 | at scala.collection.MapLike.getOrElse(MapLike.scala:131)
webapi_1 | at scala.collection.MapLike.getOrElse$(MapLike.scala:129)
webapi_1 | at scala.collection.AbstractMap.getOrElse(Map.scala:63)
webapi_1 | at org.knora.webapi.util.PermissionUtilADM$.getUserPermissionFromAssertionsADM(PermissionUtilADM.scala:414)
webapi_1 | at org.knora.webapi.util.PermissionUtilADM$.getUserPermissionFromConstructAssertionsADM(PermissionUtilADM.scala:389)
webapi_1 | at org.knora.webapi.util.ConstructResponseUtilV2$.$anonfun$splitMainResourcesAndValueRdfData$16(ConstructResponseUtilV2.scala:411)
webapi_1 | at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
webapi_1 | at scala.collection.immutable.List.foreach(List.scala:392)
webapi_1 | at scala.collection.TraversableLike.map(TraversableLike.scala:237)
webapi_1 | at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
webapi_1 | at scala.collection.immutable.List.map(List.scala:298)
webapi_1 | at org.knora.webapi.util.ConstructResponseUtilV2$.$anonfun$splitMainResourcesAndValueRdfData$15(ConstructResponseUtilV2.scala:395)
webapi_1 | at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
webapi_1 | at scala.collection.immutable.Map$Map3.foreach(Map.scala:195)
webapi_1 | at scala.collection.TraversableLike.map(TraversableLike.scala:237)
webapi_1 | at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
webapi_1 | at scala.collection.AbstractTraversable.map(Traversable.scala:108)
webapi_1 | at org.knora.webapi.util.ConstructResponseUtilV2$.$anonfun$splitMainResourcesAndValueRdfData$6(ConstructResponseUtilV2.scala:391)
webapi_1 | at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
webapi_1 | at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:234)
webapi_1 | at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:465)
webapi_1 | at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:465)
webapi_1 | at scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:465)
webapi_1 | at scala.collection.TraversableLike.map(TraversableLike.scala:237)
webapi_1 | at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
webapi_1 | at scala.collection.AbstractTraversable.map(Traversable.scala:108)
webapi_1 | at org.knora.webapi.util.ConstructResponseUtilV2$.splitMainResourcesAndValueRdfData(ConstructResponseUtilV2.scala:330)
webapi_1 | at org.knora.webapi.responders.v2.search.MainQueryResultProcessor$.getMainQueryResultsWithFullGraphPattern(MainQueryResultProcessor.scala:221)
webapi_1 | at org.knora.webapi.responders.v2.SearchResponderV2.$anonfun$gravsearchV2$13(SearchResponderV2.scala:539)
webapi_1 | at scala.util.Success.$anonfun$map$1(Try.scala:255)
webapi_1 | at scala.util.Success.map(Try.scala:213)
webapi_1 | at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
webapi_1 | at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
webapi_1 | at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
webapi_1 | at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
webapi_1 | at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
webapi_1 | at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
webapi_1 | at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
webapi_1 | at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
webapi_1 | at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
webapi_1 | at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
webapi_1 | at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
webapi_1 | at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
webapi_1 | at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
webapi_1 | at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
webapi_1 | at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
@benjamingeer and @tobiasschweizer I shared with you on switchdrive the file export-LL-23.10.2019.trig.zip (x-trig export of my graphdb repos)
To reproduce with gravsearch (being the user [email protected] / 9e1601a8) :
PREFIX knora-api: <http://api.knora.org/ontology/knora-api/v2#>
PREFIX knora-simple-api: <http://api.knora.org/ontology/knora-api/simple/v2#>
PREFIX ll: <http://0.0.0.0:3333/ontology/0113/lumieres-lausanne/v2#>
CONSTRUCT {
?Collection knora-api:isMainResource true .
?Collection ll:collectionHasTitle ?collectionHasTitle .
?Collection ll:collectionHasUrlElement ?collectionHasUrlElement .
?Collection ll:collectionHasDescription ?collectionHasDescription .
?Collection ll:hasBibliographicNoticeElement ?hasBibliographicNoticeElement .
?Collection ll:hasTranscriptionElement ?hasTranscriptionElement .
?Collection ll:hasPersonElement ?hasPersonElement .
} WHERE {
?Collection a ll:Collection .
?Collection ll:collectionHasTitle ?collectionHasTitle .
?Collection ll:collectionHasUrlElement ?collectionHasUrlElement .
OPTIONAL {
?Collection ll:collectionHasDescription ?collectionHasDescription .
}
OPTIONAL {
?Collection ll:hasBibliographicNoticeElement ?hasBibliographicNoticeElement .
?hasBibliographicNoticeElement a ll:BibliographicNotice .
}
OPTIONAL {
?Collection ll:hasTranscriptionElement ?hasTranscriptionElement .
?hasTranscriptionElement a ll:Transcription .
}
OPTIONAL {
?Collection ll:hasPersonElement ?hasPersonElement .
?hasPersonElement a ll:Person .
}
}
ORDER BY ASC(?collectionHasTitle)
OFFSET 2
The related resource and its value reported by the error seem to be fine:
.../v2/resources/http%3A%2F%2Frdfh.ch%2F0113%2F9h2huqLgS4aG9d3xLgXcIQ
{
"@id": "http://rdfh.ch/0113/9h2huqLgS4aG9d3xLgXcIQ/values/-szOyfUkRnGYbF0UULjnUg",
"@type": "knora-api:LinkValue",
"knora-api:arkUrl": {
"@type": "xsd:anyURI",
"@value": "https://ark.dasch.swiss/ark:/72163/1/0113/9h2huqLgS4aG9d3xLgXcIQ8/4TyOg7WDQFyuRQ2vYMUuBQo"
},
"knora-api:attachedToUser": {
"@id": "http://rdfh.ch/users/lumieres-lausanne-fantin_reichler"
},
"knora-api:hasPermissions": "CR http://rdfh.ch/groups/0113/lumieres-lausanne-administrator,http://rdfh.ch/groups/0113/lumieres-lausanne-director|D knora-admin:Creator|M http://rdfh.ch/groups/0113/lumieres-lausanne-phdstudent,http://rdfh.ch/groups/0113/lumieres-lausanne-researcher,http://rdfh.ch/groups/0113/lumieres-lausanne-student|V http://rdfh.ch/groups/0113/lumieres-lausanne-user,knora-admin:KnownUser,knora-admin:UnknownUser",
"knora-api:linkValueHasTarget": {
"@id": "http://rdfh.ch/0113/oDFFjxYWQ-iz6WBKwhtnCA",
"@type": "lumieres-lausanne:BibliographicNotice",
"knora-api:arkUrl": {
"@type": "xsd:anyURI",
"@value": "https://ark.dasch.swiss/ark:/72163/1/0113/oDFFjxYWQ=iz6WBKwhtnCAY"
},
"knora-api:attachedToProject": {
"@id": "http://rdfh.ch/projects/0113"
},
"knora-api:attachedToUser": {
"@id": "http://rdfh.ch/users/lumieres-lausanne-fantin_reichler"
},
"knora-api:creationDate": {
"@type": "xsd:dateTimeStamp",
"@value": "2017-03-07T12:23:41Z"
},
"knora-api:hasPermissions": "CR http://rdfh.ch/groups/0113/lumieres-lausanne-administrator,http://rdfh.ch/groups/0113/lumieres-lausanne-director|D knora-admin:Creator|M http://rdfh.ch/groups/0113/lumieres-lausanne-phdstudent,http://rdfh.ch/groups/0113/lumieres-lausanne-researcher|V http://rdfh.ch/groups/0113/lumieres-lausanne-student,http://rdfh.ch/groups/0113/lumieres-lausanne-user,knora-admin:KnownUser,knora-admin:UnknownUser",
"knora-api:userHasPermission": "D",
"knora-api:versionArkUrl": {
"@type": "xsd:anyURI",
"@value": "https://ark.dasch.swiss/ark:/72163/1/0113/oDFFjxYWQ=iz6WBKwhtnCAY.20170307T122341Z"
},
"rdfs:label": "Qu'est-ce qu'un médecin des Lumières? Portraits et discours croisés de quelques contemporains de Tissot"
},
"knora-api:userHasPermission": "D",
"knora-api:valueCreationDate": {
"@type": "xsd:dateTimeStamp",
"@value": "2019-10-22T15:30:12.399105Z"
},
"knora-api:valueHasUUID": "4TyOg7WDQFyuRQ2vYMUuBQ",
"knora-api:versionArkUrl": {
"@type": "xsd:anyURI",
"@value": "https://ark.dasch.swiss/ark:/72163/1/0113/9h2huqLgS4aG9d3xLgXcIQ8/4TyOg7WDQFyuRQ2vYMUuBQo.20191022T153012399105Z"
}
},
Is the resource very large? If so, it could be that the SPARQL query result is truncated. @tobiasschweizer would you have time to look at this?
If so, it could be that the SPARQL query result is truncated.
Yes, I think we had such problems before. But you removed the size limit, didn't you?
It's still there in the scripts in src/webapi/scripts:
owlim:query-limit-results "50000";
@gfoo Could you just try to make the limit higher and see if this fixes the problem? You have to change it it the template file, e.g., webapi/scripts/graphdb-se-knora-prod-repository-config.ttl.tmpl.
I thought it would be good to have both a time and size limit to protect agains overload.
Of course, it is hard so say what a reasonable limit is. Could we make some recommendations in terms of how many properties a resource should have at max.? I really do not want to implement paging for resource requests.
Relevant discussion with @mrivoal here:
https://github.com/dasch-swiss/knora-api/issues/1320#issuecomment-494300576
In the past I have spotted problems when GraphDB couldn't serialize a response because it was too large. This could also be problem with fewer properties that have large values like texts, see https://github.com/dhlab-basel/knora-large-texts.
In the past I have spotted problems when GraphDB couldn't serialize a response because it was too large. This could also be problem with fewer properties that have large values like texts
Yes, I think I've seen that too, but haven't had time to investigate it yet. I wonder if there's a configurable response size limit in GraphDB (in KB rather than in triples).
I wonder if there's a configurable response size limit in GraphDB (in KB rather than in triples).
Good question. I couldn't find anything like this on http://graphdb.ontotext.com/documentation/free/configuring-a-repository.html#configure-a-repository-programmatically
It would be nice to know how many bytes GraphDB is actually returning in this case. That could give us a clue.
It would be nice to know how many bytes GraphDB is actually returning in this case. That could give us a clue.
Haven't you already written some kind of logic that could be used for this in the code that handles the communication with the triplestore?
In HttpTriplestoreConnector.getSparqlHttpResponse, you could print the length of the response string before returning it:
https://github.com/dasch-swiss/knora-api/blob/f089f4b7820f47bbda93c9af33de88e5570cd8d3/webapi/src/main/scala/org/knora/webapi/store/triplestore/http/HttpTriplestoreConnector.scala#L677
If it's an extended construct response, it will be parsed by ConstructResponseTurtleHandler, which could count the statements in handleStatement and print out the statement count in endRDF:
https://github.com/dasch-swiss/knora-api/blob/f089f4b7820f47bbda93c9af33de88e5570cd8d3/webapi/src/main/scala/org/knora/webapi/messages/store/triplestoremessages/TriplestoreMessages.scala#L192
@gfoo Could you just try to make the limit higher and see if this fixes the problem? You have to change it it the template file, e.g., webapi/scripts/graphdb-se-knora-prod-repository-config.ttl.tmpl.
yes it works with owlim:query-limit-results "100000"
@gfoo Good to know. The problem with owlim:query-limit-results is that it just truncates the result if it's too big. It would be better to have a setting that returned an error if the result was too big.
yes it works with owlim:query-limit-results "100000"
So, where's your problem then? ;-)
Anyway, in my particular case ~~I could also to not ask~~ I could also avoid asking has*Element properties to really limit data retrieved.
The main problem there is to understand the error :)