rdf4j
rdf4j copied to clipboard
eclipse-rdf4j
Max execution time does not work correctly for expensive queries
I have a very, very expensive query: 300 lines, lots of inner selects and optionals and concat operations.
If I set a max execution time of 10 seconds, the execution does not stop after 10 seconds but takes forever. My code looks like this:
Query q = con.prepareQuery(QueryLanguage.SPARQL, query); q.setMaxExecutionTime(10); TupleQueryResult result =((TupleQuery) q).evaluate();
while (result.hasNext()) { //code halts here forever! //do something }
I think the problem is because of TimeLimitIteration.java
TimeLimitInteration checks before and after super.hasNext() if the iteration was interrupted, but in my case this hasNext() takes forever so I get the interrupted exception way after 10 seconds.
Propagating the time limit information further down the chain seems to be the only way to resolve this, as it is a design issue, not an issue with the current implementation of TimeLimitIteration itself.
Indeed there currently is no (easy) fix for this, that I can see. My thinking is that we should look into a design with a dedicated query-execution thread, which has all the necessary interrupt-plumbing built in.
Alternatively, we need to add methods such interrupt and isInterrupted to the Iteration interface itself (perhaps sticking an Interruptable behavioral interface on top), making sure wrapper implementations can propagate the signal down. But that seems clunkier to me.
In either case, it will require some fundamental changes and it's definitely not something we can do as a simple patch.
In the case of SPARQLRepository and HTTPRepository queries, the time limit information may need to be propagated all the way down to the HTTP client if we are hoping to implement it ourselves.
It could also be implemented using an Executor with tasks submitted to it with a fixed time limit, where the JVM can kill the Callable/Future rather than being interrupted by custom code created by us.
This is an older issue, but several closely related problems have been reported over the years, which means that this is still very much relevant and needs looking into.
I don't think so though possibly some of the work that @hmottestad had been doing on iterations recently has made tackling this easier. But it hasn't really had priority I think.
