sesame-results->seq (and therby evaluate) fails with an invokeNoArgInstanceMember exception

[lisp]

neither a direct invocation, nor an invocation via grafter.rdf.repository/evaluate succeeds:

grafter.rdf.repository=> (def query (.prepareTupleQuery connection org.openrdf.query.QueryLanguage/SPARQL "SELECT * WHERE {?s ?p ?o}"))
#'grafter.rdf.repository/query
grafter.rdf.repository=> query
#<SailTupleQuery DydraQueryRoot
   Projection
      ProjectionElemList
         ProjectionElem "s"
         ProjectionElem "p"
         ProjectionElem "o"
      StatementPattern
         Var (name=s)
         Var (name=p)
         Var (name=o)
>
grafter.rdf.repository=> (def results (.evaluate query))
#'grafter.rdf.repository/results
grafter.rdf.repository=> results
#<TupleQueryResultImpl org.openrdf.query.impl.TupleQueryResultImpl@67a95db>
grafter.rdf.repository=> (instance? info.aduna.iteration.CloseableIteration results)
true
grafter.rdf.repository=> (.close results)
nil
grafter.rdf.repository=> (sesame-results->seq query query-bindings->map)

NoSuchMethodError clojure.lang.Reflector.invokeNoArgInstanceMember(Ljava/lang/Object;Ljava/lang/String;Z)Ljava/lang/Object;  grafter.rdf.repository/sesame-results->seq (repository.clj:287)       
grafter.rdf.repository=> (use 'clojure.stacktrace)
nil
grafter.rdf.repository=> (print-stack-trace *e)
java.lang.NoSuchMethodError: clojure.lang.Reflector.invokeNoArgInstanceMember(Ljava/lang/Object;Ljava/lang/String;Z)Ljava/lang/Object;                                                             
 at grafter.rdf.repository$sesame_results__GT_seq.invoke (repository.clj:287)
    grafter.rdf.repository$eval2401.invoke (form-init5044810080896752162.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:6619)
    clojure.lang.Compiler.eval (Compiler.java:6582)
    clojure.core$eval.invoke (core.clj:2852)
    clojure.main$repl$read_eval_print__6588$fn__6591.invoke (main.clj:259)
    clojure.main$repl$read_eval_print__6588.invoke (main.clj:259)
    clojure.main$repl$fn__6597.invoke (main.clj:277)
    clojure.main$repl.doInvoke (main.clj:277)
    clojure.lang.RestFn.invoke (RestFn.java:1523)
    clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__1761.invoke (interruptible_eval.clj:72)
    clojure.lang.AFn.applyToHelper (AFn.java:159)
    clojure.lang.AFn.applyTo (AFn.java:151)
    clojure.core$apply.invoke (core.clj:617)
    clojure.core$with_bindings_STAR_.doInvoke (core.clj:1788)
    clojure.lang.RestFn.invoke (RestFn.java:425)
    clojure.tools.nrepl.middleware.interruptible_eval$evaluate.invoke (interruptible_eval.clj:56)
    clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__1803$fn__1806.invoke (interruptible_eval.clj:191)
    clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__1798.invoke (interruptible_eval.clj:159)
    clojure.lang.AFn.run (AFn.java:24)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
    java.lang.Thread.run (Thread.java:745)
nil

[lisp]

if i just go ahead and replace the grafter implementation with a naive copy, as in

(defn query-results-seq [results]
  (if (.hasNext results)
      (let [current-result (try
                            (query-bindings->map (.next results))
                            (catch Exception e
                                   (.close results)
                                   (throw e)))]
        (lazy-cat [current-result] (query-results-seq results)))
      (.close results)))

that succeeds.

[RickMoynihan]

Why do you need to access things at this level?

Can't you just use the grafter query function?

http://api.grafter.org/master/grafter.rdf.repository.html#var-query

[lisp]

as noted in my earlier comment, one is often not certain to be using the intended api.

as i recall, in this case, the query operator failed upon use and the stack trace indicated that the failure was in sequence construction function, which led me to reconstruct the proximate problem for the ticket.

[RickMoynihan]

Yes, the documentation could definitely be a lot better. Our tutorial documentation at the minute really only covers grafter.tabular and the triple templating DSL; not the sesame/SPARQL wrappers.

Regarding the APIs most of the time if the function is listed in the docs then you can use it, though this doesn't really explain the levels of abstraction (some are lower level than others).

I've just realised that we haven't built api docs for the 0.5.1 release so I'll add a ticket to do that, but the 0.6.0-SNAPSHOT docs are under master here:

http://api.grafter.org/

sesame-results->seq wasn't really intended for 3rd party use which is why it has the ^:no-doc annotation on it...

I'll have a think how we can perhaps partition the lower level APIs into an internal namespace; or perhaps I should just gather the higher level RDF functions together into grater.rdf with potemkin, as we've done for grafter.tabular.

grafter.tabular is the high level namespace for grafter.tabular.common - which perhaps we should rename to grafter.tabular.adapters or something similar.

Any more comments and thoughts appreciated...

[lisp]

there are two issues here:

• the capacity offered by your api
• that the implementation (at least in 0.5.0) failed in use.

this ticket is initially about the second issue. that is, an attempt to use the query operator with a non-sesame repository implementation fails:

dydra-csv.core=> (let [repo (com.dydra.ndk.sesame.DydraRepository. "pgsql/weather")]
            #_=>   (.initialize repo)
            #_=>   (with-open [conn (grafter.rdf.repository/->connection repo)]
            #_=>      (grafter.rdf.repository/query conn "SELECT * WHERE { ?s ?p ?o .}")))

NoSuchMethodError clojure.lang.Reflector.invokeNoArgInstanceMember(Ljava/lang/Object;Ljava/lang/String;Z)Ljava/lang/Object;  grafter.rdf.repository/sesame-results->seq (repository.clj:287)                                                                                                                     
dydra-csv.core=> (print-stack-trace *e)
java.lang.NoSuchMethodError: clojure.lang.Reflector.invokeNoArgInstanceMember(Ljava/lang/Object;Ljava/lang/String;Z)Ljava/lang/Object;                 
 at grafter.rdf.repository$sesame_results__GT_seq.invoke (repository.clj:287)
grafter.rdf.repository/fn (repository.clj:313)
    grafter.rdf.repository$fn__1258$G__1253__1263.invoke (repository.clj:301)
    grafter.rdf.repository/fn (repository.clj:358)
    grafter.rdf.protocols$fn__7501$G__7496__7510.invoke (protocols.clj:46)
    grafter.rdf.repository$query.doInvoke (repository.clj:449)
    clojure.lang.RestFn.invoke (RestFn.java:425)
    dydra_csv.core$eval2370.invoke (form-init7152653172577733991.clj:4)
    clojure.lang.Compiler.eval (Compiler.java:6619)
    clojure.lang.Compiler.eval (Compiler.java:6582)
    clojure.core$eval.invoke (core.clj:2852)
    clojure.main$repl$read_eval_print__6588$fn__6591.invoke (main.clj:259)
    clojure.main$repl$read_eval_print__6588.invoke (main.clj:259)
    clojure.main$repl$fn__6597.invoke (main.clj:277)
    clojure.main$repl.doInvoke (main.clj:277)
    clojure.lang.RestFn.invoke (RestFn.java:1523)
clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__1761.invoke (interruptible_eval.clj:72)
    clojure.lang.AFn.applyToHelper (AFn.java:159)
    clojure.lang.AFn.applyTo (AFn.java:151)
    clojure.core$apply.invoke (core.clj:617)
    clojure.core$with_bindings_STAR_.doInvoke (core.clj:1788)
    clojure.lang.RestFn.invoke (RestFn.java:425)
    clojure.tools.nrepl.middleware.interruptible_eval$evaluate.invoke (interruptible_eval.clj:56)
    clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__1803$fn__1806.invoke (interruptible_eval.clj:191)
    clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__1798.invoke (interruptible_eval.clj:159)
    clojure.lang.AFn.run (AFn.java:24)
    java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1145)
    java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
    java.lang.Thread.run (Thread.java:745)
nil
dydra-csv.core=>

this led to the investigating the underlying implementation and to the reiteration in terms of the lower-level operations which was reported in the ticket. the emulation in the first comment is a work-around, which functions. i did not investigate the discrepancy between the two behaviours, but just reported in the event that there was some known cause.

the emulation lead, however, to the questions about the api capacity, which have become the ticket's focus. this may well be for the better, as the current implementation (or its emulation) appears to yield datasets with values from the "rdf" rather than the "incanter" domain, which is not appropriate. perhaps your "restricted" datasets perform some magic here, but that question was not pursued as the error meant there was no direct path to get there.

this deficiency suggested to introduce into the emulation some form of extensible rdf->incanter value transformation

(defmulti term-value class)
(defmethod term-value org.openrdf.model.Literal [literal] (literal-datatype->type literal))
(defmethod term-value org.openrdf.model.URI [uri] (.toString uri))
(defmethod term-value org.openrdf.model.BNode [node] (clojure.string/join ["_:" (.getID node)]))

;;; from grafter/repository.clj                                                                                                                        
(defn query-bindings->map [^org.openrdf.query.BindingSet qbs]
  (let [boundvars (.getBindingNames qbs)]
    (->> boundvars
         (mapcat (fn [k]
                   [(symbol k) (-> qbs (.getBinding k) .getValue term-value)]))
     (apply hash-map))))

(defn query-results-seq [results]
  (if (.hasNext results)
      (let [current-result (try
                    (query-bindings->map (.next results))
                    (catch Exception e
                           (.close results)
                           (throw e)))]
(lazy-cat [current-result] (query-results-seq results)))
      (.close results)))

the intended result is that the dataset which is constructed from a query result will "automatically" have values in the intended incanter domains. the implementation is simplistic and should better permit the caller to provide an alternative to query-bindings->map, but it accomplished the necessary goal. the situation is analogous to that with importing datasets into the store. this order of value transformation is best situated at the interface between the two domains, available in some minimal form without additional programming and extensible where necessary.

[RickMoynihan]

Hi James,

Firstly can I suggest that we try and keep the issues separate. I see 3 issues being discussed here at the minute:

• The issue of better documentation of internals/APIs/Abstractions. If you could open a ticket on this issue describing the problems you have, we can discuss solutions to that there.
• I believe the issue about converting types to the "incanter domain" is what ticket #40 is about; if this is the case can you write up and move your suggestions into that ticket? Or open a new one.
• Finally there is what I'd consider to be "this issue" i.e. #48

Regarding this final issue I hadn't realised that you were trying to use a non sesame store. I just saw you were using the lower level APIs without realising you weren't just using a Sesame supplied Repository.

I can't test against the DydraRepository you're using (is it opensource?), but it looks like it doesn't implement evaluate in the same way sesame does.

I'd suggest that you therefore try and hook into overriding things at one level higher up by extending the top level ISPARQLable protocol to DydraRepository. Currently this assumes you also have a sesame dataset as a third argument, but you should be able to pass nil in here.

You should then be able to reuse any Grafter functions you need.

Let me know if this works for you or not... I'd also be interested in seeing the code, to see if we can make this kind of thing easier in the future.

Thanks again for your comments.

[lisp]

I can't test against the DydraRepository you're using (is it opensource?), but it looks like it doesn't implement evaluate in the same way sesame does.

yes, now we arrive at the original reason for the ticket... no, we are not open source. the clojure code will be, but it requires a local dydra installation.

I'd suggest that you therefore try and hook into overriding things at one level higher up by extending the top level ISPARQLable protocol to DydraRepository.

i had presumed, any class which specialized something for which the protocol was already defined would be acceptable. that is, given (https://github.com/Swirrl/grafter/blob/master/src/rdf-repository/grafter/rdf/repository.clj#L237)

(extend-type Repository
  pr/ISPARQLable
  (pr/query-dataset [this query-str model]
    (pr/query-dataset (.getConnection this) query-str model))

i had expected this to suffice:

dydra-csv.core=> (supers com.dydra.ndk.sesame.DydraRepository)
#{org.openrdf.repository.base.RepositoryBase org.openrdf.repository.Repository java.lang.Object org.openrdf.repository.sail.SailRepository}

[RickMoynihan]

I'd expect it to work too... but I can't easily debug interoperability with your repository, so you'll need to let me know what exactly is causing the problem and causing the Repository contract to be broken. It looks to me like the signature of the evaluate method in your case isn't quite what we're expecting... Perhaps we need to add a type hint, or alternatively, what version of Sesame are you extending?

I'm wondering if you're including an older or more recent version than we depend on?

We're still on the 2.7.x line and haven't updated to 2.8/4.0 yet, that's another thing that's on the list.

[lisp]

i am working with 2.7.3

the particularly perplexing issue is that, where i reimplemented the internals, it works. that is how i arrived at the point to complain about how you handle term values.

if you would tell me, where i might insert some sort of probe, i would be happy to do so. the stack trace is above.

[lisp]

to give a more complete indication, what we are up to, here is an initial version of the utility library:

https://bitbucket.org/dydra/dydra-dataset/