Support for join (lookup) using nested data in pull syntax

[ckshekhar]

I have a data model that has a vector of maps, and I am trying to look up another entity using a key of the map.

I have create a sample documents extracted from NextJournal, in which :movie/cast is a vector of map

(def my-docs
  [{:person/name "James Cameron",
    :person/born #inst "1954-08-16T00:00:00.000-00:00",
    :xt/id -100}
   {:person/name "Arnold Schwarzenegger",
    :person/born #inst "1947-07-30T00:00:00.000-00:00",
    :xt/id -101}
   {:person/name "Linda Hamilton",
    :person/born #inst "1956-09-26T00:00:00.000-00:00",
    :xt/id -102}
   {:person/name "Michael Biehn",
    :person/born #inst "1956-07-31T00:00:00.000-00:00",
    :xt/id -103}
   {:person/name "Ted Kotcheff",
    :person/born #inst "1931-04-07T00:00:00.000-00:00",
    :xt/id -104}
   {:person/name "Sylvester Stallone",
    :person/born #inst "1946-07-06T00:00:00.000-00:00",
    :xt/id -105}
   {:person/name "Richard Crenna",
    :person/born #inst "1926-11-30T00:00:00.000-00:00",
    :person/death #inst "2003-01-17T00:00:00.000-00:00",
    :xt/id -106}
   {:person/name "Brian Dennehy",
    :person/born #inst "1938-07-09T00:00:00.000-00:00",
    :xt/id -107}
   ;; Movies
   {:movie/title "The Terminator",
    :movie/year 1984,
    :movie/director -100,
    ;; Cast is a vector of map
    :movie/cast [{:cast-id -101 :another-attr true}
                 {:cast-id -102 :another-attr true}
                 {:cast-id -103 :another-attr false}],
    :movie/sequel -207,
    :xt/id -200}
   {:movie/title "First Blood",
    :movie/year 1982,
    :movie/director -104,
    :movie/cast [{:cast-id -105 :another-attr false}
                 {:cast-id -106 :another-attr true}
                 {:cast-id -107 :another-attr false}],
    :movie/sequel -209,
    :xt/id -201}])

I would like to get cast details for :movie/cast using :cast-id.

Ideally, we should be able to write a query something like the below example

(xt/q (xt/db node)
      {:find ['(pull ?e [:movie/title
                         :movie/year
                         {:movie/director [:person/name :person/born]}
                         ;; Get cast name using `:cast-id`
                         {:movie/cast [{:cast-id [:person/name
                                                  :person/born]}]}])]
       :where '[[?e :movie/title "First Blood"]]})

[refset]

Hi @ckshekhar - thanks again for opening this issue! The initial review/thought is that adding this capability would make pull somewhat asymmetrical with what the Datalog has access to (since only top-level values are indexed).

Have you tried using https://github.com/lilactown/pyramid before? I haven't used it, so I can't really vouch for it, but I suspect it may be helpful in the interim for processing the nested values that XT returns, using a further layer of EQL - based on this description:

Another common use case is like a select-keys on steroids: the ability to do nested selections on complex maps with nested collections pops up very often in code. Pyramid can take any non-normalized map and execute an EQL query on it, returning the result.

If you wanted to avoid post-processing of XT's pull responses, you could even call pyramid within your XT Datalog query using a 'custom function'.

[ckshekhar]

Thank you @refset for the update. I will explore https://github.com/lilactown/pyramid.