Server pagination

[dustingetz]

A good test case is this.

Make a new CLI, make new fiddle :a, copy the schema from http://mbrainz.hyperfiddle.net/:hyperfiddle.ide.schema!$/ and transact. Use this query. Remove the needle and this does not return on my machine. In prod it results in 15s timeouts.

[:find
 [(pull ?e 
        [:release/name 
         :release/gid
         {:release/artists 
          [:artist/gid
           :artist/name]
          :release/media 
          [{:medium/format [:db/ident]
            :medium/tracks 
            [:track/position
             :track/name
             :track/artistCredit
             {:track/artists 
              [:artist/gid
               :artist/name
               {:artist/type [:db/ident]
                :artist/country [:db/ident]}
               *]}
             *]}
           *]}
         *])  
  ...]
 :where
 [?e :release/name "Osibisa"]
 ]

[dustingetz]

[dustingetz]

Gzipped. No content-length or anything like that obv because transit streams.

[dustingetz]

19-05-28 22:01:02 dmbp17-2.local WARN [hyperfiddle.io.datomic.hydrate-route:77] - Slow request function 493979ms :: route: [:a []] 19-05-28 22:14:21 dmbp17-2.local WARN [hyperfiddle.io.datomic.hydrate-route:77] - Slow request function 444817ms :: route: [:a []]

[dustingetz]

Oh this time Hyperfiddle came back and rendered it (because i let it sit and left desk until computer spun down), 572 pages. Changing from tab 1 to 2 is about 30 seconds presumably due to cardinality many forms in tables. They aren't really that large (see scrollbar). The largest nested set was 2 pages (< 40 items)

[khardenstine]

Some crude profiling of the above query on the jvm (11k rows), all times in milliseconds:

:ns	:name	:count	:sum	:average	:max	:min
hypercrud.browser.browser-request	request-attr-level	594561	2346876	3.947	110	0
contrib.datomic	pull-traverse	7614958	1055008	0.139	13	0
hypercrud.browser.browser-request	request-from-route	1	876078	876078.000	876078	876078
hypercrud.browser.browser-request	requests	1	825927	825927.000	825927	825927
hypercrud.browser.browser-request	requests-here	594563	743360	1.250	23	0
hyperfiddle.data	select-many-here	594563	742478	1.249	23	0
hyperfiddle.data	select-many-here'	594563	741451	1.247	23	0
hyperfiddle.data	select-many	823240	567983	0.690	15	0
hypercrud.browser.context	links-in-dimension	823240	560176	0.680	15	0
hypercrud.browser.context	links-in-dimension-false	823238	557400	0.677	15	0
contrib.datomic	attr	36815075	372842	0.010	28	0
contrib.datomic	cardinality	31352645	358538	0.011	28	0
hypercrud.browser.context	reachable-attrs	823238	348279	0.423	13	0
hypercrud.browser.context	reachable-pullpaths	823238	347120	0.422	13	0
contrib.datomic	cardinality?	26595151	339033	0.013	12	0
hypercrud.browser.context	attribute	1497836	157445	0.105	13	0
contrib.datomic	tree-derivative	514497	129367	0.251	69	0
hypercrud.browser.context	identity?	1509272	59209	0.039	10	0
contrib.datomic	pull-union	1108822	54313	0.049	39831	0
hypercrud.browser.base	data-from-route	1	50149	50149.000	50149	50149
hypercrud.browser.context	v!	1360629	49970	0.037	9	0
hypercrud.browser.context	result	1	41533	41533.000	41533	41533
hypercrud.browser.context	result-enclosure!	1	39947	39947.000	39947	39947
contrib.datomic	result-enclosure!	1	39947	39947.000	39947	39947
contrib.datomic	pull-enclosure	1	39945	39945.000	39945	39945
hypercrud.browser.context	index-result	1497837	36639	0.024	1586	0
hypercrud.browser.context	data	1360629	35403	0.026	9	0
contrib.datomic	ref?	2198963	31842	0.014	27	0
contrib.datomic	valueType?	2198963	29365	0.013	27	0
contrib.datomic	valueType	2198963	26831	0.012	27	0
hypercrud.browser.context	attr?	1189124	21094	0.018	9	0
contrib.datomic	attr?	1189146	18994	0.016	9	0
hypercrud.browser.context	attr	1509272	18771	0.012	10	0
hypercrud.browser.context	eav	2286776	17172	0.008	9	0
hypercrud.browser.context	children	1600740	16818	0.011	10	0
hypercrud.browser.context	entity-datakey	246307	15010	0.061	13	0
hypercrud.browser.context	smart-entity-identifier	246307	14715	0.060	13	0
hypercrud.browser.context	spread-attributes	777503	13262	0.017	13	0
hypercrud.browser.context	pull-enclosure-here	1600740	11680	0.007	9	0
contrib.datomic	smart-lookup-ref-no-tempids	64870113	9919	0.000	7	0
hypercrud.browser.context	links-at	19986384	9553	0.000	12	0
contrib.datomic	pullshape-get-in	1600740	9373	0.006	9	0
hypercrud.browser.context	lookup-ref	246307	8895	0.036	10	0
hypercrud.browser.context	row-key	68598	8642	0.126	43	0
hypercrud.browser.context	ctx->id-lookup	812761	8612	0.011	7	0
contrib.datomic	find-identity-attr	246307	8595	0.035	10	0
hypercrud.browser.context	a	777505	7338	0.009	8	0
contrib.datomic	unique?	565036	7099	0.013	9	0
contrib.datomic	unique	565036	6417	0.011	9	0
hypercrud.browser.context	entity-key-v	97011	6318	0.065	13	0
hypercrud.browser.context	underlying-tempid	406381	5999	0.015	8	0
hypercrud.browser.context	scalar-key	45732	4784	0.105	10	0
contrib.datomic	pullshape-get	3704570	4465	0.001	9	0
hypercrud.browser.context	-infer-implicit-element	3647400	3126	0.001	7	0
hypercrud.browser.context	row-key-v	22866	2952	0.129	11	0
hypercrud.browser.context	qfind-level?	1406361	2158	0.002	8	0
hypercrud.browser.context	row	11433	2104	0.184	7	0
hypercrud.browser.context	browse-element	22867	2070	0.091	7	0
contrib.datomic	pull-level	1600740	1915	0.001	10	0
hypercrud.browser.context	pull-depth	3315665	1091	0.000	1	0
hypercrud.browser.context	stable-eav-av	1349197	906	0.001	1	0
hypercrud.browser.context	set-parent	1497836	890	0.001	1	0
hypercrud.browser.context	element-head	45733	707	0.015	1	0
contrib.datomic	parser-type	788938	455	0.001	1	0
hypercrud.browser.context	links-in-dimension-pred	823240	366	0.000	1	0
hypercrud.browser.context	dbname	406381	209	0.001	1	0
hypercrud.browser.context	unwind	182942	200	0.001	1	0
contrib.datomic	tempid?	406385	76	0.000	1	0
hypercrud.browser.context	stable-eav-a	148641	70	0.000	1	0
hypercrud.browser.context	spread-elements	11434	46	0.004	1	0
hypercrud.browser.context	stable-element-schema!	45733	21	0.000	1	0
hypercrud.browser.context	fiddle	11436	20	0.002	1	0
hypercrud.browser.context	depth	11437	6	0.001	1	0
contrib.datomic	pull-shape	16	5	0.313	1	0
hypercrud.browser.context	stable-eav-v	11434	4	0.000	1	0
contrib.datomic	attr-spec->shape	54	4	0.074	1	0
hypercrud.browser.context	fiddle+	1	2	2.000	2	2
hypercrud.browser.base	hydrate-fiddle	1	1	1.000	1	1
hypercrud.browser.base	request-for-fiddle	1	1	1.000	1	1
hypercrud.browser.context	map->Context	1	0	0.000	0	0
contrib.datomic	validate-element	1	0	0.000	0	0
contrib.datomic	validate-qfind-attrs!	1	0	0.000	0	0
contrib.datomic	spread-elements	1	0	0.000	0	0
hypercrud.browser.base	legacy-fiddle-ident->lookup-ref	1	0	0.000	0	0
hypercrud.browser.base	meta-request-for-fiddle	1	0	0.000	0	0
hypercrud.browser.base	validate-fiddle	1	0	0.000	0	0
contrib.datomic	validate-qfind-attrs+	1	0	0.000	0	0
contrib.datomic	pp-recursion?	42	0	0.000	0	0
hypercrud.browser.context	tempid->id+	1	0	0.000	0	0
contrib.datomic	spread-elements!	1	0	0.000	0	0
hypercrud.browser.context	spread-result	1	0	0.000	0	0
contrib.datomic	normalize-result	2	0	0.000	0	0
hypercrud.browser.context	links-in-dimension-true	2	0	0.000	0	0
hypercrud.browser.context	spread-rows	1	0	0.000	0	0
hypercrud.browser.context	validate-query-params+	1	0	0.000	0	0
contrib.datomic	pull-seq	2	0	0.000	0	0
contrib.datomic	indexed-schema	1	0	0.000	0	0
hypercrud.browser.context	clean	1	0	0.000	0	0
hypercrud.browser.q-util	parse-holes	1	0	0.000	0	0
hypercrud.browser.browser-request	cross-streams	1	0	0.000	0	0

[dustingetz]

(defn route-editor [ctx]
  [:<>
   [contrib.ui/cm-edn 
    {:value @(:hypercrud.browser/route ctx)
     :on-change
     (fn [next-route]
       (js/console.log @(:hypercrud.browser/route ctx) next-route)
       (hyperfiddle.runtime/dispatch!
        (:peer ctx) 
        (fn [dispatch! get-state]
          (hyperfiddle.actions/set-route (:peer ctx) next-route (:branch ctx) false dispatch! get-state)))
       )}]])