xtdb

xtdb copied to clipboard

Based on a report from @jacobobryant via https://discuss.xtdb.com/t/where-clause-not-working/34

This should never cause an OOM:

(with-open [l (xt/open-tx-log (xtdb-node) nil true)]
  (while (.hasNext l)
    (.next l)))

This is using Postgres JDBC for the tx-log.

A SNAPSHOT release with a change to the fetch-size parameter was created (see 247dda0c62ff7b4f1ac534ae687e711976ddd5c7) but does not seem to have improved the situation.

Related: https://github.com/xtdb/xtdb/discussions/1163

@jacobobryant does this print out anything successfully?

(def c (atom 0))
(with-open [l (xt/open-tx-log (xtdb-node) nil true)]
  (prn :next 0)
  (while (.hasNext l)
    (when (zero? (mod (swap! c inc) 1000))
      (prn :next @c))
    (.next l)))

There are one of two possibilities: (1) the open-tx-log call is fully materializing before returning (which this tests for), or (2) the consumed results are being retained somewhere.

It does print. I modified the code segment to be more verbose:

(do
  (prn (java.util.Date.))
  (with-open [l (xt/open-tx-log (xtdb-node) nil true)]
    (prn :next 0 (java.util.Date.))
    (while (.hasNext l)
      (prn :next (swap! c inc) (java.util.Date.))
      (.next l))))

See xtdb-oom-output.txt. open-tx-log takes about 30 seconds to return, then the loop proceeds at several iterations per second until iteration 355, at which point it slows to 1 iteration per 1-2 seconds, then at iteration 360 it hung for 10-15 seconds and then OOMed.

This is a behaviour of the postgres jdbc driver: https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor

It is tricky to get it to enable cursor support.

MySQL's connector also fully materializes result sets by default https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-implementation-notes.html

Moved notes to PR #1815