DNM: Try enabling RTR by default

[def-]

This is a test-once PR for https://github.com/MaterializeInc/materialize/pull/25195.

See also: https://github.com/MaterializeInc/materialize/pull/25566

[def-]

@sploiselle I was trying to run all the existing tests with RTR for Kafka enabled, but most tests then fail with ERROR: real-time recency unavailable for some sources in query. Is it possible to easily enable RTR for Kafka sources, but keep it off for other sources?

[nrainer-materialize]

According to the results of the nightly build, enabling RTR everywhere causes issues due to real-time recency unavailable for some sources in query.

[nrainer-materialize]

[...] Is it possible to easily enable RTR for Kafka sources, but keep it off for other sources?

@sploiselle

[sploiselle]

@nrainer-materialize That build is from Thu 22nd Feb; the code has been substantially refactored since then. We should no longer return real-time recency unavailable for some sources in query.

[nrainer-materialize]

@nrainer-materialize That build is from Thu 22nd Feb; the code has been substantially refactored since then. We should no longer return real-time recency unavailable for some sources in query.

Perfect, let me rebase it in this case!

[nrainer-materialize]

Nightly: https://buildkite.com/materialize/nightly/builds/7701

[nrainer-materialize]

@sploiselle: Do we expect the startup to be slower with this property enabled?

NAME                                | TYPE      |      THIS       |      OTHER      |  Regression?  | 'THIS' is:
----------------------------------------------------------------------------------------------------
StartupLoaded                       | wallclock |           6.753 |           1.846 |    !!YES!!    | 3.7 TIMES more/slower
StartupLoaded                       | messages  |       10156.000 |        6587.000 |    !!YES!!    | 54.2 pct   more/slower
StartupLoaded                       | memory    |        3936.768 |        3815.651 |      no       | 3.2 pct   more/slower

[sploiselle]

Wow––that's intense! We do additional work to determine if the collection we're querying is a source that might need RTR, but I am surprised by how much slower this is. Will start poking at this and see if I can repro and fix locally.

[nrainer-materialize]

Test verdict summarized in https://github.com/MaterializeInc/materialize/pull/25195#issuecomment-2102846136.

[def-]

Since RTR is now merged, doing another test run.

[nrainer-materialize]

I did another run in https://buildkite.com/materialize/nightly/builds/7905 recently.

See also: https://www.notion.so/Real-time-recency-88457f51e6384a8c94a51e3edd889b68?d=30ff43bbdafe40a4a7903b424e9cc76c&pvs=4#62026e326fc345409cff3f84078bb817