lazy-load-members `/sync` blocks if an event arrives in a partial-state-room

[richvdh]

lazy-load-members /sync should return new events immediately, without waiting for full state.

[squahtx]

We block waiting for full state of new events here:

2022-07-27 19:42:43,803 - synapse.storage.util.partial_state_events_tracker - 78 - INFO - GET-8 - Awaiting un-partial-stating of events ['$Z91x7vE6bB1EEdbdRR1gZOsYDk-I0tOayX11eEtKK6I']
Stack (most recent call last):
  [... lots of Twisted stack frames ...]
  File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1750, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
  File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/usr/local/lib/python3.9/site-packages/synapse/util/async_helpers.py", line 227, in _concurrently_execute_inner
    await maybe_awaitable(func(value))
  File "/usr/local/lib/python3.9/site-packages/synapse/handlers/sync.py", line 1562, in handle_room_entries
    await self._generate_room_entry(
  File "/usr/local/lib/python3.9/site-packages/synapse/handlers/sync.py", line 2072, in _generate_room_entry
    state = await self.compute_state_delta(
  File "/usr/local/lib/python3.9/site-packages/synapse/handlers/sync.py", line 1004, in compute_state_delta
    state_ids = await self._state_storage_controller.get_state_ids_for_event(
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/controllers/state.py", line 307, in get_state_ids_for_event
    state_map = await self.get_state_ids_for_events(
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/controllers/state.py", line 252, in get_state_ids_for_events
    event_to_groups = await self.get_state_group_for_events(
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/controllers/state.py", line 343, in get_state_group_for_events
    await self._partial_state_events_tracker.await_full_state(event_ids)
  File "/usr/local/lib/python3.9/site-packages/synapse/storage/util/partial_state_events_tracker.py", line 78, in await_full_state
    logger.info(

[squahtx]

We block waiting for full state of new events here:

This is only true for incremental lazy-loading /syncs. Initial and gappy /syncs also block inside compute_state_delta.

[squahtx]

Synapse:

• https://github.com/matrix-org/synapse/pull/13474
• https://github.com/matrix-org/synapse/pull/13477

Complement:

• https://github.com/matrix-org/complement/pull/440
• https://github.com/matrix-org/complement/pull/441
• https://github.com/matrix-org/complement/pull/442