Messages do not send from my homeserver over federation

[gspbirel56]

I originally filed this as a bug report https://github.com/element-hq/synapse/issues/17067, but seeing that this may not be a bug, I have pasted the same details here:

Description

Hello! My homeserver setup has consisted of a main server running Synapse, an NFS machine for media_store, and a database server for PostgreSQL. Over the weekend, I attempted to upgrade my NFS machine and the worst case scenario happened: all of media_store is gone, and my most recent backup is from December.

I have since been able to set up the new drive, albeit without any data in media_store, but somehow this has resulted in issues with federation. On the matrix.org side, federation seems to be fine. However, I can break down the issue like so:

Things I have tried that do work:

• sending messages between users on my homeserver
• typing notifications between users on my homeserver and another (federated)
• receiving messages from other homeservers

Things I have tried that do not work:

• ping rooms from my homeserver (this may be unrelated)
• sending messages in 1:1 chats
• sending messages in group chats
• creating new 1:1 chats from my homeserver to others
• 1:1 calls with other homeserver users
• verifying others over federation

Steps to reproduce

I believe the description highlights the issue: this seems more holistic than a simple cause-effect bug Homeserver

chrrreeeeesss.com Synapse Version

1.104.0 Installation Method

pip (from PyPI) Database

PostgreSQL 13 single server, always on PostgreSQL, no backup restore Workers

Single process Platform

• Hyper-V: 6 CPUs, 8 GB RAM for host machine
• FreeBSD 14.0-RELEASE-p6 for guest machine
• I have tried running as sysrc service and via synctl, but the issue is the same

Configuration

No response Relevant log output

(...)/synapse/env/lib/python3.9/site-packages/synapse/handlers/federation_event.py", line 1189, in _compute_event_context_with_maybe_missing_prevs
    raise FederationError(
synapse.api.errors.FederationError: ERROR 403: We can't get valid state history.

Anything else that would be useful to know?

No response

[gspbirel56]

It may be irrelevant, but at one point I was also seeing the same error as in this ticket: https://github.com/matrix-org/synapse/issues/4727

[reivilibre]

Sorry for the late response.

It seems infeasible for your media_store vanishing to cause the problems you describe, so I'm scratching my head a bit.

I think your log is a bit slim to figure this out though.

However plugging your server name into the federation tester is not showing promising signs: https://federationtester.matrix.org/#chrrreeeeesss.com

https://chrrreeeeesss.com/.well-known/matrix/server seems to point to chrrreeeeesss.com:8008 but this doesn't seem to be accessible from the outside.

[gspbirel56]

Hi @reivilibre! The homeserver isn't showing on the federation tester because I took the service offline. I'll report back here later once I am able to fire it back up.

[anoadragon453]

Hi @gspbirel56, is this issue still relevant for investigation, or has the moment passed now and this can be closed?

[gspbirel56]

Hi @anoadragon453! We can close this issue. I can no longer reproduce it.