I can't get federation to work (request timeouts)

[ChappIO]

I know I am supposed to post this to [email protected] but I can't seem to log on to my new account. Feel free to delete if no exceptions can be made.

I just set up my instance at https://waveform.social but I can't seem to access it from any other instance even though I have federation enable with no allow/black lists (I assume that's how open federation works right?)

1. When I visit https://lemmy.ml/c/[email protected] I get a message that the community was not found.
2. When I visit search for Lemmy.ml on my own instance I get operation timeouts: https://waveform.social/search/q/https%3A%2F%2Flemmy.ml%2Fc%2Ftest/type/All/sort/TopAll/listing_type/All/community_id/0/creator_id/0/page/1

However, if I curl to that instance from the server I have no issues at all:

curl -X HEAD -v https://lemmy.ml/c/test
# Returns status 200

Logs:

lemmy-lemmy-1     | 2023-06-11T09:20:02.528609Z ERROR HTTP request{http.method=GET http.scheme="https" http.host=waveform.social http.target=/api/v3/ws otel.kind="server" request_id=8caa4d3d-1d45-4519-9775-c2b3df86e157 http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: Request error: error sending request for url (https://lemmy.ml/c/test): operation timed out
lemmy-lemmy-1     |    0: lemmy_apub::fetcher::search::search_query_to_object_id
lemmy-lemmy-1     |              at crates/apub/src/fetcher/search.rs:17
lemmy-lemmy-1     |    1: lemmy_apub::api::resolve_object::perform
lemmy-lemmy-1     |            with self=ResolveObject { q: "https://lemmy.ml/c/test", auth: Some(Sensitive) }
lemmy-lemmy-1     |              at crates/apub/src/api/resolve_object.rs:21
lemmy-lemmy-1     |    2: lemmy_server::root_span_builder::HTTP request
lemmy-lemmy-1     |            with http.method=GET http.scheme="https" http.host=waveform.social http.target=/api/v3/ws otel.kind="server" request_id=8caa4d3d-1d45-4519-9775-c2b3df86e157 http.status_code=101 otel.status_code="OK"
lemmy-lemmy-1     |              at src/root_span_builder.rs:16
lemmy-lemmy-1     | 2023-06-11T09:20:04.150032Z  WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: couldnt_find_object: Request error: error sending request for url (https://lemmy.ml/c/test): operation timed out
lemmy-lemmy-1     |    0: lemmy_apub::fetcher::search::search_query_to_object_id
lemmy-lemmy-1     |              at crates/apub/src/fetcher/search.rs:17
lemmy-lemmy-1     |    1: lemmy_apub::api::resolve_object::perform
lemmy-lemmy-1     |            with self=ResolveObject { q: "https://lemmy.ml/c/test", auth: Some(Sensitive) }
lemmy-lemmy-1     |              at crates/apub/src/api/resolve_object.rs:21
lemmy-lemmy-1     |    2: lemmy_server::root_span_builder::HTTP request
lemmy-lemmy-1     |            with http.method=GET http.scheme="http" http.host=waveform.social http.target=/api/v3/resolve_object otel.kind="server" request_id=e614dfd0-35e4-4887-baf3-60d3531f1c2a http.status_code=400 otel.status_code="OK"
lemmy-lemmy-1     |              at src/root_span_builder.rs:16
lemmy-lemmy-1     | LemmyError { message: Some("couldnt_find_object"), inner: Request error: error sending request for url (https://lemmy.ml/c/test): operation timed out

[ChappIO]

I think I may have figured something out: I couldn't curl from the Lemmy container. Turned out I had no internet connection. Well makes sense because the default docker setup places Lemmy in an internal network. Meaning it cannot access other instances.

Did I understand this correctly?

[scottwallacesh]

That would be my guess. I'm running a brand-new installation, in Docker, and it federates correctly. Now... if we could just get #2608 resolved, I'd be very happy.

[ImLunaHey]

Same issue here, also using the docker setup.

[kigero]

I think I may have figured something out: I couldn't curl from the Lemmy container. Turned out I had no internet connection. Well makes sense because the default docker setup places Lemmy in an internal network. Meaning it cannot access other instances.

Ran into the same thing - fixed it by adding the lemmyexternalproxy network to the lemmy service in the docker compose file. Not sure if that was left off accidentally or intentionally, but it sure seems like that service needs outbound access.

[guffelman]

I think I may have figured something out: I couldn't curl from the Lemmy container. Turned out I had no internet connection. Well makes sense because the default docker setup places Lemmy in an internal network. Meaning it cannot access other instances.

Ran into the same thing - fixed it by adding the lemmyexternalproxy network to the lemmy service in the docker compose file. Not sure if that was left off accidentally or intentionally, but it sure seems like that service needs outbound access.

Added lemmyexternalproxy to my lemmy service, ran sudo docker-compose up -d --force-recreate, no change. Federation is enabled to my knowledge (checked in the admin panel) and I still cannot access other communities. It is worth noting I have a NPM proxy in front of the NGINX proxy... not sure if that is the cause of my error.

[andrewgioia]

Same issue here as well, adding the networks to various containers did not resolve this.

I have this issue with just a basic NGINX proxy, following the standard docker install docs.

[ChappIO]

What I did is remove the network separation:

version: "3.3"

services:
  proxy:
    image: nginx:1-alpine
    ports:
      # only ports facing any connection from outside
      - 80:80 
      - 443:443
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      # setup your certbot and letsencrypt config 
      - ./certbot:/var/www/certbot
      - ./letsencrypt:/etc/letsencrypt/live
    restart: always
    depends_on:
      - pictrs
      - lemmy-ui

  lemmy:
    image: dessalines/lemmy:0.17.3
    hostname: lemmy
    restart: always
    environment:
      - RUST_LOG="warn,lemmy_server=info,lemmy_api=info,lemmy_api_common=info,lemmy_api_crud=info,lemmy_apub=info,lemmy_db_schema=info,lemmy_db_views=info,lemmy_db_views_actor=info,lemmy_db_views_moderator=info,lemmy_routes=info,lemmy_utils=info,lemmy_websocket=info"
    volumes:
      - ./lemmy.hjson:/config/config.hjson
    depends_on:
      - postgres
      - pictrs

  lemmy-ui:
    image: dessalines/lemmy-ui:0.17.3
    environment:
      # this needs to match the hostname defined in the lemmy service
      - LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy:8536
      # set the outside hostname here
      - LEMMY_UI_LEMMY_EXTERNAL_HOST=localhost:1236
      - LEMMY_HTTPS=true
    depends_on:
      - lemmy
    restart: always

  pictrs:
    image: asonix/pictrs:0.3.1
    # this needs to match the pictrs url in lemmy.hjson
    hostname: pictrs
    # we can set options to pictrs like this, here we set max. image size and forced format for conversion
    # entrypoint: /sbin/tini -- /usr/local/bin/pict-rs -p /mnt -m 4 --image-format webp
    environment:
      - PICTRS__API_KEY=<SNIP>
    user: 991:991
    volumes:
      - ./volumes/pictrs:/mnt
    restart: always

  postgres:
    image: postgres:15-alpine
    # this needs to match the database host in lemmy.hson
    hostname: postgres
    environment:
      - POSTGRES_USER=lemmy
      - POSTGRES_PASSWORD=<SNIP>
      - POSTGRES_DB=lemmy
    volumes:
      - ./volumes/postgres:/var/lib/postgresql/data
    restart: always

[ImLunaHey]

https://github.com/LemmyNet/lemmy/issues/3019#issuecomment-1586547397

did exactly this and its all working now.

[andrewgioia]

Thanks @ChappIO I appreciate you sharing that. I suppose there are other issues then with my setup as I still cannot get federation working in-app, even with this docker file. The federation tests work fine via CLI/curl but search in-app never finds anything and I can't find my server via other Lemmy instances or Mastodon.

[ChappIO]

@andrewgioia try this nginx.conf:

events {
    worker_connections 1024;
}
http {
    upstream lemmy {
        # this needs to map to the lemmy (server) docker service hostname
        server "lemmy:8536";
    }
    upstream lemmy-ui {
        # this needs to map to the lemmy-ui docker service hostname
        server "lemmy-ui:1234";
    }

    server {
        # this is the port inside docker, not the public one yet
        listen 80;
        # change if needed, this is facing the public web
        server_name localhost;
        server_tokens off;

        gzip on;
        gzip_types text/css application/javascript image/svg+xml;
        gzip_vary on;

        # Upload limit, relevant for pictrs
        client_max_body_size 20M;

        add_header X-Frame-Options SAMEORIGIN;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";

        # frontend general requests
        location / {
            # distinguish between ui requests and backend
            # don't change lemmy-ui or lemmy here, they refer to the upstream definitions on top
            set $proxpass "http://lemmy-ui";

	    if ($http_accept ~ "^application/.*$") {
              set $proxpass "http://lemmy";
            }
            if ($request_method = POST) {
              set $proxpass "http://lemmy";
            }
            proxy_pass $proxpass;

            rewrite ^(.+)/+$ $1 permanent;
            # Send actual client IP upstream
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

        # backend
        location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) {
            proxy_pass "http://lemmy";
            # proxy common stuff
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";

            # Send actual client IP upstream
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

The key part is here:

           if ($http_accept ~ "^application/.*$") {
            set $proxpass "http://lemmy";
          }

The original is statement checks for exact matches of the Accept header which is not how the header works.

[andrewgioia]

@ChappIO that works! Thank you again, you saved a huge amount of time and frustration for me.

[ChappIO]

Thank @[email protected] we spent quite some time troubleshooting: https://waveform.social/post/442

[Nutomic]

https://github.com/LemmyNet/lemmy-docs/pull/209