Server error after updating to 0.18.0

[slash-zero]

Requirements

• [X] This is a bug report, and if not, please post to https://lemmy.ml/c/lemmy_support instead.
• [X] Please check to see if this issue already exists.
• [X] It's a single bug. Do not report multiple bugs in one issue.
• [X] It's a frontend issue, not a backend issue; Otherwise please create an issue on the backend repo instead.

Summary

Docker setup. After updating to the 0.18.0 images for both lemmy-ui and lemmy backend, the lemmy-ui logs show a pictrs error, and the main site returns a "Server error".

FetchError: request to https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp failed, reason: connect ECONNREFUSED 127.0.1.1:443
    at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (node:events:511:28)
    at TLSSocket.socketErrorListener (node:_http_client:495:9)
    at TLSSocket.emit (node:events:511:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED'
}

Steps to Reproduce

1. Update docker-compose.yml to 0.18.0
2. docker-compose down
3. docker-compose up
4. access site

Technical Details

Docker setup. After updating to the 0.18.0 images for both lemmy-ui and lemmy backend, the lemmy-ui logs show a pictrs error, and the main site returns a "Server error".

FetchError: request to https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp failed, reason: connect ECONNREFUSED 127.0.1.1:443
    at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
    at ClientRequest.emit (node:events:511:28)
    at TLSSocket.socketErrorListener (node:_http_client:495:9)
    at TLSSocket.emit (node:events:511:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED'
}

Lemmy Instance Version

0.18.0

Lemmy Instance URL

No response

[slash-zero]

Note: pictrs logs show no errors.

When trying to access the image directly at https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp, it works fine in a browser.

The issue appears to be that lemmy-ui is trying to load the image from 127.0.1.1:443.

ECONNREFUSED 127.0.1.1:443

[ubergeek77]

UPDATE:

Issue "fixed" by putting the lemmy-ui container on a network with access to the internet. It looks like lemmy-ui didn't previously need an internet connection on 0.17.x, but now it needs one. Probably because it used to use the internal docker network to get to pictrs, but now it's trying to resolve using the real public hostname?

In OP's case, it's going to a local IP instead, but it's using the wrong port. In both our cases, the URL it uses for pictrs is not calculated correctly for whatever reason.

The UI could fail to contact pictrs for any variety of reasons, but that shouldn't be fatal. It should just break images.

Previous comment text:

---

I get a very similar error but it's not quite the same:

 FetchError: request to https://<my-instance>/pictrs/image/<uuid>.png failed, reason: getaddrinfo EAI_AGAIN <my-instance>
     at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
     at ClientRequest.emit (node:events:511:28)
     at TLSSocket.socketErrorListener (node:_http_client:495:9)
     at TLSSocket.emit (node:events:511:28)
     at emitErrorNT (node:internal/streams/destroy:151:8)
     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
   type: 'system',
   errno: 'EAI_AGAIN',
   code: 'EAI_AGAIN'
 }

Just like OP, I can go to the URL that error is logging and it pulls the image just fine. No browser cache shenanigans, the request is being fully completed.

My server is now completely inoperable and I can't downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?

[Hellhound1]

Put pictrs and lemmy-ui on the external network if you used the ansible script. That should have been updated now though.

https://github.com/LemmyNet/lemmy-ansible/issues/96#event-9619766061

have a look here - try following the steps to remove the site icon via the database

[slash-zero]

Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB.

That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs.

[slash-zero]

My server is now completely inoperable and I can't downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?

@ubergeek77

short term fix is to remove your site’s image icon.

db query: update site set icon = null where id = 1;

then at least your site will work again, sort of. It loads at least. I am not sure federation is working 100% though and the pictrs errors still persist in the logs.

[ubergeek77]

Thanks! I just left that there for posterity, I added an edit to the top of my comment about how I "fixed" this.

It was caused due to a network error, so to fix it I just gave the container internet access, even though it otherwise wouldn't need it. That fixed things immediately.

Still, network errors shouldn't be fatal. Especially not for an icon.

[madeofstown]

My server is now completely inoperable and I can't downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?

@ubergeek77

short term fix is to remove your site’s image icon.

db query: update site set icon = null where id = 1;

then at least your site will work again, sort of. It loads at least. I am not sure federation is working 100% though and the pictrs errors still persist in the logs.

For an instance set up with Ansible: Connect with ssh and cd /srv/lemmy/<domain>. Run docker-compose down and look for the name of your postgres container (should be something like lemmyml_postgres_1) as it shuts down. Run docker-compose up -d and wait for all the containers to start back up. Run docker exec -it <postgres-container> "bash" to connect to your postgres database container. Run psql -U lemmy to connect to the database with the lemmy user. Run update site set icon = null where id = 1; then \q then exit

[madeofstown]

I can confirm that this issue is still present in Lemmy-ui 0.18.1-rc.2. Every single time the docker images restart I have to run the commands to delete my site logo or the site will be unreachable. I think this may be partially because my external NGINX proxy is on a different different host. When Lemmy-ui resolves the IP for my domain it is still using the local IP and port 443.

If believe the fix would be to use the port specified by the pictures server in the internal NGINX configuration. I'm not aware of how to do this. I will test if this can be accomplished by configuring the pictures server to listen on 443 in the internal configuration.

[madeofstown]

Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB.

That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs.

Is your "external" NGINX proxy running on the same host server as Lemmy?

[slash-zero]

I’m using the docker containers. No additional nginx running on the host.

[ubergeek77]

I think this is happening because there is not any error handling when the icon is fetched:

https://github.com/LemmyNet/lemmy-ui/blob/9e7fec772d873e7bc46a51403d9569f70619c9b8/src/server/utils/create-ssr-html.tsx#L28

https://github.com/LemmyNet/lemmy-ui/blob/main/src/server/utils/fetch-icon-png.ts

So if that call fails for any reason, it breaks the SSR render.

I think this should be changed to return a broken image icon, or a blank image, instead of aborting the entire SSR process. Otherwise, the pictrs service going down for any reason causes the entire UI to crash like this.

That goes for any other fetches for external data too. If it's not mission critical (and images aren't mission critical), the SSR would ideally continue the render.

[madeofstown]

I found a more permanent fix. The error seems to be caused by having the host name of your server set to the same thing as your lemmy instance domain-name. The easy and more permanent fix is to change the host name of your server!

[slash-zero]

@randommouse interesting. I’ll give that a try.

[madeofstown]

@slash-zero "It's always DNS"

[slash-zero]

@randommouse:

OK, I updated to:

• lemmy-ui: 0.18.1-rc.7
• lemmy be: 0.18.1-rc.4

Then, I changed my host's server name to not match the domain name, and rebooted. Local images were still not working after a docker-compose down and docker-compose up -d --force-rebuild. I still couldn't upload images either, and the site banner was not working (broken image).

The final step was to revert the lemmy.hjson back to http://pictrs:8080 (from the public domain name) and restart again, and now images are working, and having an icon for my instance no longer breaks the site.

So, to clarify, what I did was:

1. Change my host name in /etc/hostname so it no longer matches my site's domain name
2. Update lemmy.hjson to set pictrs back to the original http://pictrs:8080
3. Reboot, because why not?
4. "Restart" all lemmy containers via down, then up
5. 👍

[betodealmeida]

I found a more permanent fix. The error seems to be caused by having the host name of your server set to the same thing as your lemmy instance domain-name. The easy and more permanent fix is to change the host name of your server!

Had the same problem when I upgraded to 0.18.0, this fixed it.

[slrgt]

@randommouse:

OK, I updated to:

* lemmy-ui: 0.18.1-rc.7

* lemmy be: 0.18.1-rc.4

Then, I changed my host's server name to not match the domain name, and rebooted. Local images were still not working after a docker-compose down and docker-compose up -d --force-rebuild. I still couldn't upload images either, and the site banner was not working (broken image).

The final step was to revert the lemmy.hjson back to http://pictrs:8080 (from the public domain name) and restart again, and now images are working, and having an icon for my instance no longer breaks the site.

So, to clarify, what I did was:

1. Change my host name in `/etc/hostname` so it no longer matches my site's domain name

2. Update lemmy.hjson to set pictrs back to the original http://pictrs:8080

3. Reboot, because why not?

4. "Restart" all lemmy containers via down, then up

5. 👍

can you write the instructions to change the hostname?

[slash-zero]

can you write the instructions to change the hostname?

sudo nano /etc/hostname Change the host name in that file

sudo reboot

[ubergeek77]

You'll have to change it in /etc/hosts too most likely, unless your distro does that automatically somehow.

I would still like to see this issue changed to track the lack of error handling here :p

And probably the fact that lemmy-ui is trying to SSR by polling the real public hostname instead of the internal hostname. It didn't used to do that in 0.17.3.

[mvdkleijn]

@madeofstown I was getting the error on my server as well during its shakedown tests. The hostname is definitely not the same as the domain name.

The behaviour I'm seeing:

• Start containers
• Add icon through UI
• Test everything to see it works fine
• Check DB, icon URL points correctly to exterior domain name
• Restart containers
• Site is borked
• Enter DB, check icon URL. Still fine
• Delete icon entry from DB
• Site works (no restart needed)
• Re-add icon through UI
• Verify site to see it still works. It does (still no restart)

My conclusion: the pass through to Pictrs that Lemmy is doing, is not working properly somehow and becomes borked after a restart.

(in case that wasn't clear, my pictrs is not exposed to the internet directly and Lemmy is providing a /pictrs endpoint which is used to access pictrs as the developers intended)

I will investigate a little more to see what else I can find out.

[slrgt]

Check that your lemmy.hjson has the new pictrs format instead of the old pictrs_url.

I had to update my config file to the new pictrs format as well because everytime i updated via the ansible script it was using the config file to revert the lemmy.hjson to the old pictrs_url format causing the Server Error

On Thu, Jul 13, 2023 at 8:46 AM, Martijn @.***(mailto:On Thu, Jul 13, 2023 at 8:46 AM, Martijn < wrote:

@.***(https://github.com/madeofstown) I was getting the error on my server as well during its shakedown tests. The hostname is definitely not the same as the domain name.

The behaviour I'm seeing:

• Start containers
• Add icon through UI
• Test everything to see it works fine
• Check DB, icon URL points correctly to exterior domain name
• Restart containers
• Site is borked
• Enter DB, check icon URL. Still fine
• Delete icon entry from DB
• Site works (no restart needed)
• Re-add icon through UI
• Verify site to see it still works. It does (still no restart)

My conclusion: the pass through to Pictrs that Lemmy is doing, is not working properly somehow and becomes borked after a restart.

(in case that wasn't clear, my pictrs is not exposed to the internet directly and Lemmy is providing a /pictrs endpoint which is used to access pictrs as the developers intended)

I will investigate a little more to see what else I can find out.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[mvdkleijn]

@slrgt no, that's set correctly. It doesn't matter whether you do a whole docker compose down && docker compose up -d or simply do a docker compose restart.

The behavior remains the same. Lemmy suddenly tries to use the external IP and port 443. That causes the entire service to be unreachable due to lack of basic error handling in Lemmy UI.

Simply removing the site icon URL from the DB corrects that behavior. The site then immediately becomes available again without the need to restart or anything.

edit: just to be clear... re-adding the site icon doesn't break the site. The issue doesn't reappear until after the containernis restarted.

[madeofstown]

@mvdkleijn

The behavior remains the same. Lemmy suddenly tries to use the external IP and port 443.

In your docker config have you added the lemmyexternalproxy network to your pictrs settings? I ask because the issue others were having was that Lemmy was trying to use the internal IP. It should be using the external IP.

[ubergeek77]

It's a bit silly to have the UI call out to the internet, just to come back to the same server on the same IP. This might even break routing in some situations.

What should happen is, Lemmy-UI should have a config option to set the Pictrs URL, just like the Lemmy backend does. That way, in a Docker setup, it can connect directly to Pictrs via Docker internal routing. And in any other setup, it's configurable.

[madeofstown]

I agree that this is a bug. I am just trying to help people work around it. When I said:

It should be using the external IP.

I meant that is how things work in the current implementation.

[mvdkleijn]

@mvdkleijn

The behavior remains the same. Lemmy suddenly tries to use the external IP and port 443.

In your docker config have you added the lemmyexternalproxy network to your pictrs settings? I ask because the issue others were having was that Lemmy was trying to use the internal IP. It should be using the external IP.

Pictrs is not directly accessible through the internet. It is fronted by Traefik, which has no rules to forward traffic to pictrs. In other words:

browser -> traefik -> lemmy -> pictrs

Keep in mind that this setup works fine until a container restart. After the restart, simply removing and re-adding the site icon in the DB is enough to make it work again. (i.e. no need to restart again)

This leads me to believe the issue is not DNS or routing based but something inside Lemmy itself.

Putting pictrs on an external network temporarily "solves" the problem because it bypasses Lemmy.

I have seen quite a number of weird issues during shakedown testing and it has led me to believe that Lemmy is not production ready in any sense of the word, so I'll probably be evaluating the situation with Kbin.

[madeofstown]

This issue is back worse than before with 0.19. The previous solution of changing you server hostname no longer works and the database entry must be cleared manually. I cannot use lemmy-ui consistently but fortunately I've figured out how to run Photon on the base domain.