[Bug] RSS: Japanese video description encoding issues

[CatPlanet]

Describe the bug Despite RSS channels' output being in UTF8 japanese video descriptions are total gibberish

Steps to Reproduce Steps to reproduce the behavior:

1. Go to https://yewtu.be/feed/channel/UCcp9uRaBwInxl_SZqGRksDA
2. Cry

[unixfox]

Can you replicate the issue locally with a local instance of invidious?

[syeopite]

Seems like a behavior specific to yewtu.be. I can't replicate this locally

[RadoslavL]

Seems like a behavior specific to yewtu.be. I can't replicate this locally

I can also replicate it in invidious.fdn.fr and inv.tux.pizza.

[absidue]

I wonder if there is something between Invidious and the user on those major instances that is messing up the encoding, maybe a reverse proxy or something that is incorrectly configured?

[unixfox]

@syeopite did you try if you can replicate the issue in docker too?

[CatPlanet]

Chinese channels are affected too https://yewtu.be/feed/channel/UCoC47do520os_4DBMEFGg4A Taiwan https://yewtu.be/feed/channel/UC8ROUUjHzEQm-ndb69CX8Ww Korean https://yewtu.be/feed/channel/UC5BMQOsAB8hKUyHu9KI6yig Vietnam https://yewtu.be/feed/channel/UCOYdjTcOmle5oItjpEK0VpQ Arabic noodles https://yewtu.be/feed/channel/UCfiwzLy-8yKzIbsmZTzxDgw

Titles, names are formatted correctly, same with actual video pages (not RSS ones). It's only the RSS description issue.

[syeopite]

Invidious under docker is still able to render the RSS feeds correctly for me. But I am able to replicate this on public instances like onion.tube though.

iv.ggtyler.dev seems to be able to render the RSS feeds correctly

[absidue]

Does it work correctly on the test instance? afaik that is a default setup without any of the custom/non-standard stuff that most public instances do.

[RadoslavL]

Does it work correctly on the test instance? afaik that is a default setup without any of the custom/non-standard stuff that most public instances do.

/feed/channel/UCoC47do520os_4DBMEFGg4A doesn't work properly on the test instance. /feed/channel/UC8ROUUjHzEQm-ndb69CX8Ww doesn't work as well.

[absidue]

Maybe it's caused by the reverse proxy then?

[sefidel]

Can replicate this issue (NixOS 24.05), it's also happening on /api/v1/comments.

I've observed the same behaviour without reverse proxy (localhost also suffers from this problem). Perhaps it's something with the build system?

Using the docker image works fine.

[sefidel]

This is most likely caused by an outdated version of Crystal.

See https://github.com/crystal-lang/crystal/issues/13703 for more details. (Relevant Comment)

[kubo6472]

Possibly related, I found that the instance is okay, but the API is apparently the issue: lamarios/clipious#491

[sefidel]

The web interface doesn’t suffer from this problem since it uses the ‘contentHtml’ field, which doesn’t go through the XML parser.

In other words, I’m 99% certain that it’s related to this issue (which the instance admin should fix)

[iGerman00]

I'm able to reproduce this issue locally, behind Caddy as the reverse proxy, and through Tailscale - I'm pretty sure it's coming out of Invidious that way. I use the Docker imagequay.io/invidious/invidious:latest-arm64 as the way to run Invidious. It also happens to anything /api/. Is there any specific fix? What should the instance admin fix exactly?

[sefidel]

@iGerman00 Is there any specific fix? What should the instance admin fix exactly?

Compiling invidious with Crystal >=1.10.0 should solve the issue. If you want to keep using that container image, I suggest opening a new issue/PR to bump the crystal version on the Dockerfile. (looks like the image uses 1.8.2)

[syeopite]

The crystal version for the docker images will get bumped once https://github.com/iv-org/invidious/pull/4654 is merged in the next batch.

[syeopite]

#4654 has been merged and as such this should be fixed.