[Bug] Missing Hash Key (Navigation Endpoint) after searching "ind bios"

[KeshaunRocks]

Describe The Issue Searching for "ind bios" crashes. I think this is the new bug on Invidious. Please review.

Steps To Reproduce Bug

1. Go to https://yewtu.be
2. Type "ind bios" and tap Enter
3. Tap on "Next page"
4. Error appears on: https://yewtu.be/search?q=ind+bios&page=2

Note: Sometimes it crashes, sometimes it doesn't.

Error Message Title: Missing hash key: "navigationEndpoint" (KeyError) Date: 2021-10-21T22:08:34Z Route: /search?q=ind+bios&page=2 Version: 2021.10.21-788605b @ master

Backtrace

Missing hash key: "navigationEndpoint" (KeyError)
  from /usr/lib/crystal/core/json/any.cr:100:0 in '[]'
  from /usr/lib/crystal/core/json/any.cr:102:10 in 'dig'
  from /usr/lib/crystal/core/json/any.cr:215:3 in 'get_browse_id'
  from src/invidious/yt_backend/extractors.cr:243:41 in 'process'
  from src/invidious/yt_backend/extractors.cr:551:17 in 'extract_item'
  from src/invidious/yt_backend/extractors.cr:582:28 in 'extract_items'
  from src/invidious/search.cr:36:11 in 'search'
  from src/invidious/search.cr:227:20 in 'search'
  from /usr/lib/crystal/core/primitives.cr:266:3 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/static_file_handler.cr:167:15 in 'call'
  from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/lib/crystal/core/http/server.cr:500:5 in '->'
  from /usr/lib/crystal/core/primitives.cr:266:3 in 'run'
  from /src/boringssl/src/google-boringssl-251b516/ssl/internal.h:334:8 in '???'

Additional Context Other instances I’ve tried work fine. The following instances also had the same error (this is not a complete list):

• invidious.kavin.rocks
• invidious.snopyta.org
• invidious-jp.kavin.rocks
• invidious.namazso.eu

Browser and OS

• Browser: Safari
• OS (if applicable): iOS

[KeshaunRocks]

Not sure if this bug would persist. I'll close the issue for now, but if it does persist, I'll reopen the issue.

[unixfox]

The bug is still here :)

[syeopite]

Hooray more edge cases 🎉!

The playlist doesn't have an author linked so it's likely auto generated. The two authors you see on the preview seems to actually be the artist for the video within. Anyhow in this case, the problem is that yungteenager has an channel (albeit auto-generated) while Light663 does not.

Invidious attempts to parse both of those as links via finding a navigationEndpoint, which doesn't exist in the author label for Light663:

Similar issues can likely arise for deleted channels as well.

---

We should be able to patch this pretty easily but the fact that Invidious handles these "nil" values by assigning an empty string isn't ideal. The way we serialize YT data needs to be revisited in the future.

[nonetrix]

I've gotten this too

Title: `Missing hash key: "navigationEndpoint" (KeyError)`
Date: `2021-10-25T21:24:31Z`
Route: `/search?q=Brodie+Roberson`
Version: `2021.10.25-9da1af5 @ master`
<details>
<summary>Backtrace</summary>
<p>
\`\`\`
Missing hash key: "navigationEndpoint" (KeyError)
  from /usr/lib/crystal/core/json/any.cr:100:0 in '[]'
  from /usr/lib/crystal/core/json/any.cr:102:10 in 'dig'
  from /usr/lib/crystal/core/json/any.cr:215:3 in 'get_browse_id'
  from src/invidious/yt_backend/extractors.cr:243:41 in 'process'
  from src/invidious/yt_backend/extractors.cr:551:17 in 'extract_item'
  from src/invidious/yt_backend/extractors.cr:582:28 in 'extract_items'
  from src/invidious/search.cr:36:11 in 'search'
  from src/invidious/search.cr:227:20 in 'search'
  from /usr/lib/crystal/core/primitives.cr:266:3 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/static_file_handler.cr:167:15 in 'call'
  from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/lib/crystal/core/http/server.cr:500:5 in '->'
  from /usr/lib/crystal/core/primitives.cr:266:3 in 'run'
  from /src/boringssl/src/google-boringssl-251b516/ssl/internal.h:334:8 in '???'
\`\`\`
</p>
</details>

[dragnev-dev]

Example video: Тони и Орлин Павлов: Докосвам те

Any of the keywords докосвам те, Орлин Павлов докосвам те, докосвам те Тони throw the error. Searching for dokosvam te works.

[Newbytee]

I get the same if I search for "crazy frog nightcore".

[lucic71]

The same issue for https://invidious.flokinet.to/search?q=manele+tanga

[jonas-klein]

The same issue for the search term selects vol. I noticed it by using the instance yt.cdaut.de, but it seems to throw an exception on all other instances as well.

Title: Missing hash key: "navigationEndpoint" (KeyError)Date:2023-10-30T13:53:15ZRoute:/search?q=selects+volVersion:2023.10.20-b809e877 @ master`

Backtrace

Missing hash key: "navigationEndpoint" (KeyError)
  from /usr/share/crystal/src/hash.cr:1077:11 in '[]'
  from /usr/share/crystal/src/json/any.cr:103:7 in '[]'
  from /usr/share/crystal/src/json/any.cr:139:5 in 'dig'
  from src/invidious/yt_backend/extractors.cr:827:12 in 'get_browse_id'
  from src/invidious/yt_backend/extractors.cr:332:41 in 'parse'
  from src/invidious/yt_backend/extractors.cr:319:16 in 'process'
  from src/invidious/yt_backend/extractors.cr:845:17 in 'parse_item'
  from src/invidious/yt_backend/extractors.cr:892:14 in 'extract_items'
  from src/invidious/search/processors.cr:12:18 in 'regular'
  from src/invidious/search/query.cr:116:9 in 'process'
  from src/invidious/routes/search.cr:55:17 in 'search'
  from src/invidious/routing.cr:184:20 in '->'
  from lib/kemal/src/kemal/route.cr:12:26 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'process_request'
  from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:212:5 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:145:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:70:5 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:162:16 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/logger.cr:17:35 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'
  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process'
  from /usr/share/crystal/src/http/server.cr:517:5 in 'handle_client'
  from /usr/share/crystal/src/http/server.cr:470:13 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from /usr/share/crystal/src/fiber.cr:98:34 in '->'
  from ???

`

Whereas the above mentioned search terms ind bios, Brodie Roberson, crazy frog nightcore works well for me.

[squeaktoy]

cdr thats cdr

Title: Missing hash key: "navigationEndpoint" (KeyError) Date: 2023-11-19T20:34:31Z Route: /search?q=cdr+thats+cdr Version: 2023.11.08-c5b87e3 @ master

Backtrace

Missing hash key: "navigationEndpoint" (KeyError)
  from /usr/share/crystal/src/hash.cr:1080:9 in '??'
  from /usr/share/crystal/src/json/any.cr:139:5 in 'get_browse_id'
  from src/invidious/yt_backend/extractors.cr:332:41 in 'process'
  from src/invidious/yt_backend/extractors.cr:845:17 in 'parse_item'
  from src/invidious/yt_backend/extractors.cr:892:14 in 'extract_items'
  from src/invidious/search/processors.cr:12:18 in 'process'
  from src/invidious/routes/search.cr:55:17 in 'search'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:151:29 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:106:14 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call'
  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'handle_client'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???

[AnirudhRevanur]

Get the same error when searching for the keyword ello jogappa

Have tried different hosts, but to no luck List of hosts tried:

1. yewtu.be
2. vid.priv.au
3. yt.cdaut.de

This is not a complete list, but there are more that give the same error

In the browser console get the error: 500 Internal Server Error

[haltosan]

/search?q=manele+tanga /search?q=selects+vol /search?q=cdr+thats+cdr /search?q=ello+jogappa /search?q=onstercat+solac

All of these look like they're broken (at least on https://inv.tux.pizza and like 10 random instances from redirect.invidious.io). I couldn't remove any more letters off of "onstercat solac" and still reproduce the error (initial string was "monstercat solace").

After looking a bit more, I think it has something to do with the results. For example "what does the fox say onstercat solac" doesn't break on inv.tux.pizza. But "monstercat soulace" will produce the error even though it is a misspelling, "monster cat solace" also produces an error, etc..

invidious.asir.dev actually worked for "manele tanga" invidious.perennialte.ch worked for "ello jogappa"

I was looking at #3812 too, but I wasn't able to reproduce any of those errors on inv.tux.pizza

Hope this helps

[krypciak]

This also happens to me on /search?q=crosscode+ultimate at https://yewtu.be https://invidious.baczek.me

[GTP95]

Just got this error by searching on two different instances for "incognito ceo". This search query used to work up to a few days ago. Here's the error message:

Title: `Missing hash key: "navigationEndpoint" (KeyError)`
Date: `2024-02-08T15:06:47Z`
Route: `/search?q=incognito+ceo`
Version: `2024.02.04-84e8aec7 @ master`

<details>
<summary>Backtrace</summary>
<p>
   
[removed backtics]
Missing hash key: "navigationEndpoint" (KeyError)
  from /usr/share/crystal/src/hash.cr:1080:9 in '??'
  from /usr/share/crystal/src/json/any.cr:139:5 in 'dig'
  from /usr/share/crystal/src/json/any.cr:237:3 in 'get_browse_id'
  from src/invidious/yt_backend/extractors.cr:332:41 in 'process'
  from src/invidious/yt_backend/extractors.cr:845:17 in 'parse_item'
  from src/invidious/yt_backend/extractors.cr:892:14 in 'extract_items'
  from src/invidious/search/processors.cr:12:18 in 'process'
  from src/invidious/routes/search.cr:55:17 in 'search'
  from src/invidious/helpers/handlers.cr:30:37 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:151:29 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:106:36 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process'
  from /usr/share/crystal/src/http/server.cr:500:5 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from ???
[removed backtics]
</p>
</details>

[unixfox]

Please only post if you have more info than the usual bug related to doing a search on invidious. Thank you.

[GTP95]

@unixfox maybe the fact that my query used to work up until very recently could help in finding the cause