invidious Cleanup videos

Cleanup videos

Open SamantazFox opened this issue 1 year ago • 12 comments

Follow up to #3214 Related: #441

Fixes #3014 Fixes #3168 Fixes #3291 Fixes #3315

In essence, parse most of the video data before storing it.

Aug 03 '22 21:08 SamantazFox

Doesn't work on /watch?v=2isYuQZMbdU:

Nil assertion failed (NilAssertionError)
  from /usr/lib/crystal/nil.cr:108:5 in 'not_nil!'
  from src/invidious/videos.cr:828:17 in 'parse_video_info'
  from src/invidious/videos.cr:628:3 in 'extract_video_info:video_id'
  from src/invidious/videos.cr:898:3 in 'fetch_video'
  from src/invidious/videos.cr:886:13 in 'get_video'
  from src/invidious/videos.cr:869:1 in 'get_video:region'
  from src/invidious/routes/watch.cr:63:15 in 'handle'
  from src/invidious.cr:333:1 in '->'
  from lib/kemal/src/kemal/route.cr:12:26 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'process_request'
  from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:212:5 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:145:12 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:70:5 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:188:11 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/logger.cr:17:35 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'
  from /usr/lib/crystal/http/server/request_processor.cr:51:11 in 'process'
  from /usr/lib/crystal/http/server.cr:515:5 in 'handle_client'
  from /usr/lib/crystal/http/server.cr:468:13 in '->'
  from /usr/lib/crystal/fiber.cr:146:11 in 'run'
  from /usr/lib/crystal/fiber.cr:98:34 in '->'
  from ???

Doesn't work on /watch?v=PM2f835zx88:

Missing hash key: "captions" (KeyError)
  from /usr/lib/crystal/hash.cr:1077:11 in '[]'
  from src/invidious/videos.cr:421:58 in 'captions'
  from src/invidious/routes/watch.cr:155:5 in 'handle'
  from src/invidious.cr:333:1 in '->'
  from lib/kemal/src/kemal/route.cr:12:26 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'process_request'
  from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:212:5 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:145:12 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:70:5 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:188:11 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/logger.cr:17:35 in 'call'
  from /usr/lib/crystal/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'
  from /usr/lib/crystal/http/server/request_processor.cr:51:11 in 'process'
  from /usr/lib/crystal/http/server.cr:515:5 in 'handle_client'
  from /usr/lib/crystal/http/server.cr:468:13 in '->'
  from /usr/lib/crystal/fiber.cr:146:11 in 'run'
  from /usr/lib/crystal/fiber.cr:98:34 in '->'
  from ???

Aug 06 '22 10:08 unixfox

Also maybe you could tackle in this PR the bug in these two videos (iuevw6218F0 and LKsWIKl-wng) that have no comments, no description: https://github.com/iv-org/invidious/issues/3215

Aug 06 '22 10:08 unixfox

Still doesn't work on /watch?v=PM2f835zx88, /watch?v=iuevw6218F0, /watch?v=LKsWIKl-wng:

Missing hash key: "captions" (KeyError)
  from /usr/share/crystal/src/hash.cr:1077:11 in '[]'
  from src/invidious/videos.cr:421:58 in 'captions'
  from src/invidious/routes/watch.cr:155:5 in 'handle'
  from src/invidious/routing.cr:138:19 in '->'
  from lib/kemal/src/kemal/route.cr:12:26 in '->'
  from src/invidious/helpers/handlers.cr:30:37 in 'process_request'
  from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:212:5 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:145:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/handlers.cr:70:5 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/ext/kemal_static_file_handler.cr:162:16 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/logger.cr:17:35 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'
  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process'
  from /usr/share/crystal/src/http/server.cr:515:5 in 'handle_client'
  from /usr/share/crystal/src/http/server.cr:468:13 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from /usr/share/crystal/src/fiber.cr:98:34 in '->'
  from ???

Aug 23 '22 11:08 unixfox

I won't make further changes for now. I prefer to make multiple smaller PRs rather than big monoliths.

Sep 11 '22 13:09 SamantazFox

Deployed on https://test.invidious.io, if anyone wants to try it.

Sep 11 '22 14:09 SamantazFox

So far so good. @unixfox All the videos you mentioned in here aren't giving an error page anymore.

Sep 11 '22 21:09 SamantazFox

I need to fix #3168...

Sep 12 '22 21:09 SamantazFox

Issue on https://test.invidious.io/watch?v=MQqPsEKTKu8

Invalid Int64: "16,798" (ArgumentError)
  from /usr/share/crystal/src/string.cr:462:5 in 'to_i64'
  from invidious.git/src/invidious/videos/parser.cr:164:14 in 'parse_video_info'
  from invidious.git/src/invidious/videos/parser.cr:91:3 in 'extract_video_info:video_id'
  from invidious.git/src/invidious/videos.cr:350:3 in 'fetch_video'
  from invidious.git/src/invidious/videos.cr:338:13 in 'get_video'
  from invidious.git/src/invidious/videos.cr:321:1 in 'get_video:region'
  from invidious.git/src/invidious/routes/watch.cr:63:15 in 'handle'
  from invidious.git/src/invidious/routing.cr:138:19 in '->'
  from invidious.git/lib/kemal/src/kemal/route.cr:12:26 in '->'
  from invidious.git/src/invidious/helpers/handlers.cr:30:37 in 'process_request'
  from invidious.git/lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/src/invidious/helpers/handlers.cr:212:5 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/src/invidious/helpers/handlers.cr:94:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/src/invidious/helpers/handlers.cr:145:12 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/src/invidious/helpers/handlers.cr:70:5 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/src/ext/kemal_static_file_handler.cr:162:16 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/src/invidious/helpers/logger.cr:17:35 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from invidious.git/lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'
  from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process'
  from /usr/share/crystal/src/http/server.cr:515:5 in 'handle_client'
  from /usr/share/crystal/src/http/server.cr:468:13 in '->'
  from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
  from /usr/share/crystal/src/fiber.cr:98:34 in '->'
  from ???

Sep 13 '22 20:09 unixfox

@unixfox this has been fixed :) There was also another issue with lastModified not being available on livestreams.

Sep 14 '22 18:09 SamantazFox

Will test on my instance too for a few days.

Sep 17 '22 11:09 unixfox

(for this video the error was Missing JSON element "microformat" (BrokenTubeException) but it changed to another one (Unknown enum VideoType) after I've loaded the video into the cache on a previous invidious version)

/watch?v=0rYrAUqUmKo:

Unknown enum VideoType value: regular (ArgumentError)
  from /usr/lib/crystal/core/string.cr:5286 in 'video_type'
  from src/invidious/views/watch.ecr:105:14 in 'handle'
  from lib/kemal/src/kemal/route.cr:13:9 in '->'
  from lib/kemal/src/kemal/config.cr:92:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/lib/crystal/core/http/server.cr:500:5 in '->'
  from /usr/lib/crystal/core/fiber.cr:146:11 in 'run'
  from /src/boringssl/src/google-boringssl-251b516/ssl/internal.h:334:8 in '??'

/watch?v=kNXFd_Lu7Jo:

Missing JSON element "microformat" (BrokenTubeException)
  from /usr/lib/crystal/core/array.cr:114:31 in 'parse_video_info'
  from src/invidious/videos/parser.cr:91:12 in 'extract_video_info:video_id:context_screen'
  from /usr/lib/crystal/core/hash.cr:1185:13 in 'fetch_video'
  from src/invidious/videos.cr:338:13 in 'get_video:region'
  from src/invidious/routes/watch.cr:63:15 in 'handle'
  from lib/kemal/src/kemal/route.cr:13:9 in '->'
  from lib/kemal/src/kemal/config.cr:92:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call'
  from /usr/lib/crystal/core/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/lib/crystal/core/http/server.cr:500:5 in '->'
  from /usr/lib/crystal/core/fiber.cr:146:11 in 'run'
  from /src/boringssl/src/google-boringssl-251b516/ssl/internal.h:334:8 in '??'

Sep 17 '22 18:09 unixfox

(for this video the error was Missing JSON element "microformat" (BrokenTubeException) but it changed to another one (Unknown enum VideoType) after I've loaded the video into the cache on a previous invidious version)

Ah, yeah :/ I changed the format of the JSON in the DB, so different invidious versions can't cohabit...

Sep 19 '22 16:09 SamantazFox

@unixfox I've added the required logic to prevent the cache problem mentioned in https://github.com/iv-org/invidious/pull/3238#issuecomment-1250119986.

I've also fixed the Unknown enum VideoType value: regular issue.

Oct 03 '22 22:10 SamantazFox

In testing on https://yewtu.be

Oct 19 '22 08:10 unixfox

New bug: https://github.com/iv-org/invidious/issues/3352

Oct 19 '22 13:10 unixfox

New bug: #3352

Bug has been fixed, btw

Nov 05 '22 17:11 SamantazFox

In testing again on yewtu.be

Nov 05 '22 18:11 unixfox

New bug: https://github.com/iv-org/invidious/issues/3381 Removed testing from yewtu.be

Nov 05 '22 21:11 unixfox

Should be also fixed, but I want to do some testing first

Nov 06 '22 22:11 SamantazFox

Ok, so now all the videos listed in this PR, as well as all the linked issues works, except for two:

MQqPsEKTKu8 ("livestream recording not available") => Was that an old Lofi girl stream?
0rYrAUqUmKo (can't be played) => because copyrighted music, I guess?

Nov 13 '22 13:11 SamantazFox

New bug: https://github.com/unixfox/invidious-custom/issues/47

Nov 16 '22 05:11 unixfox

invidious invidious copied to clipboard

Cleanup videos

invidious
invidious copied to clipboard