iv-org
[Bug] Missing column index: 0 (IndexError)
Describe the bug
While trying to import any youtube playlist to my self hosted Invidious instance I get the error in the title with the dump below.
This occurs on all youtube playlist export CSV files that I try to import. It also occurs when attempted in a freshly created test account.
Steps to Reproduce
- Go to the gear in the upper right corner.
- Click on Import/export data.
- Click on Import Youtube playlist (CSV) "Browse".
- Select the appropriate .csv file as exported from Youtube using the Google Takeout feature.
- Error occurs.
Logs
Title: Missing column index: 0 (IndexError)
Date: 2023-06-10T00:40:40Z
Route: /data_control?referer=%2Ffeed%2Fpopular
Version: 2023.06.06-545a593 @ master
Backtrace
Missing column index: 0 (IndexError)
from /usr/share/crystal/src/csv.cr:368:7 in '[]'
from /usr/share/crystal/src/reference.cr:28:3 in 'update_data_control'
from lib/kemal/src/kemal/route.cr:12:9 in '->'
from src/invidious/helpers/handlers.cr:30:37 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/ext/kemal_static_file_handler.cr:112:11 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
from /usr/share/crystal/src/http/server.cr:500:5 in '->'
from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
from ???
Screenshots
Additional context Browser: Firefox 114.0, Chrome 114.0.5735.106 Client OS: MacOS Ventura 13.4 Server OS: Ubuntu Server 22.04 (Jammy) Docker version: 24.0.2, build cb74dfc Docker compose version: v2.18.1
- Invidious image: 0bc34cadb0ee
- Postgres image: ab8f641699b4
I was able to go into the postgres container to inspect the DB. I found that the 'playlists' table has entries for each of the playlists I tried to import, as many times as I tried to import each. Additionally the entries all contain an empty index column.
invidious=# \dt
List of relations
Schema | Name | Type | Owner
--------+-----------------+-------+-------
public | annotations | table | kemal
public | channel_videos | table | kemal
public | channels | table | kemal
public | nonces | table | kemal
public | playlist_videos | table | kemal
public | playlists | table | kemal
public | session_ids | table | kemal
public | users | table | kemal
public | videos | table | kemal
(9 rows)
invidious=# select * from playlists;
title | id | author | description | video_count | created | updated | privacy | index
--------------+-------------------------------------+--------+-------------+-------------+-------------------------------+-------------------------------+---------+-------
Music | IVPLtok-eYZ8FnxUzGLxd_R7mgarNHsEoh- | brian | | 0 | 2023-06-09 02:51:29.567816+00 | 2023-06-09 02:51:29.567816+00 | Private | {}
Watch later | IVPL52EYylidEqihPXApkZYwmIOlXjWUJ5H | brian | | 0 | 2023-06-09 02:52:30.878848+00 | 2023-06-09 02:52:30.878848+00 | Private | {}
Liked videos | IVPLwO5KY2BP7rui_RZ1xFHgOtOcEoobr_s | brian | | 0 | 2023-06-09 02:52:41.73208+00 | 2023-06-09 02:52:41.732081+00 | Private | {}
Watch later | IVPLMNHp5fiNnKBUKyacXdB6tK7A9FLmNnN | brian | | 0 | 2023-06-09 02:52:56.453253+00 | 2023-06-09 02:52:56.453253+00 | Private | {}
Music | IVPLEzV2m8L9-gPptRbIhlTcBW_Byhv8qQF | brian | | 0 | 2023-06-09 03:09:43.845614+00 | 2023-06-09 03:09:43.845615+00 | Private | {}
Watch later | IVPL0rJz7SlmLIXabSkQBS4rcJbfT9bNp_1 | brian | | 0 | 2023-06-09 04:28:40.627025+00 | 2023-06-09 04:28:40.627026+00 | Private | {}
Liked videos | IVPLBCHiOfIB0O-8FEyhnEGX3wSf0geDRGb | brian | | 0 | 2023-06-09 04:40:12.553453+00 | 2023-06-09 04:40:12.553454+00 | Private | {}
(7 rows)
invidious=# \dn
List of schemas
Name | Owner
--------+-------
public | kemal
(1 row)
invidious=# select * from pg_tables WHERE schemaname = 'public'
invidious-# ;
schemaname | tablename | tableowner | tablespace | hasindexes | hasrules | hastriggers | rowsecurity
------------+-----------------+------------+------------+------------+----------+-------------+-------------
public | channels | kemal | | t | f | f | f
public | annotations | kemal | | t | f | f | f
public | videos | kemal | | t | f | f | f
public | playlists | kemal | | t | f | t | f
public | channel_videos | kemal | | t | f | f | f
public | playlist_videos | kemal | | t | f | t | f
public | users | kemal | | t | f | f | f
public | session_ids | kemal | | t | f | f | f
public | nonces | kemal | | t | f | f | f
(9 rows)
invidious=# \d playlists
Table "public.playlists"
Column | Type | Collation | Nullable | Default
-------------+--------------------------+-----------+----------+---------
title | text | | |
id | text | | not null |
author | text | | |
description | text | | |
video_count | integer | | |
created | timestamp with time zone | | |
updated | timestamp with time zone | | |
privacy | privacy | | |
index | bigint[] | | |
Indexes:
"playlists_pkey" PRIMARY KEY, btree (id)
Referenced by:
TABLE "playlist_videos" CONSTRAINT "playlist_videos_plid_fkey" FOREIGN KEY (plid) REFERENCES playlists(id)
Seeing similar.
Linux lat5430r1 6.1.25-1-MANJARO #1 SMP PREEMPT_DYNAMIC Thu Apr 20 13:48:36 UTC 2023 x86_64 GNU/Linux
Docker version 24.0.0, build 98fdcd769b
Docker Compose version v2.6.1
Title: Missing column index: 4 (IndexError)
Date: 2023-06-10T03:49:18Z
Route: /data_control?referer=%2Ffeed%2Fsubscriptions
Version: 2023.06.09-e7bed765 @ master
Backtrace
Missing column index: 4 (IndexError)
from /usr/share/crystal/src/csv.cr:368:7 in '[]'
from /usr/share/crystal/src/csv.cr:253:5 in '[]'
from src/invidious/user/imports.cr:40:7 in 'parse_playlist_export_csv'
from src/invidious/user/imports.cr:210:20 in 'from_youtube_pl'
from src/invidious/routes/preferences.cr:315:21 in 'update_data_control'
from src/invidious/routing.cr:67:27 in '->'
from lib/kemal/src/kemal/route.cr:12:26 in '->'
from src/invidious/helpers/handlers.cr:30:37 in 'process_request'
from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/handlers.cr:212:5 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/handlers.cr:94:12 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/handlers.cr:145:12 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/handlers.cr:68:12 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/ext/kemal_static_file_handler.cr:112:11 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/logger.cr:17:35 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'
from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process'
from /usr/share/crystal/src/http/server.cr:515:5 in 'handle_client'
from /usr/share/crystal/src/http/server.cr:468:13 in '->'
from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
from /usr/share/crystal/src/fiber.cr:98:34 in '->'
from ???
Could one of you please provide the relevant CSV file(s)? If you don't want to expose publicly this data, you can send us a PM on Matrix or an email!
I sent them in an email to the Invidious contact email listed on the main site.
@bhoglan thanks for the email!
My guess is that yours seem to contain a lot of empty lines, which may confuse our parser.
here is the redacted content of one of them, for reference:
Playlist Id,Channel Id,Time Created,Time Updated,Title,Description,Visibility
PLazertyuiopqsdfghjk,UCxxxxxxxxxxxxxxxxxxxxxx,2023-06-09 00:00:00 Z,2023-06-10 00:25:00 Z,Watch later,,Private
Video Id,Time Added
xxxxxxxxx_a,2023-06-10 00:01:00 Z
xxxxxxxxx_b,2023-06-10 00:02:00 Z
xxxxxxxxx_c,2023-06-10 00:03:00 Z
xxxxxxxxx_d,2023-06-10 00:04:00 Z
xxxxxxxxx_e,2023-06-10 00:05:00 Z
xxxxxxxxx_f,2023-06-10 00:06:00 Z
xxxxxxxxx_g,2023-06-10 00:07:00 Z
xxxxxxxxx_h,2023-06-10 00:08:00 Z
xxxxxxxxx_i,2023-06-10 00:09:00 Z
xxxxxxxxx_j,2023-06-10 00:10:00 Z
xxxxxxxxx_k,2023-06-10 00:11:00 Z
xxxxxxxxx_l,2023-06-10 00:12:00 Z
xxxxxxxxx_m,2023-06-10 00:13:00 Z
xxxxxxxxx_n,2023-06-10 00:14:00 Z
xxxxxxxxx_o,2023-06-10 00:15:00 Z
xxxxxxxxx_p,2023-06-10 00:16:00 Z
xxxxxxxxx_q,2023-06-10 00:17:00 Z
xxxxxxxxx_r,2023-06-10 00:18:00 Z
xxxxxxxxx_s,2023-06-10 00:19:00 Z
xxxxxxxxx_t,2023-06-10 00:20:00 Z
xxxxxxxxx_u,2023-06-10 00:21:00 Z
xxxxxxxxx_v,2023-06-10 00:22:00 Z
xxxxxxxxx_w,2023-06-10 00:23:00 Z
xxxxxxxxx_x,2023-06-10 00:24:00 Z
xxxxxxxxx_y,2023-06-10 00:25:00 Z
Yup, I was right:
https://play.crystal-lang.org/#/r/f6lf (not working) https://play.crystal-lang.org/#/r/f6li (works, after stripping empty newlines)
Awesome I appreciate the quick feedback. I'll trim the files down and try again. Should I submit a feature request to teach the parser to ignore whitespace? I guess that would depend on whether this is a new normal format for the exports?
On Saturday, Jun 10, 2023 at 11:27, Samantaz Fox @.*** @.***)> wrote:
Yup, I was right:
https://play.crystal-lang.org/#/r/f6lf (not working) https://play.crystal-lang.org/#/r/f6li (works, after stripping empty newlines)
— Reply to this email directly, view it on GitHub (https://github.com/iv-org/invidious/issues/3891#issuecomment-1585762317), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AKD472O5ZIBC7A6FSCQOTL3XKS37BANCNFSM6AAAAAAZBJBM6U). You are receiving this because you were mentioned.Message ID: @.***>
I checked this evening, and I did indeed have lines at the end of the file. I removed them via 'dd' in vi. After that, I did a :wq and attempted the import again, but no joy:
Title: Index out of bounds (IndexError)
Date: 2023-06-11T03:34:18Z
Route: /data_control?referer=%2Ffeed%2Fsubscriptions
Version: 2023.06.09-e7bed765 @ master
Backtrace
Index out of bounds (IndexError)
from /usr/share/crystal/src/indexable.cr:897:8 in '[]'
from src/invidious/user/imports.cr:35:17 in 'parse_playlist_export_csv'
from src/invidious/user/imports.cr:210:20 in 'from_youtube_pl'
from src/invidious/routes/preferences.cr:315:21 in 'update_data_control'
from src/invidious/routing.cr:67:27 in '->'
from lib/kemal/src/kemal/route.cr:12:26 in '->'
from src/invidious/helpers/handlers.cr:30:37 in 'process_request'
from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/handlers.cr:212:5 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/handlers.cr:94:12 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/handlers.cr:145:12 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/handlers.cr:68:12 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/ext/kemal_static_file_handler.cr:112:11 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from src/invidious/helpers/logger.cr:17:35 in 'call'
from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call'
from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process'
from /usr/share/crystal/src/http/server.cr:515:5 in 'handle_client'
from /usr/share/crystal/src/http/server.cr:468:13 in '->'
from /usr/share/crystal/src/fiber.cr:146:11 in 'run'
from /usr/share/crystal/src/fiber.cr:98:34 in '->'
from ???
Samesies. No joy.
I tried by deleting all the newlines at the EOF in vim, that didn't work, so I deleted the newlines between the playlist header and the playlist content as well, that didn't work. I'm getting the same Index out of bounds error.
Title: Index out of bounds (IndexError)
Date: 2023-06-11T04:11:58Z
Route: /data_control?referer=%2Ffeed%2Fpopular
Version: 2023.06.09-e7bed765 @ master
Backtrace
Index out of bounds (IndexError) from /usr/share/crystal/src/indexable.cr:949:8 in '[]' from src/invidious/user/imports.cr:35:17 in 'parse_playlist_export_csv' from src/invidious/user/imports.cr:210:20 in 'from_youtube_pl' from src/invidious/routes/preferences.cr:315:21 in 'update_data_control' from src/invidious/routing.cr:67:27 in '->' from lib/kemal/src/kemal/route.cr:12:26 in '->' from src/invidious/helpers/handlers.cr:30:5 in 'process_request' from lib/kemal/src/kemal/route_handler.cr:17:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from lib/kemal/src/kemal/websocket_handler.cr:13:14 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/handlers.cr:212:5 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/handlers.cr:94:12 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/handlers.cr:145:12 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/handlers.cr:68:12 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/ext/kemal_static_file_handler.cr:112:11 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from src/invidious/helpers/logger.cr:17:35 in 'call' from /usr/share/crystal/src/http/server/handler.cr:30:7 in 'call_next' from lib/kemal/src/kemal/init_handler.cr:12:7 in 'call' from /usr/share/crystal/src/http/server/request_processor.cr:51:11 in 'process' from /usr/share/crystal/src/http/server.cr:517:5 in 'handle_client' from /usr/share/crystal/src/http/server.cr:470:13 in '->' from /usr/share/crystal/src/fiber.cr:146:11 in 'run' from /usr/share/crystal/src/fiber.cr:98:34 in '->' from ???
PS. This is a new fresh install of Invidious as of a few minutes ago. The original install was via Docker, the new one is directly on Linux (Ubuntu 22.04) using the instructions in the Invidious docs.
Try doing this (with line 3 being the only empty line):
Playlist Id,Channel Id,Time Created,Time Updated,Title,Description,Visibility
PLazertyuiopqsdfghjk,UCxxxxxxxxxxxxxxxxxxxxxx,2023-06-09 00:00:00 Z,2023-06-10 00:25:00 Z,Watch later,,Private
Video Id,Time Added
xxxxxxxxx_a,2023-06-10 00:01:00 Z
xxxxxxxxx_b,2023-06-10 00:02:00 Z
Yours imports fine.
Is there an email I can send my file to? I'd rather not post all that info publicly.
SUCCESS! Thanks @SamantazFox the last suggestion of removing all newlines except line 3 worked perfectly (including removing the empty lines at the bottom of the file for anyone googling this in the future (I'm looking at you, Future Brian))
Tried solutions described and still getting Missing column index: 6 (IndexError)
Line 3 is the only empty line.
Tried this with all my playlist csvs.
Title:
Missing column index: 6 (IndexError)Date:
2023-07-29T17:17:21ZRoute:/data_control?referer=%2Ffeed%2FpopularVersion:2023.07.20-d956b18 @ masterBacktrace
Missing column index: 6 (IndexError) from /usr/share/crystal/src/csv.cr:368:7 in '[]' from src/invidious/user/imports.cr:44:10 in 'update_data_control' from lib/kemal/src/kemal/route.cr:12:9 in '->' from src/invidious/helpers/handlers.cr:30:37 in 'call' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next' from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next' from src/ext/kemal_static_file_handler.cr:112:11 in 'call' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call' from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next' from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process' from /usr/share/crystal/src/http/server.cr:500:5 in '->' from /usr/share/crystal/src/fiber.cr:146:11 in 'run' from ???
