ecko
ecko copied to clipboard
Published
20 hours ago •
magicstone-dev
magicstone-dev
Problem: Import blocklist csv fails on empty double-quotes
Got an error importing what seemed to me to be a valid CSV file.
Log:
$ RAILS_ENV=development bin/tootctl domains import_blocked "https://csv.rathersafe.space/f.php?d=1&h=2cu04wzMC"
DEPRECATION WARNING: add_whitelist is deprecated in favor of add_safelist. It will be removed from the next major release. (called from block (2 levels) in <top (required)> at /vagrant/config/environments/development.rb:84)
Traceback (most recent call last):
22: from bin/tootctl:8:in `<main>'
21: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
20: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
19: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
18: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
17: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor.rb:243:in `block in subcommand'
16: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/invocation.rb:116:in `invoke'
15: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
14: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
13: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
12: from /vagrant/lib/mastodon/domains_cli.rb:171:in `import_blocked'
11: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv.rb:686:in `parse'
10: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv.rb:1289:in `read'
9: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv.rb:1289:in `to_a'
8: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv.rb:1280:in `each'
7: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv.rb:1280:in `each'
6: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/parser.rb:336:in `parse'
5: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/parser.rb:823:in `parse_quotable_loose'
4: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/parser.rb:49:in `each_line'
3: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/parser.rb:49:in `each_line'
2: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/parser.rb:52:in `block in each_line'
1: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/parser.rb:862:in `block in parse_quotable_loose'
/home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/parser.rb:923:in `parse_quotable_robust': Illegal quoting in line 4. (CSV::MalformedCSVError)
https://csv.rathersafe.space/f.php?d=1&h=2cu04wzMC file doesn't seem to exist. Navigated but said no files found.
I guess there's some sort of hotlink protection. I can download it after I visit https://rathersafe.space/blocklist-csv
Also tried after removing the double quotes and then got this:
$ RAILS_ENV=development bin/tootctl domains import_blocked "https://bitcoinexchangerate.org/files/blocklist-clean.csv"
DEPRECATION WARNING: add_whitelist is deprecated in favor of add_safelist. It will be removed from the next major release. (called from block (2 levels) in <top (required)> at /vagrant/config/environments/development.rb:84)
Traceback (most recent call last):
29: from bin/tootctl:8:in `<main>'
28: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
27: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
26: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
25: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
24: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor.rb:243:in `block in subcommand'
23: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/invocation.rb:116:in `invoke'
22: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
21: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
20: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
19: from /vagrant/lib/mastodon/domains_cli.rb:174:in `import_blocked'
18: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/table.rb:337:in `each'
17: from /home/vagrant/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/csv/table.rb:337:in `each'
16: from /vagrant/lib/mastodon/domains_cli.rb:182:in `block in import_blocked'
15: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/inheritance.rb:72:in `new'
14: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/inheritance.rb:72:in `new'
13: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/core.rb:510:in `initialize'
12: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
11: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/attr_encrypted-3.1.0/lib/attr_encrypted/adapters/active_record.rb:28:in `perform_attribute_assignment'
10: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activemodel-6.1.4.1/lib/active_model/attribute_assignment.rb:34:in `assign_attributes'
9: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/attribute_assignment.rb:13:in `_assign_attributes'
8: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/attribute_assignment.rb:13:in `each'
7: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/attribute_assignment.rb:21:in `block in _assign_attributes'
6: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activemodel-6.1.4.1/lib/active_model/attribute_assignment.rb:49:in `_assign_attribute'
5: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activemodel-6.1.4.1/lib/active_model/attribute_assignment.rb:49:in `public_send'
4: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activemodel-6.1.4.1/lib/active_model/attribute_methods.rb:254:in `severity='
3: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/attribute_methods/write.rb:40:in `_write_attribute'
2: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activemodel-6.1.4.1/lib/active_model/attribute_set.rb:51:in `write_from_user'
1: from /home/vagrant/.rvm/gems/ruby-2.7.4/gems/activemodel-6.1.4.1/lib/active_model/attribute.rb:72:in `with_value_from_user'
/home/vagrant/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/enum.rb:148:in `assert_valid_value': '#severity' is not a valid severity (ArgumentError)
The severity can only be either one enum severity: [:silence, :suspend, :noop]. Maybe we can rescue this error and show a friendly text.
Whatever works. I feel like header rows more often can be commented once at the beginning and that should stop further processing of that row.