command-t icon indicating copy to clipboard operation
command-t copied to clipboard
verified publisher icon wincent

ArgumentError: invalid byte sequence in UTF-8

Open mgedmin opened this issue 6 years ago • 2 comments

I'm trying to work on linkchecker and I get the following error, most likely because there's a file in the test scaffolding with a non-UTF-8 filename:

/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `=~'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `path_excluded?'                        
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:39:in `block (2 levels) in paths!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `each_line'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `block in paths!'
/usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run'
/usr/lib/ruby/2.5.0/open3.rb:95:in `popen3'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:27:in `paths!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:37:in `paths'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:307:in `list_matches'
(eval):3:in `list_matches'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:360:in `list_matches!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:266:in `clear'
(eval):3:in `clear'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:380:in `show'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:147:in `show_file_finder'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `=~'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `path_excluded?'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:39:in `block (2 levels) in paths!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `each_line'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `block in paths!'
/usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run'
/usr/lib/ruby/2.5.0/open3.rb:95:in `popen3'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:27:in `paths!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:37:in `paths'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:307:in `list_matches'
(eval):3:in `list_matches'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:360:in `list_matches!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:266:in `clear'
(eval):3:in `clear'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:380:in `show'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:147:in `show_file_finder'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `=~'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `path_excluded?'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:39:in `block (2 levels) in paths!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `each_line'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `block in paths!'
/usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run'
/usr/lib/ruby/2.5.0/open3.rb:95:in `popen3'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:27:in `paths!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:37:in `paths'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:307:in `list_matches'
(eval):3:in `list_matches'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:360:in `list_matches!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:266:in `clear'
(eval):3:in `clear'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:380:in `show'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:147:in `show_file_finder'
Error detected while processing function commandt#FileFinder:
line    2:
ArgumentError: invalid byte sequence in UTF-8
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `=~'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:84:in `path_excluded?'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:39:in `block (2 levels) in paths!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `each_line'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:38:in `block in paths!'
/usr/lib/ruby/2.5.0/open3.rb:205:in `popen_run'
/usr/lib/ruby/2.5.0/open3.rb:95:in `popen3'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner/find_file_scanner.rb:27:in `paths!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/scanner/file_scanner.rb:37:in `paths'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/finder.rb:39:in `sorted_matches_for'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:307:in `list_matches'
(eval):3:in `list_matches'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:360:in `list_matches!'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:266:in `clear'
(eval):3:in `clear'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:380:in `show'
/home/mg/.vim/bundle/command-t/ruby/command-t/lib/command-t/controller.rb:147:in `show_file_finder'
(eval):3:in `show_file_finder'
eval:1:in `<main>'
Press ENTER or type command to continue

To reproduce:

  • git clone https://github.com/linkchecker/linkchecker
  • cd linkchecker
  • vim +CommandT

The bad file is in tests/checker/data/dir/.

I'm fine with Command-T being unable to open or even list this file, but denying me access to the rest of the tree is a bit inconvenient.

mgedmin avatar Sep 09 '19 06:09 mgedmin

Sadly, I can't repro this:

On cloning the repo, the file in question is hidden in a zip file:

Screen Shot 2019-09-10 at 21 13 24

And the unzip tool which comes with macOS refuses to open it. 😢

linkchecker___unzip

Do you know the full path (the raw bytes of it) that has the invalid sequence? I could try to fix this blindly, but I'd like to test it with a real test case before I do.

wincent avatar Sep 10 '19 19:09 wincent

On cloning the repo, the file in question is hidden in a zip file:

Oh, interesting! It must've been extracted while running the test suite. A quick git clean -dfx and I can use command-t again!

So, from what I've heard about MacOS, its filesystem requires all filenames to be valid UTF-8 (and normalizes them into the NFD form, IIRC). I don't think it's possible for you to test it on your local system without using something like a Docker image or a Vagrant VM running Linux.

If you need an exact byte sequence, here's one that is enough to trigger the error in an empty directory:

  • mkdir /tmp/test
  • cd /tmp/test
  • python -c 'open(b"\xff", "w")'
  • vim +CommandT

but, again, I'm on Linux. I would expect the Python command that creates the file to fail on Mac OS.

mgedmin avatar Sep 11 '19 07:09 mgedmin

Given the big rewrite for v6.0.x, I'm closing all older issues as there is unlikely to be anything significant happening on the 5-x-devel branch from here on[^patches]. Feedback issue for 6.0.x is here:

  • https://github.com/wincent/command-t/issues/393

[^patches]: Patches and PRs would be welcome, but my personal efforts are going to be directed towards main.

wincent avatar Aug 26 '22 21:08 wincent