chardet2

chardet2 copied to clipboard

The `EscCharSetProber` class was quite buggy, and I got errors when testing for files with ISO-2022-JP encoding. This have fixed it at least for the files I tested here.

danhper

``` ruby $ ruby --version ruby 2.0.0p451 (2014-02-24 revision 45167) [universal.x86_64-darwin13] $ ruby -rUniversalDetector -e 'p UniversalDetector::chardet("(\342\214\220\xe2\x96 _\xe2\x96 )")' /Library/Ruby/Gems/2.0.0/gems/chardet2-1.0.1/lib/UniversalDetector.rb:134:in `=~': invalid byte sequence in UTF-8 (ArgumentError) from /Library/Ruby/Gems/2.0.0/gems/chardet2-1.0.1/lib/UniversalDetector.rb:134:in `feed'...

ghazel

Encoding::CompatibilityError: incompatible encoding regexp match

7

Encoding::CompatibilityError: incompatible encoding regexp match (ASCII-8BIT regexp with UTF-8 string) from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/chardet2-1.0.1/lib/UniversalDetector.rb:134:in `=~' from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/chardet2-1.0.1/lib/UniversalDetector.rb:134:in`feed' from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/chardet2-1.0.1/lib/UniversalDetector.rb:46:in `chardet' from (irb):12

orbanbotond

I got ArgumentError: invalid byte sequence in UTF-8 in ruby 1.9.3 while trying to detect a 'ISO-8859-1' encoded csv file.

5

ArgumentError: invalid byte sequence in UTF-8 from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/chardet2-1.0.1/lib/UniversalDetector.rb:134:in `=~' from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/chardet2-1.0.1/lib/UniversalDetector.rb:134:in`feed' from /Users/boti/.rvm/gems/ruby-1.9.3-p327@search_server/gems/chardet2-1.0.1/lib/UniversalDetector.rb:46:in `chardet'

orbanbotond