Issues when completing multi-byte characters

[abinoam]

At the HighLine PR 149 https://github.com/JEG2/highline/pull/149 we were considering replacing builtin readline with the rb-readline implementation. But we have run in some issues when trying to complete for multi-byte characters.

I have made a gist with the code I'm using to reproduce the issue at https://gist.github.com/abinoam/c8e81f7c0e851bee40f4

See it copied bellow:

#coding: utf-8

# This will remove old Readline if it's already defined
require 'rb-readline'

# This will load RbReadline for real
require 'rbreadline'

# This will load Readline that relies on RbReadline
require 'readline'

Readline.completion_proc = lambda do |str|
  [ "pure_ascii", "with_utf_8_characters_like_ã_and_ç" ].grep(/\A#{Regexp.escape(str)}/)
end

line = Readline.readline("> ", false)

puts "Got: #{line}"

STDOUT.puts "RbReadline::RB_READLINE_VERSION #{RbReadline::RB_READLINE_VERSION}"
STDOUT.puts "Readline::VERSION #{Readline::VERSION}"
STDOUT.puts "RUBY_DESCRIPTION #{RUBY_DESCRIPTION}"

I got

RbReadline::RB_READLINE_VERSION 0.5.3
Readline::VERSION 5.2
RUBY_DESCRIPTION ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin14]
>
pure_ascii                         with_utf_8_characters_like_ã_and_ç
> wi/Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/rbreadline.rb:3863:in `codepoints': invalid byte sequence in UTF-8 (ArgumentError)
    from /Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/rbreadline.rb:3863:in `_rl_col_width'
    from /Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/rbreadline.rb:2956:in `update_line'
    from /Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/rbreadline.rb:3542:in `rl_redisplay'
    from /Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/rbreadline.rb:4683:in `_rl_internal_char_cleanup'
    from /Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/rbreadline.rb:4744:in `readline_internal_charloop'
    from /Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/rbreadline.rb:4808:in `readline_internal'
    from /Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/rbreadline.rb:4830:in `readline'
    from /Users/abinoam/.rvm/gems/ruby-2.2.3/gems/rb-readline-0.5.3/lib/readline.rb:45:in `readline'
    from ptests/test_rbreadline_multibyte_support.rb:20:in `<main>'

[Spakman]

This may be aided when we get round to #75.