additional option to perform matching and scoring with unicode support

[techgaun]

[axelson]

Is it expected for ZXCVBN to raise an exception if provided a non-unicode string until this option is implemented?

Currently I see:

  test "unicode support" do
    password = "🏆🔥 and words"
    ZXCVBN.zxcvbn(password)
  end

     ** (UnicodeConversionError) incomplete encoding starting at <<240, 159>>
     code: ZXCVBN.zxcvbn(password)
     stacktrace:
       (elixir) lib/string.ex:2157: String.to_charlist/1
       (zxcvbn) lib/zxcvbn/matching.ex:565: anonymous fn/4 in ZXCVBN.Matching.do_sequence_match/2
       (elixir) lib/enum.ex:3003: Enum.reduce_range_inc/4
       (zxcvbn) lib/zxcvbn/matching.ex:561: ZXCVBN.Matching.do_sequence_match/2
       (elixir) lib/enum.ex:1327: Enum."-map/2-lists^map/1-0-"/2
       (elixir) lib/enum.ex:1327: Enum."-map/2-lists^map/1-0-"/2
       (zxcvbn) lib/zxcvbn/matching.ex:101: ZXCVBN.Matching.omnimatch/2
       (zxcvbn) lib/zxcvbn.ex:92: ZXCVBN.zxcvbn/2
       test/zxcvbn_test.exs:44: (test)

This test is with emoji but I believe it holds for foreign language characters as well.

[techgaun]

@axelson yep that's a known issue and is happening because it handles ASCII only at this time. I am working on getting support for this. I believe non-ASII passwords are used by very less percentage of people given the lack of support on various different platforms.

[axelson]

@techgaun any progress on this?

[techgaun]

@axelson I've added some support for this on master branch but need some time to do testing. Feel free to give master branch a try and see if that works in your case. I do need to spend some time to clean that up but I've not been finding time due to job and personal reasons for last few months.

[axelson]

Okay, I'll take a look at the version in master when I get a chance

[gmile]

FWIW, generally unicode appears to work nicely, but emojis are not handled still it seems: