UTF-8 issues

[picobyte]

In case anyone has the same issue.. Building my Harlowe format file failed:

../bin/twee2 build casino.tw2 casino.html
$HOME/.gem/ruby/2.4.0/gems/twee2-0.5.0/lib/twee2/story_format.rb:12:in `match': invalid byte sequence in US-ASCII (ArgumentError)
        from $HOME/.gem/ruby/2.4.0/gems/twee2-0.5.0/lib/twee2/story_format.rb:12:in `match'
        from $HOME/.gem/ruby/2.4.0/gems/twee2-0.5.0/lib/twee2/story_format.rb:12:in `initialize'
        from $HOME/.gem/ruby/2.4.0/gems/twee2-0.5.0/lib/twee2.rb:23:in `new'
        from $HOME/.gem/ruby/2.4.0/gems/twee2-0.5.0/lib/twee2.rb:23:in `build'
        from ../bin/twee2:13:in `build'
        from $HOME/.gem/ruby/2.4.0/gems/thor-0.19.4/lib/thor/command.rb:27:in `run'
        from $HOME/.gem/ruby/2.4.0/gems/thor-0.19.4/lib/thor/invocation.rb:126:in `invoke_command'
        from $HOME/.gem/ruby/2.4.0/gems/thor-0.19.4/lib/thor.rb:369:in `dispatch'
        from $HOME/.gem/ruby/2.4.0/gems/thor-0.19.4/lib/thor/base.rb:444:in `start'
        from ../bin/twee2:45:in `<main>'

but it worked fine with:

LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 LC_ALL=en_US.UTF-8 bash  ./build.sh

[purrloftruth]

Alternately, if you're using bundler, add:

Encoding.default_external = Encoding::UTF_8
Encoding.default_internal = Encoding::UTF_8

to your Gemfile.

I think this then requires you run twee2 by using bundle exec twee2 instead of just twee2.

[xErik]

On Mac 10.13.4 I installed twee2 with gem, then modified first line of twee2 to avoid the mentioned error:

#!/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby --external-encoding=UTF-8

[bursaar]

Getting the same issue here, works natively on my machine, but trying to set up a CI it fails:

[0;m[0K  on docker-auto-scale 72989761
[0;msection_start:1556791745:prepare_executor
[0K[0KUsing Docker executor with image ruby:2.5 ...
[0;m[0KPulling docker image ruby:2.5 ...
[0;m[0KUsing docker image sha256:e86557c9a8ab97ec7e9ba705f3e3411428b50d19d38643a0f52cb49faf735d3d for ruby:2.5 ...
[0;msection_end:1556791780:prepare_executor
[0Ksection_start:1556791780:prepare_script
[0KRunning on runner-72989761-project-12119332-concurrent-0 via runner-72989761-srm-1556791694-7b87fbc7...
section_end:1556791784:prepare_script
[0Ksection_start:1556791784:get_sources
[0KInitialized empty Git repository in /builds/BenTek/games/twine/ci-test/.git/
[32;1mFetching changes...[0;m
[32;1mCreated fresh repository.[0;m
From https://gitlab.com/BenTek/games/twine/ci-test
 * [new branch]      master     -> origin/master
[32;1mChecking out bcb10fb6 as master...[0;m

[32;1mUpdating/initializing submodules recursively...[0;m
section_end:1556791788:get_sources
[0Ksection_start:1556791788:restore_cache
[0Ksection_end:1556791790:restore_cache
[0Ksection_start:1556791790:download_artifacts
[0Ksection_end:1556791792:download_artifacts
[0Ksection_start:1556791792:build_script
[0K[32;1m$ gem install therubyracer twee2[0;m
Successfully installed libv8-3.16.14.19-x86_64-linux
Successfully installed ref-2.0.0
Building native extensions. This could take a while...
Successfully installed therubyracer-0.12.3
Successfully installed builder-3.2.3
Successfully installed execjs-2.7.0
Successfully installed coffee-script-source-1.12.2
Successfully installed coffee-script-2.4.1
!    The 'trollop' gem has been deprecated and has been replaced by 'optimist'.
!    See: https://rubygems.org/gems/optimist
!    And: https://github.com/ManageIQ/optimist
Successfully installed trollop-2.9.9
Successfully installed filewatcher-0.5.4
Successfully installed tilt-2.0.9

HEADS UP! Haml 4.0 has many improvements, but also has changes that may break
your application:

* Support for Ruby 1.8.6 dropped
* Support for Rails 2 dropped
* Sass filter now always outputs <style> tags
* Data attributes are now hyphenated, not underscored
* html2haml utility moved to the html2haml gem
* Textile and Maruku filters moved to the haml-contrib gem

For more info see:

http://rubydoc.info/github/haml/haml/file/CHANGELOG.md

Successfully installed haml-4.0.7
Successfully installed mini_portile2-2.4.0
Building native extensions. This could take a while...
Successfully installed nokogiri-1.10.3
Building native extensions. This could take a while...
Successfully installed ffi-1.10.0
Successfully installed rb-inotify-0.10.0
Successfully installed rb-fsevent-0.10.3
Successfully installed sass-listen-4.0.0

Ruby Sass has reached end-of-life and should no longer be used.

* If you use Sass as a command-line tool, we recommend using Dart Sass, the new
  primary implementation: https://sass-lang.com/install

* If you use Sass as a plug-in for a Ruby web framework, we recommend using the
  sassc gem: https://github.com/sass/sassc-ruby#readme

* For more details, please refer to the Sass blog:
  https://sass-lang.com/blog/posts/7828841

Successfully installed sass-3.7.4
Successfully installed thor-0.20.3
Successfully installed twee2-0.5.0
20 gems installed
[32;1m$ LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 LC_ALL=en_US.UTF-8 twee2 build $project_name.tw2 $project_name.html[0;m
/usr/local/bundle/gems/twee2-0.5.0/lib/twee2/story_format.rb:12:in `match': invalid byte sequence in US-ASCII (ArgumentError)
	from /usr/local/bundle/gems/twee2-0.5.0/lib/twee2/story_format.rb:12:in `match'
	from /usr/local/bundle/gems/twee2-0.5.0/lib/twee2/story_format.rb:12:in `initialize'
	from /usr/local/bundle/gems/twee2-0.5.0/lib/twee2.rb:23:in `new'
	from /usr/local/bundle/gems/twee2-0.5.0/lib/twee2.rb:23:in `build'
	from /usr/local/bundle/gems/twee2-0.5.0/bin/twee2:13:in `build'
	from /usr/local/bundle/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
	from /usr/local/bundle/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
	from /usr/local/bundle/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
	from /usr/local/bundle/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
	from /usr/local/bundle/gems/twee2-0.5.0/bin/twee2:45:in `<top (required)>'
	from /usr/local/bundle/bin/twee2:23:in `load'
	from /usr/local/bundle/bin/twee2:23:in `<main>'
section_end:1556791862:build_script
[0Ksection_start:1556791862:after_script
[0Ksection_end:1556791863:after_script
[0Ksection_start:1556791863:upload_artifacts_on_failure
[0Ksection_end:1556791865:upload_artifacts_on_failure
[0K[31;1mERROR: Job failed: exit code 1
[0;m```

[SirEelBiscuits]

On Mac 10.13.4 I installed twee2 with gem, then modified first line of twee2 to avoid the mentioned error:

#!/System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby --external-encoding=UTF-8

Adding the same argument to my twee2 in WSL also worked (that's windows 10, WSL2 with some ubuntu version or other)