JavaScript compatibility

[ggrossetie]

Hi,

I'm working on the Asciidoctor Browser extension (which uses Asciidoctor.js) and I got the following question:

Hi, Does the live preview support asciidoctor-bibtex? My bibliographies don't show up, but perhaps I need to configure something? Gavin

As far as I understand, "asciidoctor-bibtex" uses https://github.com/inukshuk/bibtex-ruby so it may not be an easy task to compile/port this library to JavaScript.

Anyway if you think it's possible to compile this library to JavaScript using Opal I can get you set-up :wink:

Thanks!

[ProgramFan]

asciidoctor-bibtex and bibtex-ruby are pure ruby and I can try to make asciidoctor-bibtex conform to a certain standard so that it can be compiled to JavaScript. asciidoctor-bibtex is such a simple package that the porting shall not be hard on its own. But you would have to check for all its dependencies.

[Nadano]

Would this allow to use asciidoc-bibtex in the live preview of vscode? that would be great!

[ggrossetie]

@Nadano Yes, the live preview of VSCode is based on Asciidoctor.js so once this library is transpiled to JavaScript it can be integrated in JavaScript-based tools.

[ProgramFan]

I am not familiar with opal at the moment. How can this be done?

[ggrossetie]

Basically, you will need to add a Rake task to produce a JavaScript file from this Ruby library. You can take a look at the Asciidoctor reveal.js converter:

https://github.com/asciidoctor/asciidoctor-reveal.js/blob/e443b549741782663d648ef40c37952624eafefa/Rakefile#L32-L50

require 'opal'

builder = Opal::Builder.new(compiler_options: {
  dynamic_require_severity: :error,
})
builder.append_paths 'lib'
builder.build 'asciidoctor-bibtex'

File.write 'asciidoctor-bibtex.js', builder.to_s

That's the easy part! Then you will probably need to make some adjustments to the code base to workaround some limitations/constraints dictated by Opal.

If you open a pull request with this Rake task, I can provide guidance to resolve these issues one step at a time (you will also need to run this task on Travis so we have a shared environment to reproduce issues).

[Leon0402]

Is there some progress on this topic?

I would help, in fact I even tried it a little bit myself, but I neither know ruby nor javascript very well making this quite difficult.

@Mogztter For example how does this work with dependencies. For instance asciidoctor.js, I tried specifying it like this: builder.use_gem 'asciidoctor'

It then stuck here: https://github.com/asciidoctor/asciidoctor/blob/master/lib/asciidoctor.rb#L8

I then just disabled the require for asciidoctor completly as I saw that somewhere else require 'asciidoctor' unless RUBY_PLATFORM == 'opal'. But I'm unsure if this is the right way.

Similarly I have problems with other libs. I just did again something like builder.use_gem 'bibtex-ruby'. I think this will basically convert these libs to javascript as well?

Currently it is stuck with can't find file: "digest/md5" in:. Which seems to be a dependency of bibtex-ruby. Do I need to specify with "use_gem" all libs that are used as dependencies recursively?

[ggrossetie]

Similarly I have problems with other libs. I just did again something like builder.use_gem 'bibtex-ruby'. I think this will basically convert these libs to javascript as well?

Yes, you could say that.

I then just disabled the require for asciidoctor completly as I saw that somewhere else require 'asciidoctor' unless RUBY_PLATFORM == 'opal'. But I'm unsure if this is the right way.

This is the right way, Asciidoctor is a runtime dependency, you should not need it to transpile to JavaScript.

Currently it is stuck with can't find file: "digest/md5" in:. Which seems to be a dependency of bibtex-ruby. Do I need to specify with "use_gem" all libs that are used as dependencies recursively?

No, digest/md5 is part of the stdlib of Ruby: https://ruby-doc.org/stdlib-2.4.0/libdoc/digest/rdoc/Digest/MD5.html Unfortunately, Opal does not provide an implementation: https://github.com/opal/opal/issues/1661

Having said that, I didn't find any usage of Digest::MD5 in bibtex-ruby. Maybe this require can be removed? You should probably ask the maintainer.

[Leon0402]

This is the right way, Asciidoctor is a runtime dependency, you should not need it to transpile to JavaScript.

How do I know that it's only a runtime issue?

Having said that, I didn't find any usage of Digest::MD5 in bibtex-ruby. Maybe this require can be removed? You should probably ask the maintainer.

You're right, I did search a little bit with git blame, the code using it was removed already. I forked the project for now and removed it.

Next I run into an issue with dynamic require, but I think I solved this:

https://github.com/Leon0402/bibtex-ruby/commit/bcbdf04ebbd0e93c70e649381a156c528ecb067f

The next issue is that it requires "java".

A file required by "bibtex/filters.rb" wasn't found.
A file required by "bibtex/filters/latex.rb" wasn't found.
A file required by "latex/decode.rb" wasn't found.
A file required by "latex/decode/compatibility.rb" wasn't found.
can't find file: "java" in:

I don't quite understand it as I thought RUBY_PLATTFORM is "opal". Anyway I just completly excluded compatibility.rb the same way I did with asciidoctor for now (in my local gem). Which is probably a very bad idea. Do you know how to fix this properly

The next issue then is this line: https://github.com/Leon0402/bibtex-ruby/blob/master/lib/bibtex/names.rb#L252

Based on your previous answer this seems again to be part of the stdlib and again not to be supported by opal: https://github.com/opal/opal/issues/1749? But I guess this code is important for the lib. So what can we do about it?

[ggrossetie]

I don't quite understand it as I thought RUBY_PLATFORM is "opal".

RUBY_ENGINE is equal to "opal", not sure about RUBY_PLATFORM.

Anyway I just completly excluded compatibility.rb the same way I did with asciidoctor for now (in my local gem). Which is probably a very bad idea. Do you know how to fix this properly

You can simplify the code for Opal by removing the if condition that contains the require 'java'. You can also add a special condition for Opal: if RUBY_ENGINE == 'opal'.

Based on your previous answer this seems again to be part of the stdlib and again not to be supported by opal: opal/opal#1749? But I guess this code is important for the lib. So what can we do about it?

That's correct but I don't think we are using to_xml in this library so we could probably remove it (when transpiling).

[Leon0402]

Do I then need to fork all these things and modify the source code? Or is there a better / easier way? (At least for the latex gem for instance ... I don't want to fork everything :-)) And just modifying it locally is not soo great.

RUBY_ENGINE is equal to "opal", not sure about RUBY_PLATFORM.

At least require 'asciidoctor' unless RUBY_PLATFORM == 'opal' worked fine, so I assumed it has to be opal

Edit: I see the issue. Opal does try to compile the whole file no matter what the variables are I think.

[ggrossetie]

At least require 'asciidoctor' unless RUBY_PLATFORM == 'opal' worked fine, so I assumed it has to be opal

You're right, both are defined: https://github.com/opal/opal/blob/master/opal/corelib/constants.rb#L1-L2

Edit: I see the issue. Opal does try to compile the whole file no matter what the variables are I think.

It only works when you explicitly check if the engine/platform is (or isn't) opal, see: https://github.com/opal/opal/blob/master/lib/opal/rewriters/opal_engine_check.rb

Do I then need to fork all these things and modify the source code? Or is there a better / easier way? (At least for the latex gem for instance ... I don't want to fork everything :-)) And just modifying it locally is not soo great.

I think the best approach right now is probably to modify the source code locally. Once you get something working, we will need to apply these changes before transpiling probably using monkey patch. That's what we are doing in Asciidoctor.js: https://github.com/asciidoctor/asciidoctor.js/blob/master/packages/core/lib/asciidoctor/js/asciidoctor_ext/document.rb. In this case, we replace the Document#fill_datetime_attributes method with a "JavaScript compatible" implementation.

If bibtex-ruby and/or latex-decode projects are willing to support Opal that's even better. In this case, you can submit patches upstream.

[Leon0402]

Okay I got something compiled yay :)

asciidoctor-bibtex.zip

What I have done up to now:

• Removed MD5 require from bibtex-ruby (fork)
• Removed all rdf and xml converter stuff from bibtex-rube (fork)
• Excluded compatibilities file of latex-decode, cls and citeproc (local)
• Remove all require and code of nokogiri from cls (local) -> Could be bad?
• And somehwere I removed some dynamic import, which might be bad as well (But I don't know where currently)

So I now try to use the extension:

npx asciidoctor-web-pdf --require ./extern/asciidoctor-bibtex.js src/main.adoc

But instantly get the error message from the js "ERROR: TypeError: $$ is not a function". This looks like jquery if I'm not mistaken.

[ggrossetie]

Okay I got something compiled yay :)

Yay! That's a first step, well done 👍🏻

Removed MD5 require from bibtex-ruby (fork)

I think you can submit this patch upstream since it's not used anymore. All other changes would probably be added via monkey-patch (at least at first).

But instantly get the error message from the js "ERROR: TypeError: $$ is not a function". This looks like jquery if I'm not mistaken.

Could you please copy/paste the complete stacktrace? JQuery is indeed using $ but Opal is also using $$ as an alias for Opal.const_get_relative: https://github.com/opal/opal/blob/584d8e758fa12b378b44e5beacc15e0298d51af9/opal/corelib/runtime.js#L420

You should use Opal 0.11.0 to transpile this library (because Opal 1.1.1 and Opal 0.11.0 are not compatible and Asciidoctor.js 2.2.x is using Opal 0.11.0). You can use the following declaration to use the exact same version as Asciidoctor.js (safest option): https://github.com/asciidoctor/asciidoctor-reveal.js/blob/ba16105cedb80171a127d543215b53db2b3f01a4/Gemfile#L6

[Leon0402]

Thanks! Using Opal 0.11.0 already helped with that.

Next issue with full stack trace:

ERROR:  datetime_format=: undefined method `datetime_format=' for #<Logger:0x476>
    at constructor.$method_missing (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:3938:56)
    at constructor.method_missing_stub (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1310:35)
    at Opal.send (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1671:19)
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:9618:5
    at Opal.modules.bibtex (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:9625:5)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17)
    at Opal.modules.asciidoctor-bibtex/processor (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:21870:8)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17) {
  args: [ '%Y-%m-%d %H:%M:%S' ]
}

in the bibtext module section of the lib the following line is: Opal.add_stubs(['$require', '$new', '$key?', '$level=', '$-', '$datetime_format=', '$attr_accessor']);

Seems to be https://github.com/inukshuk/bibtex-ruby/blob/a2b74add5c5597cc193e307478193d1b566a9237/lib/bibtex.rb#L45

Removed it and next error. Is this how you deal with these issues? :D

The next error I could resolve myself. Now I get this:

ERROR:  SyntaxError: Invalid regular expression: /\\(r|H|u|v|G|M|d|c|k|b|B|t)\{(\p{L}\p{M}*)([[:alpha:]]*)\}/: Lone quantifier brackets
    at new RegExp (<anonymous>)
    at $Diacritics$4 (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3990:16)
    at Object.Opal.yieldX (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1475:18)
    at Function.$ruby_19 (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3758:19)
    at Opal.send (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1671:19)
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3987:233
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3998:9
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:3999:7
    at Opal.modules.latex/decode/diacritics (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:4000:5)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)

The module is latex/decode/diacritics

      @patterns = [
        ruby_18 {
          /\\(#{ @macros.keys.map { |k| Regexp.escape(k) }.join('|') })\{([[:alpha:]]?)([[:alpha:]]*)\}/ou          
        } ||
        ruby_19 {
          /\\(#{ @macros.keys.map { |k| Regexp.escape(k) }.join('|') })\{(\p{L}\p{M}*)([[:alpha:]]*)\}/ou
        },
        /\\(o|O|l|L|aa|AA|ae|AE)\b/,
      ].freeze

[ggrossetie]

Removed it and next error. Is this how you deal with these issues? :D

Not really 😆 In this case, it's not crucial but the right way to fix it is to implement the missing method in Opal. Here, you should open a pull request to implement the method def datetime_format= in https://github.com/opal/opal/blob/master/stdlib/logger.rb

The next error I could resolve myself. Now I get this:

Ruby regular expression and JavaScript regular expression share some similarities but do not use the same engine and do not support the same features/syntaxes.

This regular expression is probably valid in Ruby but invalid in JavaScript. That's the reason why we are using the following block in Asciidoctor Ruby: https://github.com/asciidoctor/asciidoctor/blob/7d754d2ceb609ba54e83a058347ce64d3d843bec/lib/asciidoctor.rb#L427-L437 And in Asciidoctor.js, we override these constants: https://github.com/asciidoctor/asciidoctor.js/blob/master/packages/core/lib/asciidoctor/js/rx.rb

Having said that running /\\(r|H|u|v|G|M|d|c|k|b|B|t)\{(\p{L}\p{M}*)([[:alpha:]]*)\}/.test('foo') on the DevTools console (Chrome) is working fine:

Where do you get this error? Are you using Node? Which version?

[Leon0402]

Not really laughing In this case, it's not crucial but the right way to fix it is to implement the missing method in Opal. Here, you should open a pull request to implement the method def datetime_format= in https://github.com/opal/opal/blob/master/stdlib/logger.rb

Ah I see

Where do you get this error? Are you using Node? Which version?

It's from latex/decode/diacritics, the code snippet above or what do you mean?

The exact code should be:

new RegExp("" + "\\\\(" + ($send(self.macros.$keys(), 'map', [], ($$5 = function (k) {
  var self = $$5.$$s || this;
  if (k == null) {
    k = nil;
  };
  return $$($nesting, 'Regexp').$escape(k);
}, $$5.$$s = self, $$5.$$arity = 1, $$5)).$join("|")) + ")\\{(\\p{L}\\p{M}*)([[:alpha:]]*)\\}", 'u')

The regex should be more than what was outputed. At least the two "()" are missing and possibly the rest in the middle

I'm not sure I'm using Node? I guess :-) My Version is: v12.22.3

[ggrossetie]

It's from latex/decode/diacritics, the code snippet above or what do you mean?

Sorry I wasn't clear. What command do you run? npx asciidoctor-web-pdf --require ./extern/asciidoctor-bibtex.js src/main.adoc?

I'm not sure I'm using Node? I guess :-) My Version is: v12.22.3

If you are using asciidoctor-web-pdf then you are using Node.js. You can type node -v to get the version.

Could you please share the extension transpiled to JavaScript so I can troubleshoot/debug on my machine?

[ggrossetie]

I can reproduce this issue, [[:alpha:]] is not valid in a JavaScript regular expression. You need to replace it by \p{Alpha}.

@patterns = [
  ruby_19 {
    /\\(#{ @macros.keys.map { |k| Regexp.escape(k) }.join('|') })\{(\p{L}\p{M}*)(\p{Alpha}*)\}/ou
  },
  /\\(o|O|l|L|aa|AA|ae|AE)\b/,
].freeze

(I removed ruby_18 because we don't need it when running with Opal)

[Leon0402]

npx asciidoctor-web-pdf --require ./extern/asciidoctor-bibtex.js src/main.adoc

Correct

If you are using asciidoctor-web-pdf then you are using Node.js. You can type node -v to get the version.

Yeah I do, it's v12.22.3

I can reproduce this issue, [[:alpha:]] is not valid in a JavaScript regular expression. You need to replace it by \p{Alpha}.

Why does it run in the browser developer console though? Is this Node specific?

ERROR:  constructor [ArgumentError]: wrong number of arguments (3 for 0)

  NOTE:If you want to enable passing a String argument please add "require 'opal-parser'" to your script

    at Opal.send (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1671:19)
    at Function.$exception (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:4993:14)
    at Function.$raise (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:4681:31)
    at Function.$module_eval (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:3320:32)
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5255:12
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5304:7
    at Opal.modules.bibtex/name_parser (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5305:5)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17)
    at Opal.modules.bibtex (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:9681:8)

Her the js asciidoctor-bibtex.zip

Has this something to do with module_eval? As eval seems to be a feature of "opal-parser". Based on a quick search bibtex seems to use this quite heavily

[ggrossetie]

Why does it run in the browser developer console though? Is this Node specific?

I didn't understand 🤔

Her the js asciidoctor-bibtex.zip

Thanks!

Has this something to do with module_eval? As eval seems to be a feature of "opal-parser". Based on a quick search bibtex seems to use this quite heavily

That's bad 😞 But yes, since bibtex is evaluating Ruby as runtime, we need the opal-parser library to convert Ruby code as JavaScript code at runtime. To be honest, I've never used this library so I can't really help you but I'm pretty sure it will make things complicated since we will need to read a file named names.y at runtime.

I'm not entirely sure why this is not done at compile time (in bibtex) since module_eval can be resolve statically (at least in https://github.com/inukshuk/bibtex-ruby/blob/master/lib/bibtex/name_parser.rb)

[Leon0402]

Why does it run in the browser developer console though? Is this Node specific?

I didn't understand thinking

Her the js asciidoctor-bibtex.zip

Thanks!

Has this something to do with module_eval? As eval seems to be a feature of "opal-parser". Based on a quick search bibtex seems to use this quite heavily

That's bad disappointed But yes, since bibtex is evaluating Ruby as runtime, we need the opal-parser library to convert Ruby code as JavaScript code at runtime. To be honest, I've never used this library so I can't really help you but I'm pretty sure it will make things complicated since we will need to read a file named names.y at runtime.

I'm not entirely sure why this is not done at compile time (in bibtex) since module_eval can be resolve statically (at least in https://github.com/inukshuk/bibtex-ruby/blob/master/lib/bibtex/name_parser.rb)

Perhaps because of:

#
# DO NOT MODIFY!!!!
# This file is automatically generated by Racc 1.4.15
# from Racc grammer file "".
#

See beginning of file.

As far as I can see it's name_parser.rb and parser.rb, which are definitly needed :(

Any suggestions how to move forward? This sounds like it might be simpler to rewrite this whole thing manually in javascript :disappointed:

[ggrossetie]

I'm not familiar with racc but apparently it's possible to regenerate files using the -l. In this case, module_eval is not used:

lib/bibtex/name_parser.rb

Racc_debug_parser = false

##### State transition tables end #####

# reduce 0 omitted

def _reduce_1(val, _values, result)
 result = []
    result
end

# reduce 2 omitted

def _reduce_3(val, _values, result)
 result = [val[0]]
    result
end

def _reduce_4(val, _values, result)
 result << val[1]
    result
end

def _reduce_5(val, _values, result)
 result = Name.new(val[1])
    result
end

I think you should suggest this change at: https://github.com/inukshuk/bibtex-ruby

[Leon0402]

Wow awesome! :+1:

I regenerated the files in my fork: https://github.com/Leon0402/bibtex-ruby/blob/master/lib/bibtex/name_parser.rb

The error message hasn't changed though.

Here is a snippet from the js code. self.$module_eval("" + "\n" + " @patterns = {\n" + " :and => /,?\\s+and\\s+/io,\n" + " :space => /\\s+/o,\n" + " :comma => /,/o,\n" + " :lower => /[[:lower:]][[:lower:][:upper:]]*/uo,\n" + " :upper => /[[:upper:]][[:lower:][:upper:].]*/uo,\n" + " :other => /[^\\s,\\{\\}\\[:upper:][:lower:]]+/uo,\n" + " :lbrace => /\\{/o,\n" + " :rbrace => /\\}/o,\n" + " :braces => /[\\{\\}]/o,\n" + " :escape => /\\./o,\n" + " :last_first => /[\\p{Han}\\p{Hiragana}\\p{Katakana}\\p{Hangul}]/uo\n" + " }\n" + "\n" + " class << self\n" + " attr_reader :patterns\n" + " end\n" + "\n" + " def initialize(options = {})\n" + " self.options.merge!(options)\n" + " end\n" + "\n" + " def options\n" + " @options ||= { :debug => ENV['DEBUG'] == true }\n" + " end\n" + "\n" + " def parse(input)\n" + " @yydebug = options[:debug]\n" + " scan(input)\n" + " do_parse\n" + " end\n" + "\n" + " def next_token\n" + " @stack.shift\n" + " end\n" + "\n" + " def on_error(tid, val, vstack)\n" + " BibTeX.log.debug(\"Failed to parse BibTeX Name on value %s (%s) %s\" % [val.inspect, token_to_str(tid) || '?', vstack.inspect])\n" + " end\n" + "\n" + " def scan(input)\n" + " @src = StringScanner.new(input)\n" + " @brace_level = 0\n" + " @last_and = 0\n" + " @stack = [[:ANDFL,'(^start)']]\n" + " @word = [:PWORD,'']\n" + " do_scan\n" + " end\n" + "\n" + " private\n" + "\n" + " def do_scan\n" + " until @src.eos?\n" + " case\n" + " when @src.scan(NameParser.patterns[:and])\n" + " push_word\n" + " @last_and = @stack.length\n" + " @stack.push([:ANDFL,@src.matched])\n" + "\n" + " when @src.skip(NameParser.patterns[:space])\n" + " push_word\n" + "\n" + " when @src.scan(NameParser.patterns[:comma])\n" + " push_word\n" + " @stack.push([:COMMA,@src.matched])\n" + "\n" + " when @src.scan(NameParser.patterns[:lower])\n" + " is_lowercase\n" + " @word[1] << @src.matched\n" + "\n" + " when @src.scan(NameParser.patterns[:upper])\n" + " is_uppercase\n" + " @word[1] << @src.matched\n" + "\n" + " when @src.scan(NameParser.patterns[:other])\n"

And a couple more of "module_eval" all related to these two files.

There is only one "eval" left https://github.com/Leon0402/bibtex-ruby/blob/master/lib/bibtex/filters.rb#L10. But this doesn't seem at all related?

[ggrossetie]

There's still a self.$module_eval in the JavaScript file but I have no idea why... 🤔

And a couple more of "module_eval" all related to these two files.

That's weird... I can't find any module_eval in the source (Ruby) so I'm not sure why Opal generates self.$module_eval.

But this doesn't seem at all related?

I don't think it's related, here we are using class_eval.

[Leon0402]

I figured it out. First there was a problem with another lib we have as an indirect dependency https://github.com/berkmancenter/namae. This seems pretty much the same code generated with racc, so I fixed it with the -l flag.

And then it didn't use my newest master of the bibtex fork. And I still don't understand why. I put branch "master" in my gem file and it worked all the time. I now pinned it to the newest ref on master and now it's installing that version with "bundle install".

No module_eval anymore :-)

The next error is regex again:

Invalid regular expression: /[[:lower:]][[:lower:][:upper:]]*/: Lone quantifier brackets

These come from names.y in ruby-bibtex.

  @patterns = {
    :and => /,?\s+and\s+/io,
    :space => /\s+/o,
    :comma => /,/o,
    :lower => /[[:lower:]][[:lower:][:upper:]]*/uo,
    :upper => /[[:upper:]][[:lower:][:upper:].]*/uo,
    :other => /[^\s,\{\}\\[:upper:][:lower:]]+/uo,
    :lbrace => /\{/o,
    :rbrace => /\}/o,
    :braces => /[\{\}]/o,
    :escape => /\\./o,
    :last_first => /[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]/uo
  }

I suppose it will be similar to the fix you had above with [[:xyz:] becoming to \p{xyz}. I'm not sure though how to replace [[:xyz:][:abc:]]

[Leon0402]

I'm note 100% sure, but I think I figured this out:

https://github.com/Leon0402/bibtex-ruby/commit/c605037343517011b1a87695ec133adeb609f390

It now says

SyntaxError: Invalid regular expression: /[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]/: Invalid property name in character class

I think the /u in the end might be the problem?

"braces": /[\{\}]/, "escape": /\\./, "last_first": /[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]/u});

at least without there is no problem. Can I just leave it away in ruby?

ERROR:  Iconv: uninitialized constant BibTeX::Iconv
    at Function.$const_missing (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:3083:52)
    at const_missing (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:224:32)
    at Opal.const_get_relative (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:299:38)
    at /home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5717:20
    at Opal.modules.bibtex/compatibility (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:5741:5)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17)
    at Opal.modules.bibtex (/home/leon/Projects/asciidoctor-thesis-template/extern/asciidoctor-bibtex.js:10355:8)
    at Object.Opal.load (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2311:7)
    at constructor.Opal.require (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:2339:17)

Which might come from https://github.com/inukshuk/bibtex-ruby/blob/a2b74add5c5597cc193e307478193d1b566a9237/lib/bibtex/compatibility.rb. But I don't see the problem.

[ggrossetie]

I think the reason is that the unicode property escapes Han, Hiragana, Katakana and Hangul are not supported in JavaScript.

at least without there is no problem. Can I just leave it away in ruby?

If you remove the /u the regular expression won't work the same. I'm not sure why the Bibtex library is matching Han, Hiragana, Katakana, and Hangul unicode characters? Please note that it's possible to replace unicode property escapes with unicode range. For instance, if we want to match Kanji, we can use: /[\u4E00-\u9FAF\u3040-\u3096\u30A1-\u30FA\uFF66-\uFF9D\u31F0-\u31FF]/.

See: https://stackoverflow.com/questions/7344871/javascript-regular-expression-to-catch-kanji

Which might come from inukshuk/bibtex-ruby@a2b74ad/lib/bibtex/compatibility.rb. But I don't see the problem.

Iconv is a wrapper class for the UNIX 95 iconv() function family. I doubt it will work in a JavaScript environment. You should update the code as follows:

module BibTeX
  @iconv_replacements = Hash['ä', 'ae', 'ö', 'oe', 'ü', 'ue', 'Ä', 'Ae', 'Ö', 'Oe', 'Ü', 'Ue', 'ß', 'ss']

  # Returns +str+ transliterated containing only ASCII characters.
  def self.transliterate(str)
    str.gsub(/[äöüÄÖÜß]/, @iconv_replacements)
  end
end

[Leon0402]

I'm not sure why the Bibtex library is matching Han, Hiragana, Katakana, and Hangul unicode characters?

I have no idea either, but I replaced it and it at least compiles. We will see if it works in the end :-)

You should update the code as follows:

Oh right, yeah! That's an easy fix, thanks.

ERROR:  []: super: no superclass method `[]' for {"date"=>["accessed", "container", "event-date", "issued", "original-date", "submitted"], "names"=>["author", "collection-editor", "composer", "container-author", "recipient", "editor", "editorial-director", "illustrator", "interviewer", "original-author", "translator", "director", "reviewed-author"], "number"=>["chapter-number", "collection-number", "edition", "issue", "number", "number-of-pages", "number-of-volumes", "volume"], "text"=>["abstract", "annote", "archive", "archive_location", "archive-place", "authority", "call-number", "citation-label", "citation-number", "collection-title", "container-title", "container-title-short", "dimensions", "DOI", "event", "event-place", "first-reference-note-number", "genre", "ISBN", "ISSN", "jurisdiction", "keyword", "locator", "medium", "note", "original-publisher", "original-publisher-place", "original-title", "page", "page-first", "PMID", "PMCID", "publisher", "publisher-place", "references", "reviewed-title", "scale", "section", "source", "status", "title", "title-short", "URL", "version", "year-suffix"]}
    at Object.Opal.find_super_dispatcher (/home/leon/Projects/asciidoctor-thesis-template/node_modules/asciidoctor-opal-runtime/src/opal.js:1386:32)

This seems to be https://github.com/inukshuk/bibtex-ruby/blob/master/lib/bibtex/entry/citeproc_converter.rb.

Is something with Hash not supported?

[ggrossetie]

Your last error does not ring a bell. Probably related to this "advanced" syntax:

Hash.new { |_h, k| k }.merge(Hash[*%w[date issued].map(&:intern)]).freeze

Apparently, the star symbol is a "Variable Length Argument List, Asterisk Operator": https://stackoverflow.com/questions/4170037/what-does-the-star-mean-in-ruby 😆

This syntax is probably not supported by Opal. For reference, it produces:

{:date=>:issued}

I'm pretty sure we can use another syntax to get the same result.

[ggrossetie]

The following does the same:

CSL_FILTER = %w[
    date issued
    isbn ISBN
    booktitle container-title
    journal container-title
    journaltitle container-title
    series collection-title
    address publisher-place
    pages page
    number issue
    url URL
    doi DOI
    pmid PMID
    pmcid PMCID
    year issued
    type genre
    school publisher
    institution publisher
    organization publisher
    howpublished publisher
    type genre
    urldate accessed
  ]
  .map(&:intern)
  .each_slice(2)
  .to_a.reduce({}) { |memo, element| memo[element[0]] = element[1]; memo}

Maybe it compiles?