Better language detection

[gabegorelick]

Closes #22, #40, and probably others

[gabegorelick]

Is there a reason we don't just parse everything that's not JS as HTML? Currently, some tests expect files to be skipped if they have an unknown extension, but treating everything as HTML would simplify lots of things, and is probably just a NOOP if it doesn't have any SGML in it. Or am I missing something?

[rubenv]

I don't mind the current approach of manually adding test cases: it ensures that what we promise to support will stay supported.

[gabegorelick]

I don't mind the current approach of manually adding test cases: it ensures that what we promise to support will stay supported.

That's fine, but right now we have to manually add support for every arbitrary HTML templating language that pops up. What's wrong with treating everything as HTML by default, and then adding test cases as we go?

[rubenv]

Oh crap, it's been a year this has been gathering dust, doh!

If I understand it correctly, we try to detect the language correctly, with extensions being an override map?

That actually makes sense.

[rubenv]

What's wrong with treating everything as HTML by default, and then adding test cases as we go?

Sounds reasonable!

[gabegorelick]

If I understand it correctly, we try to detect the language correctly, with extensions being an override map?

That's the idea. We could maybe use a test to make sure overriding extensions still works if there isn't one already in the repo.

The detection discerns whether the source is a "markup" language, which covers most HTML-like languages. We also manually check the extension for "html", which catches a few other cases (hopefully that shouldn't cause any false positives.

[rubenv]

I tried merging this, but I can't seem to get it working again after resolving the conflicts.

First (solveable) problem is that it doesn't recognize JSP / Tapestry files. Easily fixed by adding an override in extensions (though there's probably a better way to fix that).

More bizarre is that I get these failures:

  1) Extract Merges multiple views into one .pot:

      AssertionError: 4 == 2
      + expected - actual

      -4
      +2

      at Context.<anonymous> (test/extract.js:30:16)

  2) Extract Merges duplicate strings with references:

      AssertionError: 4 == 2
      + expected - actual

      -4
      +2

      at Context.<anonymous> (test/extract.js:45:16)

Any idea why that might suddenly happen?