Is there a way to tell tern to automatically load modules for later code completion?

[vincentwoo]

One currently acknowledged problem in tern is that completion for modules can take a long time. It would be cool if there were a way to say "digest these node modules so that future completions against those types will be fast". Right now the current experience is basically the first person to try to complete against a model basically gets timed out, but later requests will succeed. Would be nice to be able to prime the pump!

[marijnh]

You can do this, but only manually. Use bin/condense to condense the library, store the resulting JSON somewhere, and then add it to your "libs" setting. If you're using a module system, Tern should automatically figure out that it already has a definition for that module, and not try to load it from source.

[vincentwoo]

That sounds promising. Can you clarify a little bit? I thought condense was just for individual files. How would you condense, say, the lodash library (composed of many small files)? Is there a way to say "condense a module using normal node module resolution semantics"?

[marijnh]

You'd condense the 'index' module of the library, enabling the plugins needed (probably node) to make it able to include the rest of the library. If it has multiple modules that your code requires, you have to do this for all externally referenced modules, which isn't great.

[vincentwoo]

That thought occurred to me as well but didn't quite work:

coderpad@0e5e2b78b930:~$ ./node_modules/tern/bin/condense --plugin node node_modules/lodash/index.js
{
  "!name": "node_modules/lodash/index.js"
}

And the contents of index.js here is simply module.exports = require('./lodash');

It appears lodash.js is actually the main entrypoint for the module, but even trying to condense that doesn't seem to work:

coderpad@3dcbdad8abae:~$ ./node_modules/tern/bin/condense --plugin node node_modules/lodash/lodash.js
{
  "!name": "node_modules/lodash/lodash.js"
}

any thoughts?

[vincentwoo]

I should add that condensing single-file libs like underscore or q seems to work okay - it's just that perhaps it's lodash in particular. Is there something quirky about the way lodash does module loading detection that throws off tern? https://github.com/lodash/lodash/blob/master/lodash.js

The odd thing is that completing lodash seems to work in normal tern operation, albeit slowly.

[marijnh]

Might be something in the way it uses the exports object -- I don't immediately see anything that Tern doesn't support about it, but it's plenty indirect, so it might somehow end up tripping Tern up.

[vincentwoo]

Is it possible that tern behaves differently during normal vs condense operation on expressions like:

  /** Detect free variable `exports`. */
  var freeExports = (objectTypes[typeof exports] && exports && !exports.nodeType)
    ? exports
    : undefined;

  /** Detect free variable `module`. */
  var freeModule = (objectTypes[typeof module] && module && !module.nodeType)
    ? module
    : undefined

[vincentwoo]

I did some more experimenting, and hopefully this can help people out in the future. Perhaps this info can be incorporated into the manual too.

Let's say you're trying to cache the module definition for underscore. That is to say, you want var _ = require('underscore') to be fast.

If you look at the provided definition file (initially added by @sqs), you'll see that it begins with:

{
  "!name": "underscore",
  "_": {
    "!doc": "Save the previous value of the `_` variable.",
    "!type": "fn(obj: ?) -> +_",
    "VERSION": {
      "!type": "string",
      "!url": "http://underscorejs.org/#VERSION"
    },
    "after": {
      "!doc": "Returns a function that will only be executed after being called N times.",
      "!url": "http://underscorejs.org/#after",
      "!type": "fn(times: number, func: fn()) -> !1"
    },
    "all": "_.every",

This incarnation doesn't seem to help tern complete _ = require('underscore'), probably because it doesn't work with node module resolution. How resolution interacts with definition files is unclear from the manual.

Continuing on, if you try to produce such a definition on your own, with condense --plugin node node_modules/underscore/underscore.js, you get something like:

{
  "!name": "node_modules/underscore/underscore.js",
  "!define": {
    "!modules": {
      "node_modules/underscore/underscore`js": {
        "prototype": {
          "<i>": {
            "!type": "fn() -> !this._wrapped",
            "!span": "50952[1492:18]-50956[1492:22]"
          },
          "value": {
            "!type": "fn() -> !this._wrapped",
            "!span": "51950[1523:14]-51955[1523:19]",
            "!doc": "Extracts the result from a wrapped and chained object."
          },
          "toString": {
            "!type": "fn() -> string",
            "!span": "52205[1531:14]-52213[1531:22]"
          },
          "toJSON": "!modules.node_modules/underscore/underscore`js.prototype.value",
          "valueOf": "!modules.node_modules/underscore/underscore`js.prototype.value"
        },

However, tern complains about this definition on load:

/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:85
      throw new Error("Unrecognized type spec: " + this.spec + " (at " + this.pos + ")");
      ^

Error: Unrecognized type spec: fn(func: fn() -> ?, context: !modules.node_modules/underscore/underscore`js.bind.!1) -> fn() -> ? (at 50)
    at Object.TypeParser.error (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:85:13)
    at Object.TypeParser.parseFnType (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:103:33)
    at Object.TypeParser.parseTypeInner (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:160:21)
    at Object.TypeParser.parseTypeMaybeProp (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:146:25)
    at Object.TypeParser.parseType (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:130:23)
    at parseType (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:262:59)
    at passTwo (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:424:9)
    at passTwo (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:442:11)
    at passTwo (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:442:11)
    at doLoadEnvironment (/Users/vwoo/ubuntu/autocomplete/node_modules/tern/lib/def.js:480:38)

If you don't supply the --plugin node flag to condense, you get:

{
  "!name": "node_modules/underscore/underscore.js",
  "!define": {
    "exports.forEach.!0": "[string]",
    "exports.forEach.!1": {
      "!type": "fn(value: ?, index: number, list: [?])",
      "!span": "11651[336:18]-11896[342:7]"
    },
    "exports.collect.!0": "[?]",
    "exports.collect.!1": {
      "!type": "fn(obj: ?) -> !0.<i>",
      "!span": "4262[125:11]-4329[127:5]"
    },
    "exports.collect.!ret": "[number]",
    "exports.detect.!1": {
      "!type": "fn(obj: ?) -> bool",
      "!span": "44983[1307:11]-45040[1309:5]"
    },

This, along with a

  "libs": [
    "underscore"
  ],

clause in your .tern-project, appears to work.

[vincentwoo]

However, problematically, sometimes you can't condense without supplying the node plugin:

vwoo:autocomplete vwoo$ ./node_modules/tern/bin/condense node_modules/q/q.js
{
  "!name": "node_modules/q/q.js"
}
vwoo:autocomplete vwoo$ ./node_modules/tern/bin/condense --plugin node node_modules/q/q.js
{
  "!name": "node_modules/q/q.js",
  "!define": {
    "!modules": {
      "node_modules/q/q`js": {
        "nextTick": {
          "runAfter": {
            "!type": "fn(task: fn())",
            "!span": "7556[239:13]-7564[239:21]",
            "!doc": "runs a task after all other tasks have been run\nthis is useful for unhandled rejection tracking that needs to happen\nafter all `then`d tasks have been run."
          },

[iensu]

Hi! Where should I store the condensed json files in order for tern to pick them up?

[marijnh]

It doesn't automatically do that, but you can put it anywhere and put a relative path from the project dir to it in your libs option.

[iensu]

Ah I see, thank you!

[chmln]

IMO it'd be great if tern did this by default. There's always that ~second delay when getting suggestions for the first time in a file.