Pipeline tiddlers

[hoelzro]

Lunr.js allows you to hook into the pipeline it uses to massage documents in order to index them; currently I have a single custom pipeline function that expands synonymns in queries.

It might be helpful to expose the pipeline to external clients, such as other plugins - this way we can prototype new ideas outside of the plugin itself, and it makes the plugin more flexible.

I'm thinking the plugin could load modules with module-type: lunrpipelinefunction, and a configuration list variable could specify the ordering of the pipeline functions. A few questions about the configuration, though:

• How do we make sure the pipeline includes the default functions if it's a freeform list? Do we need to make such a guarantee?
  • You might not want to - think of replacing the default stopwords/stemmer with non-English variants!
• How do we keep well-meaning but non-advanced users from hosing their wiki by throwing in too many pipeline plugins? Even for knowledgable users, arranging the functions in the pipeline is somewhat of an art!

A preview pane of how the pipeline would massage data would be very cool, but probably a bit beyond the scope of this issue.