Level
module system
rvagg:
I don't feel strongly either way about this. I'm not a strict minimalist but like Dominic I tend to use batch() programatically so a chaining API is less helpful (but I may use it).
What we really need is a proper plugin system for Node so a project like LevelUP can load optional plugins that may be installed in the NODE_MODULES path(s) and do things like expose the main LevelUP.prototype to plugins that may want to augment it. Then we could say something on our README like: "If you npm install levelup-batch, LevelUP will make the chaining batch() API available. And levelup-batch can monkey-patch LevelUP.prototype.batch to provide chaining.
I need something like this for Ender too so maybe I'll actually build a generic plugin helper system some day..
rvagg:
Re plugins, Grunt and DocPad do something similar already but certainly not in a modular way that you can pull out and re-use, we need something that smaller projects like LevelUP can easily include. We should collaborate on it so we get the API right!
- [x] it should detect collisions
- [ ] it should load all plugins from a specified folder
- [ ] it should extend the prototype
Will it do the extending automatically or should we say db.use('batch', require('cbatch'))
I'm thinking that package.json specifies the extension 'types' that the main project exports. So in the case of LevelUP we may have an extension 'type' of "prototype" and if the package.json includes that then LevelUP needs to expose it to the plugin system somehow which then passes it in a predefined manner to the plugin(s) that need it.
I think extending should be done automatically. How about:
db.require('cbatch')
or perhaps .use() is common practice?
db.use('cbatch')
So, levelup-batch has a "levelup": { "pluginType": [ "prototype" ] } in package.json, then the plugin system scans NODE_MODULES and finds this module with a "levelup" descriptor in package.json and adds it to the list of plugins. Then LevelUP somewhere does something like plugins.expose('prototype', LevelUP.prototype) (this is simplistic, you'd want to expose other stuff that wouldn't fit nicely into this pattern). and then somewhere else can do 'plugins.load(callback) which does all the exposing/extending stuff by working through all the plugins it found.
Make sense? It does in my head at least.
@ralphtheninja I want something that doesn't require LevelUP to be aware of the various plugins that it can load; that way you can write a plugin without LevelUP-core even knowing about it, you could even write a completely private one that never sees the light of day outside your company.
I think we could scratch the "pluginType" for now and just assume one way of doing it. Pick the simplest use case and make it work, then introduce other ways of extending.
automatic plugins will come back and bite us on the ass.
in my levelup stuff (see all my modules that are prefixed level-*) i initially used a db.use(plugin) pattern
but then abandoned that, and just went with plugin(db) the plugin is responsible for checking that it's not already attached, and to make sure that it only minimally alters the db object - I stuck with only adding one patch that has the same name as the plugin itself.
This method is rather ugly, but at least it's simple.
So, I built a bunch of stuff with this pattern, some plugins just patched the db object, but others actually inserted data into the database. This meant prefixing the keys of inserted data.
I did that enough, that now I'm thinking that a much better approach would be to abstract that all away,
go db2 = db.namespace('PREFIX') to get another db, and then do something like
map(db1, db2, function map(k, v, emit) { emit(k, 1) }) etc...
this could be much more flexible, and also cleaner - but there is a bit of thinking on how to get everything working together. Most of my stuff is based at some point on hooking into a put and turning it into a batch that inserts a job, or other keys atomically. it would be nice to cleanly isolate these sections, but i'd need also need to be able to insert into several sections atomically to get consistency which is the objective at the end of the day.
There's no need for a plugin systems. We already have commonJS
var myPlugin = require("level-plugin")
myPlugin(db, "do shit", function (err, result) {
})
Functions are really, really simple and work universally.
Now if you want a plugin system because you want to hook into put / del / batch etc, then we should have a single module that uses hooks and everyone uses that.
As for detecting collisions. We already have a namespaces enforced by npm. It'll be easier to use those.
@Raynos calm down and clearly explain what you think, and why you think it. I know you, so I know how you really mean it, but carrying on like this does not make for a pleasant and constructive discussion.
I'll be sensible.
Why do we need a plugin system? What benefit does it give us over just using require ? Can we enumerate the use-cases for plugins and the scenarios where they are useful, as I can't think of any.
we have a collection of modules, which are extensions to leveldb - they are firmly coupled, because they all interact with levelup some how.
We want them to be as loosly coupled as possible, so that it's possible to iterate on the levelup implementation without breaking them, if they depend on incidental implementation details of levelup then they will be fragile.
So, we already have a partial list here https://github.com/rvagg/node-levelup/wiki/Modules
And there are more that we want to build, but havn't gotten around to implementing yet. I think what this discussion really wants to be about is 'what is the interface that extensions should interact with?'.
certainly speaking for myself, getting the plugins that I wrote correct was quite difficult, and I think there is a lot of room for improvement here.
@dominictarr the interface is the same as the interface for levelup.
If we make a breaking change in levelup then
a) we need to update all apps that use levelup to use the new API b) we need to update all levelup modules to use the new API
We should be using peer-dependencies in our levelup modules to lock the version.
I agree that making a breaking improvement in levelup is a pain for module owners but that's how it works. This the downside of modular javascript, if you have many intrarelated modules then you need to update them all in one batch. I don't know any answer for this.
I have had issues with it with Raynos/signal-channel, Raynos/read-write-stream and @gozala has issues with it for gozala/reducers
Since I have being mentioned in this thread so here is my two cents. I'm not very familiar with levelup specefics so bare with me if my comment are little off:
Method chaining
From my personal experience I find implicit extensions more problematic than helpful, specially considering the weights they carry. In most of the cases libs tend do this just for the sake of method chaining but that problem is a different domain and should be solved independently IMO. For example reducers and all the libraries that work with it don't come with any method chaining support instead they embrace alternative drop in dsl libraries like:
- https://github.com/Gozala/enchain
- https://github.com/Gozala/synthesize
- https://github.com/gozala/gum
- https://github.com/gordonbrander/choochoo
As a matter of fact decision on which methods should be included in the chaining API is a domain of the app code using it. Making decisions at the library level is harmful as the non-core functions become second class and users tend to hesitate using them.
Updates
Making breaking changes in big ecosystem is problematic and requires quite an effort regardless. If work is distributed across diff libraries well you're facing same distributed architecture challenges... If you have a lot of small libraries that require updates on every breaking change, you'll end with some additional npm related boilerplate for updates but it's not being too bad for me & hopefully npm will get better at this too.
agree - so I think the way to go about this is to experiment rapidly, and then discuss the our experiments in detail - we are already well into the experimentation phase, hmm, I guess I should explain in detail how all my stuff interacts... I'll try and do that today...
Yes please @dominictarr, would be interesting to see how you're extending and what kind of extension points might be needed.
Slight digression:
While I'm sympathetic to the 'everything as a separate module' argument championed mainly by folks in #stackvm, I'm sceptical that it's actually the best approach to making libs friendly to those outside the Node-elite. The main problems I see for the average user are:
- discovery (!!)
- trust (i.e. can I trust this author's code without having to read through it myself? can I trust that it actually works with these other components?)
- understanding how the pieces can fit together
- making sure all the pieces continue to fit together through release cycles that don't match up
- documentation (it still amazes me how many awesome libs exist in the Node ecosystem that have garbage documentation and therefore remain out of reach to people who just want to get stuff done)
It's usually pretty difficult for us "authors" to think more as "users" even tho we do plenty of using; we're too used to being able to churn out our own solutions if we don't like what's already on offer (or just because we feel like it!). I don't believe it's reasonable to expect that of most people who are using Node. I suspect that the number of people using Node every day to build stuff is many times larger than the number of people who have packages in npm.
Sooo; that's the perspective that I'm attempting to see things from.
discovery
Discovery is a difficult problem. One way is the framework way, jQuery, ember, bla bla handle discovery by bundling everything in one thing and having all the docs & tutorials and shit in one place.
The downside of that is that it's not modular and you buy either the entire thing or not. That sounds very much like mongoDB and postgres. LevelDB is the most modular DB ever, I think it's worthwhile to embrace that and do a modular JS thing
maybe someone needs to write a book on the stackvm way?
Most books about functional programing embrace and cover modularity part in great detail. For starters I would really recommend SICP
modularity can mean a number of things, it doesn't necessarily mean that your modules need to be in completely separate places (i.e. individually downloaded from npm).
(note I'm not advocating bundling everything into LevelUP, that's the point of this discussion now; to figure out how we can foster an easy to approach ecosystem of modules & plugins around LevelDB in Node).
trust
Trust is also a difficult problem. For me I solve this by do I know the author or does someone I know recommend them. I also solve this by travis badge / testling badge / good docs.
understanding how the pieces can fit together
This can be solved with good blog posts / tutorials / wiki articles / talks about leveldb etc.
making sure all the pieces continue to fit together through release cycles that don't match up
Upto individual authors to make sure their shit still runs. If an author maintains a module it will continue to fit. If he doesnt maintain it then it does or another author maintains it.
documentation
I know how you feel. Both me and @dominictarr don't write enough docs. There can be some kind of levelup community curated list of modules where each module needs to meet a good standard of documentation / tests to be on the list. This may help or may cause a walled garden instead.
This module trust and discoverability problem is out of scope, there arn't gonna be 20k levelup modules, it's closer to the scale of connect-middleware -- whether or not you approve of the fact they exist --
To work as a community, we need to agree on certain things, like in node, we have Callbacks, EventEmitters, and Streams.
If your module doesn't follow that pattern - well, I'm not gonna use it, for one.
But, with leveldb, the module have a lot more in common, because it's specifically an embedded database.
The more we can agree on, the more we can make interoperate, but paradoxically, the less we need to agree on the easier it will be to interoperate.
this wiki page outlines what I want to use levelup plugins for... please comment if you have questions, or want me to clarify something