meteor-peerdb
meteor-peerdb copied to clipboard
peerlibrary
Anyway simpleschema, collection2 and peerdb can work together
First of all, I want to say kudos for putting in the work to get a reactive layer on top of mongo (server-side) to handle declarative updates. Both collection2 and simpleschema solve specific issues, same as peerdb. I personally see peerdb more of a better package to replace collection-hooks but I wonder if peerdb api can be made to work with collection2 and simpleschema. I really see 3 concerns here:
Data schema integrity and validation -> SimpleSchema Collection data schema validation on insert, upsert and update -> Collection2 Provide Collection data hooks on insert, upsert, update and delete -> Collection-hooks || Peerdb
I will personally like to use Peerdb over collection-hooks because of its reactive nature and the ability to handle hooks even if the changes are made by another client. For me peerdb is a clear winner but I cant seem to feel like it will even be much better if it can work in tandem with SimpleSchema and Collection2.
My question is - how do I get the benefit of simpleschema + collection2 + peerdb. Is it be possible to integrate all 3 to work together. Will it be better to integrate peerdb definitions into simpleschema since both of them are in a way defining metadata information. Example, If would be great if we could do something like
//data validation provided by simpleschema
PostSchema = new SimpleSchema({
content: {
type: String,
optional: false,
},
author: {
type:Object,
optional: false
},
"author.username": {
type: String
},
"author.displayName": {
type: String
},
hits: {
type: Number
}
//etc
});
//peerdb hook through SimpleSchema
PostSchema.References({
fields: function() {
return {
author: Post.ReferenceField(Person, ['username', 'displayName']),
};
}
});
PostSchema.Triggers({
//trigger definition here ....
});
Post = new Meteor.Collection('posts');
Post.attachSchema(PostSchema); //provided by collection2
In the approve sample code, simpleschema handles validation (i.e. make sure author's username and displayname is a string), collection2 handle running validation and peerdb extension to simpleschema + collection2 maintains data references and hooks. This is just a thought but I think something along these lines will provide the community with 80% of what its needed for data manipulation. I want to use peerdb now but I also don't want to loose my schema validation checks on my collection (as well as on the client through autoform)
So how I imagine this to work is that I extend definitions of fields also to include non-reference fields and then validate documents in objectify. Once we have that we can also add a way to export schema definition as simple-schema, so that you can use it in meteor-autoforum. The issue I have by using simple-schema internally as well is that PeerDB support references, so schema returned will have to be normalized for simple-schema output (so subdocuments will not be seen as references), while objectify might be able to recursively go and check things because it will know where are subdocuments in fact instances of another document.
So I think that PeerDB replaces collection2, but can use simple-schema for compatibility with other packages.
@mitar Thanks for your response. You know best but I think integration with simpleschema will increase peerdb adaptation since its widely used for Meteor projects. Another place of concern (subjective) is the name and constructor api.
Name
When I saw peerdb, I thought it was a new reactive/ddp database like redis-livedata. Just like you said, its a better collection2 so might be better to give it a name relating to mongo/collections. Examples peer:meteor.collection or peer:collection. This will make it easy for people to identify its use at first glance.
Definition api
In relation to collection definition api, I love coffeescript but I think I'm right to say there are a lot more people pure javascript devs than coffeescrip. With that said, the current peerdb document definition api seems a bit ....verbose if written in javascript by hand. Below is a person document definition in pure javascript without coffeescript
var Person,
__hasProp = {}.hasOwnProperty,
__extends = function(child, parent) { for (var key in parent) { if (__hasProp.call(parent, key)) child[key] = parent[key]; } function ctor() { this.constructor = child; } ctor.prototype = parent.prototype; child.prototype = new ctor(); child.__super__ = parent.prototype; return child; };
Person = (function(_super) {
__extends(Person, _super);
function Person() {
return Person.__super__.constructor.apply(this, arguments);
}
Person.Meta({
name: 'Person'
});
return Person;
})(Document);
Compared to Person collection definition is Mongo.Collection or Collection2
var Person = new Meteor.Collection('Person')
Is it possible to make the metadata definition api as simple (hide the inheritance complexity under a simple api). Example. Peerdb collection/document can be defined using the same kind of definition style as Meteor.Collection/Collections2 and simpleschema such as ...
//definition
var Person = new Peer.Collection('Person',{
idGeneration: 'STRING' //options passed to Mongo.Collection
} ,
fields :{
username: {
type: String,
optional: false,
},
...
});
var Posts = new Peer.Collection('Posts',{
idGeneration: 'MONGO' //options passed to Mongo.Collection
} ,
fields :{
content: {
type: String,
optional: false,
},
author: {
type: Object
reference: ReferenceField(Person, ['username', 'displayName']),
optional: false
},
subscribers: {
type: [Object]
reference: [ReferenceField(Person)],
optional: true
}
}
}
//fetch
Posts.find({}).fetch()
//schema
Posts.schema()
This is all subjective. I just wanted to offer my thoughts as a dev and supporter. Keep up the great work.
I personally dislike SimpleSchema but I think @emmanuelbuah makes a lot of good points. I would agree the name could be better. Also while I love Coffeescript and I feel @emmanuelbuah is right people will want a choice between pure JS and Coffee.
Just for the record, name stays. :-) I have many projects starting with "peer" in this is one of them. :-) It is a family of related projects and while PeerDB currently resembles collection2, it has quite some other features in the future I am planning to implement. Like versioning. My goal with the project is to create an easy way to build collaborative tools on top of Meteor. And PeerDB is a layer for that. If others find some other uses to it, even better. But in my context PeerDB is a nice name for something which helps collaboration.
About JS integration: I would welcome a pull request with somebody making an alternative API which you can call with something like Document.create(...) or something, which is more suitable for JS programmers. And which would then internally extend CoffeeScript class and do the rest. Maybe it could be simply something similar to what you pass to Meta. So same function or something. (You need function to be able to define lazy references between documents. So a way how SimpleSchema works cannot really work once you have references.) So, some boiler-plate code for making it easy to use in JavaScript.
About SimpleSchema integration: I welcome also a pull request which would add to Meta a method which converts it into SimpleSchema compatible schema. So once you create a document, you can access fields for example with Post.Meta.fields, so there could be Post.Meta.simpleSchema(). This can then be used to pass to autoform and similar packages.
For that to work we first have to create also other fields, not just references. So the idea is to create similar classes for other types of fields, where then community can create even more specialized fields.
It would be interesting also to create fields which allow serialization/deserialization to internal MongoDB types from higher-level types (think IP field), but on the other hand Meteor does this through EJSON, so an idea would be that defining a PeerDB document would also based on fields define an EJSON type for direct serialization from JavaScript document. Just some ideas.
I am designing PeerDB to be pretty similar to how Django (or Mongoose) is doing things (class-based, having abstract class, using Meta and document managers). So if anyone like to implement parts of that (for example, field classes for various basic types found in MongoDB) in similar spirit, that would be a pull request I would be happy about. (So if you ever wonder how to design something that I would gladly merge, think from Django/Mongoose approach.)
I see above also another implicit suggestion, to allow other forms of ID generation. I think that could be made into another ticket and could become part of Meta definition if somebody wants to tackle. For now, you can always define class like:
class Post extends Document
@Meta =>
name: 'Post'
collection: new Meteor.Collection 'Posts', idGeneration: 'MONGO'
I think it should work, but I have not tried it. Feel free to make a pull request with a test. (We already have 0.5 MB of tests, so why not more.)
@mitar Yes. I think providing a way to pass default options to the Meteor collection (like idGeneration etc) will be a very good addition. I noticed your PeerLibrary project a few days back. I think its a great project. I actually browsed some of the other relating libs. I will spend sometime to digest your feedback in relation to where help is needed. I might be able to take some of the tasks.
I think providing a way to pass default options to the Meteor collection (like idGeneration etc) will be a very good addition.
I would not pass things directly, but provide configuration options for Meta.
I can confirm that the name "peerdb" had me overlook this project, because it sounded like an alternative db backend, not like bringing a missing completion for meteor collections.
BTW: The corresponding collection2 issue is here: https://github.com/aldeed/meteor-collection2/issues/31 (simple-schema definitions will get references/relations support)
Thank you for your name explanations above mitar. I think it created the understanding in me that peerdb is not at the database level, but above meteor collections, it provides "peer collections" or "peer documents" :-)
Concerning the new "meteor-middleware" package you mentioned above, please allow me to comment that its naming may be non-self-descriptive for the community as well. (Making it unfortunately and unnecessarily hard to find, notice and grasp.)
If i understant it at correctly, the package improves what meteor users know as publish functions, and provides stackable publish hooks? May this be described as "peer publishing stack" or simmilar?
I think the best way to find packages is to add links to them to wikis, reference them in mails, etc. :-)
Hmm, I think a good library name (same for functions etc.) is one that can give a new user an idea of what the lib does and handles (building on the conventions that already exist among potential users).
Note that my thinking of peer documents and a stackable publish hooks may totally describe wrong holes for the nails. I don't know if "peerdocs" or "peerpublish" -stack? is something that could be matching your thoughts of your packages closely enough.
So for me middleware is something intuitive because it is the way express works. And for me this is exactly what publish is in Meteor, a stack of things you want to send. Express sends HTML, Meteor sends documents.
Ok then, I understand where you're comming from. It easily happens to mix contexts, and confuse or misunderstand each other. I don't think midleware has a specific meaning in meteor yet, sure it describes other things in other contexts. From (my limited) meteor user side view your improving the publish workflow (possibly dramatically I can't tell yet).
(I kind of like many naming conventions in meteor also because they use words that have matching "real world" meaning like "publish".)
@testbird
I completely agree that the naming choices made for the peerdb, and meteor-middleware packages are non-descriptive. This indeed makes both packages unnecessarily hard to find, notice and grasp.
List of potential alternative names:
peerdocs peerpublish publishstack hookstack publishhookstack
@mitar any of these look better to you? :)
Back to topic, peoples' thoughts about the planned references/relations feature for simple-schema, collection2 and autoforms would be interesting. https://github.com/aldeed/meteor-collection2/issues/31
Maybe code could be shared to define/handle references/relations. And code could be shared to keep fields in sync. https://github.com/aldeed/meteor-collection2/issues/31#issuecomment-33363256 While the peer* packages can continue to provide its class based interface.
So there could be references with cached fields (with automatic hadling of syncs) as well as reference only fields (with automatic handling of publications). Forms to switch references, and forms that cascade down to edit referenced docs...
IMHO it would be great if collection2 and "peerdocs" could both provide read-time optimized, (de)referenced and cached, write-time sychronized embedded fields based on shared code.
@testbird @mitar @derekrazo In relation to what makes a good name, I tend to use the rule "Don't make the user think". A good name does an extremely good job of convey what the entity it represents is used for. @mitar I really like the work you guys are doing with peer library and believe some of the projects you have created will be very useful to the the meteor community as a whole. To make them reusable in other projects other than peer library, I will encourage they be treated as such which includes naming and implementation(decoupled from the peer library ecosystem/no strict dependencies). I think you guys are on your way to achieving both.
As to peerdb, I still think it would benefit from a name change. I like @derekrazo suggestion of peerdocs(or peercollection for peerdb) and peerpublish(for meteor-middleware) is great. They are good name that convey the use of both libraries as well as giving credit to the peer community.
I will make an executive decision: the name stays. I understand your concerns and I really appreciate that you care about the package and that you gave me thoughtful feedback, but I think it is a good name: it is short, quite unique (only ~3000 hits on Google), searching for "peerdb" already gives the first hit this repository, and as mentioned previously, it aligns with other components which are of interest to me. And I think it would already be too much work for everybody to change it.
Let's focus discussion to other things please. And let's just make this project even more popular and then the name will not matter.
(BTW: One issue here is that Meteor currently searches only Meteor package names and not also descriptions, see https://github.com/meteor/meteor/issues/2709, so this makes the package hard to find. Once that is addressed name will have less importance.)
All right. As you say, it was just feedback about the current search and comprehension experience, and it remains your decisionion how to make the best out of the current pre 1.0 meteor package renaming mess. Concerning the google paths, nevertheless, I dont't think it would be much of a problem, if one happens to find the old git project, it should just properly point to the new project page.
Ok now, brainstorming to working with collection2.
Both projects provide their database object abstraction layer. Both want to support relations/references among collections. Peerdb already has code to support references and maintain "write-time embedded cache documents" of referenced documents.
Peerdb and collection2 follow different API styles.
What could be the benefits to factor out and sharing code to maintain "write-time embedded cache documents"? Or even moving it into collection2?
- Peerdb bulding on tried and tested collection2 objects?
- Autoform support to properly render all the kinds of forms required to manage relations (switching/adding/removing links, editing referenced documents (cascading) for peer documents.
- Peerdb focusing on providing a different API based on collection2 objects?
- Benefitting from many allready existing and tested field and validation features.
- Eventually also benefitting from support for traditional referencing scheme (read-time fetching of references)?
???
Couldn't you make peerdb work "on top" of collection2s? Meaning one could define and use several collection2s, and peerdb package could use those collections, plus implements automatic syncing of references? (even if, for now, it would require defining the fields a second time in collection2)
If that is possible simple-schema/collection2 folks may be able to quickly support relation definitions in their syntax, by making use of the working auto-sync implementation from peerdb.
And the good part beyond sharing code is, once this works, collection2 folks could start working on the autoform support for easy handling of relations, while peerdb could focus on its own syntax for schema definition, possibly translating to the new relation-aware simple-schema internally.
Hey all, just now getting around to reading this thread. I hadn't looked at peerdb until now either. So...
- Haven't tried it yet, but peerdb seems to solve a number of issues. Nice job!
- With regard to simple-schema, the thing I hear most from those who use it is that they want to define the relationships in the schema object.
- In most projects, I think I would personally prefer to not have the ORM layer on top, but in a very large app with complex relationships, I think peerdb would be super helpful.
- I'm hoping to start c2 relationships development by creating a joins package within the next few weeks. It will be targeted at those simpler cases where you don't necessarily want the full peerdb Document approach. For starters, I only plan to support find/publish.
- Later, more advance features like the reactive cascading operations could possibly be added, and maybe some of the peerdb logic could be pulled out into a common package for us to share.
- I agree that we should try to let peerdb and collection2 work together as much as possible. I don't necessarily see how peerdb is a replacement since it doesn't validate against a schema, does it? It would replace
autoValuefunctions maybe, but not anything else. Possibly I could make a peerdb extension package that allows you to attach a schema to the underlying collection, too, using collection2. - I don't think it would be overly hard to make autoform work out of the box with peerdb, provided we figure out a way to attach a simple-schema to the underlying collection and expose it.
Hi Eric! Mitar also provides a package for publishing relations https://github.com/peerlibrary/meteor-related, its appoach is most straight forward from the packages I've seen. It simply allows to use publish functions within publish functions recursively. Interesting if you want to generate default publishers from schema definitions?
But IIUC, once relationships can be defined in simple-schema/collection2, there are two alternatives to impementing them. One is the once-at-write-time updated "embedded reference field cache" (peerdb) and the other is the at-each-read-read-time (in meteor plus at-publish&recomputation-time) "normalized" references fetching. Currently I can think for an app to decide towards normalized fetching only if the amount of duplication would be unbearable or the apps write-load is actually higher then read-load.
Mitar, on the peerdb project page you mention a limitiation for peerdb that I think I have not fully understood, yet: It says [peerdb syncing embedded fields] "requires that those relations are the same for all users". Isn't it possible to let the publish function only publish different sets of (synced and not synced) fields to different users? Since as for the user references in peer documents are regular fields of regularly published documents? Now I noticed the project page says nothing about actually publishing peer documents, so may that where I'm thinking wrong here?
It simply allows to use publish functions within publish functions recursively.
I must say that I have not tested it with more levels. If you did so and it works for you, please tell me. Or contribute a test case. :-)
(Mostly because I believe that things probably get very underperforming at that stage and you should probably rethink your approach to data design.)
Isn't it possible to let the publish function only publish different sets of (synced and not synced) fields to different users?
You can. But that is not the task of PeerDB. That's why there is middleware, so that you can easily filter things for particular users, and reuse such filters.
What I am trying to say there that PeerDB works on data without an user context. So when you are syncing relations, they are done for all users. Of course you can filter things out, but they will require that you store things for all users.
So it is best used for data which are somehow static for all users, static to data itself. For example, author of a blog post is the same for everyone, is part of data itself. And is good candidate for PeerDB. But a post field which would contain likes of all user's friends of that blog post is a bad candidate. It is not the same for all users. In that case you probably want to embed data for all users (which can become quite large), or dynamically resolve things (using related).
So for me all those three packages nicely play with each other and address various use cases.
Re meteor-related, I have been using publish-composite lately, but I think all the similar packages (there are 5 or 6 now) have good and bad points. I am planning to dig into them all more, figure out the differences and which is best, and then hopefully just use that package as a base and write the collection2 joins package on top of it.
As @mitar says, I think different relationships deserve different treatment. Sometimes a read-time join might be best while other times a write-time join might be best. My sense is that read-time joins are simpler to maintain and understand, and so I'd like for a collection2 rels package to focus on that but eventually provide a very simple "switch" you can flip to begin storing a property in denormalized and synced fashion (and ideally be able to "unflip" the switch too if you change your mind). This can get tricky and might be better left to a full scale solution like peerdb.
This can get tricky and might be better left to a full scale solution like peerdb.
I was thinking about a similar thing. To have some analysis of your relative queries and would then propose you to optimize data at write time.