Custom Horizon commands

[deontologician]

We should allow creating custom commands that map to parameterized reql queries.

There would be two parts:

1. The config to add a custom command
2. Adding the method to the horizon object so it can be used

The server and client would need to be changed

Example config:

[commands]
name = "countLabels"
params = [ "x", "y" ]
reql = "r.table('foo').getAll([ x, y ], { index: 'x_and_y_compound' }).count()"

On the frontend, you may use it like:

horizon.countLabels({x: 12, y: 13}).fetch()

If you use fetch, it would run the query directly. If you use watch() it will append .changes() to the query and then run it. The query would simply fail if it doesn't support changes.

This would add easy extensibility, where instead of going from easy Horizon straight to roll your own backend, you can go to RPC reql queries as a nicer intermediate step.

[niieani]

That's a really great idea! It might also be cool and even more powerful to pair it with the comment-parsing / annotation-parsing suggested and commented on here. A dev could simply stick a ReQL query where Horizon queries fall short, defining the custom command directly above the call-code (in a comment or annotation).

For future-proofing / forwards-compatibility it might make more sense to provide a single API for custom commands, like:

horizon.command('countLabels', {x: 12, y: 13}).fetch()

[deontologician]

I think theoretically we could have the server send custom supported commands and arities in the connection handshake. So it should be possible to integrate them a bit better dynamically

[segphault]

I've been thinking about supporting something like this in the admin user interface, so you can graphically add new endpoints to Horizon—similar to what I've already done in Basejump. I think it'd be a great feature, and I think being able to manage it through the admin interface would be fantastic.

[marshall007]

This was previously discussed in #12. I think part of this should include exposing horizon.command(...) (or similar) on the server so commands can be added programmatically as well.

[marshall007]

I also think it should be possible to write non-ReQL custom commands, which is where the programmatic API is more powerful. This should probably include the ability to implement a "watch" for such commands. For example, you might poll some third-party service or a .count() query on the server and still treat it as a typical .watch() on the client.

[stellanhaglund]

Hi, I'm hoping I can give some feedback on this.

I've been trying out add_request_handler on a server using horizon as a package. And its truly awesome.

Right now im using it to both do the first query that i mostly setup as my "trigger" for to when new data should be sent to the client, mostly using .changes()

The really awesome thing that after that i in some cases have four new queries after that changeFeed placed in a custom subscription handler, and when all this data is collected i then process it using moment.js for charts, i do joi validations for inserts, so on, so on.

So i read the suggestions for a ui and stuff like that and its all good, but please keep this way of doing it open.

I understand your best case scenario would be to cover as many suggestions as possible and thats great but i see the greatest potentials in this.

Just this one thing i did last week, i did an endpoint for a simple insert, but to it i added joi validations to the complete object with custom error messages, than if something was wrong i just return an object like {type: 'error', msg:' joi error message ' }

So now i can benefit from the speed of sockets delivering my error msgs i do server side validation and as soon as i have some auth running it will all be protected by jwt.

Also a previous suggestion is to allow non query calls, that i want to add to.

Just an option to returning false instead of an query in add_request_handler would be such a great thing, then the query could be skipped and return an object to the response handler like {type:'bypass'} for example then i could do what i want in the response instead. Also this would allow me to even do joi validations on the request data before a query is even runned wow. From what i see in the code this addition wouldnt be that hard but would make a big difference.

An example for this is that i have a system setup with lots and lots of filters built up as trees based on values of the documents and lets say its 5000 documents. I could do the inital query return both the documents and the tree for the filters, but then if the user changes one of those filters i want to skip the query and just do my processing and send back what should be removed from the document list. Authenticated server processing and db queries using sockets using a simple client library.

And just think what these endpoints actually could be capable of, since connecting to them requires an handshake that is in some way is authorizing the user.

If i could also do some customizations to the actual emits i could for example add possibilites to have pausable file uploads/downloads that is always automatically authenticated.

If there is or will be a node lib for the client, all communication between my servers could be authenticated through the db and be using sockets just by those endpoints.

So to sum it up. Please keep a way to do it like i do it now. Add an easy way to skip the initial query. Maybe even allow some control over the socket events created by using add_request_handler.

[Tryneus]

@stellanhaglund - I've actually reworked how endpoints are created in the permissions branch (see PR #331). Adding a request handler is the same, but the endpoint must expose a different interface.

At the moment, that is a single function:

const run = (raw_request, context, ruleset, metadata, send, done) => ...

• raw_request is the resulting object from the parsed JSON, which has been validated as a request, but not as the specific request type to be run - that is left up to the endpoint implementation.
• context is the user info on the connection attempting to run the request - at the moment this is pretty much just the user ID and the groups the user belongs to. This should be automatically updated if the user info is ever changed in the data layer.
• ruleset is the set of rules that match the raw_request at the template level (for permissions). Individual reads/writes would still need to pass the validator of at least one of these rules. This should be automatically updated if the user's groups are changed, or when rules are added/removed/modified.
• metadata is the Metadata object from metadata.js, which allows for resolving table names and getting the current ReQL connection (among other things)
• send is a callback with one argument (res), which sends a message back to the client, and can be called multiple times
• done is a callback with zero or one argument (res), which indicates the request is complete, and it will be removed. res may be a normal message or an Error, in which case an error response is sent for the request.

So now endpoints are in charge of managing the lifetime of their own ReQL queries and cursors, but this is far more flexible in allowing access to different APIs or types or requests. Let me know if this causes problems for you, or if you have any suggestions for this interface.

[stellanhaglund]

@Tryneus okay so something like this?

server.add_request_handler('myendpoint', (raw_request, context, ruleset, metadata, send, done) => {
    // do my stuff
}

do I need to add some kind of groups, permissions and rules?

saw you have slack a profile, started a conversation there if you have time.

[deontologician]

This feature should be implemented as a plugin, see #588

[danielmewes]

It might be worth making this an integrated part of Horizon in my opinion since easy extensibility is a central feature of Horizon. Though an initial implementation could be done as a plugin I suppose.

[deontologician]

Just for modularity's sake we should probably split it out, since not everyone will want this. It also seems like it wouldn't require more deep integration into the horizon guts than a plugin can provide, so there's not really much reason to couple it more tightly

[jeffijoe]

I think this makes sense in core.

[michaelwills]

I agree that this should be in core and (potentially?) automatically reloaded on change with a flag or with --dev. Nice to have, definitely not need to have. :)

I am new to Horizon and after the initial wow factor wore off, the real wow factor kicked in. That's when you start asking "Oh and can I…?" questions and I hit limitations. I am about to combine horizon and rethinkdb so that I can keep the easy wins of Horizon but add rethinkdb code and other services.

I really like the potential of @marshall007's suggestion that non-ReQL commands could be used and the proposal in #345 might allow for arbitrarily complex commands per development needs but still allow good organization, etc.

Really good stuff happening here!

[michaelwills]

I don't yet know enough of horizon/rethinkdb internals but is there a way to start experimenting with these concepts the way @stellanhaglund mentioned?

I've been trying out add_request_handler on a server using horizon as a package. And its truly awesome.

[edit]

@stellanhaglund @Tryneus Just confirming if this is what's needed:

const horizon_server = horizon(http_server, options);
horizon_server.add_request_handler('myendpoint', (raw_request, context, ruleset, metadata, send, done) => {
    // do my stuff
})

meaning we do have a way to do this sort of thing now without modifying the core.

[deontologician]

Definitely, check out horizon.io/docs/embed

On Sun, Jul 17, 2016, 04:32 Michael Wills [email protected] wrote:

I don't yet know enough of horizon/rethinkdb internals but is there a way to start experimenting with these concepts the way @stellanhaglund https://github.com/stellanhaglund mentioned?

I've been trying out add_request_handler on a server using horizon as a package. And its truly awesome.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rethinkdb/horizon/issues/337#issuecomment-233177584, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAFVo465ayKy7k10RiVVw90w-gJUQvTks5qWhLfgaJpZM4IaldJ .

[rnenjoy]

It would be truly awesome if i could use the horizon socket system to make an call to my backend, and not only being able to do advanced reql queries, but doing anything. I could do validation with joi, throw errors, connect to other systems etc.

[tony-garcia]

Exposing a way to run custom ReQL queries through Horizon would be huge. My initial excitement over Horizon was tempered when I quickly ran into the limitations of the current Collections API. The standard response to this that I've seen is to embed Horizon into, say, an express app and implement custom endpoints for doing more complex ReQL. But then that leaves me with two different ways to access my data, which I don't like one bit. I've been looking into other solutions including this, but I would love to be able to just use Horizon.

[sdebaun]

Definitely, check out horizon.io/docs/embed

@deontologician I don't see any mention of add_request_handler there? im still confused as how you can do pagination with this.