dataflow
dataflow copied to clipboard
meemoo
universal dataflow graph editor?
I have been working on the Meemoo graph editor for a year or so, and I'm pretty happy with it. I have been thinking that it would benefit my project if I abstracted out all of the graph editing code from the Meemoo-specific code. Then I was thinking that all of the dataflow js projects could potentially benefit from working on a common editor.
Things that I would keep:
- JSON graph format
- Save to localStorage / gist
- Selectable, draggable, droppable, resizable
- Cut, copy, paste, with x,c,v keyboard shortcuts
- Module library
Things to add:
- Polishing touchscreen UX (lots of potential and room for improvement here)
- Cleaning CSS, making it possible to look like Pure Data, Meemoo, or beyond using mostly CSS (+ some JS for SVG edge drawing)
- Abstracting all data flow stuff from the views
- Testing (:flushed:)
- Abstract out save local/gist as optional plugins, to make it easy to extend to other services, implement auth, etc.
Questions:
- Macros / subgraphs. How to represent this in the graph JSON?
- Stick with Backbone.js?
- Stick with jQueryUI?
- Best interface to define how the different versions of
source.connect(destination)is called from the GUI.- Named inputs/outputs
What do you think? Would this benefit your JS dataflow projects? Would you contribute to a common kickass graph editor?
@forresto - https://github.com/meemoo/iframework @sebpiq - https://github.com/sebpiq/WebPd @oampo - https://github.com/oampo/Audiolet @brianchirls - https://github.com/brianchirls/Seriously.js @dashersw - https://github.com/dashersw/pedalboard.js @cwilso - https://github.com/cwilso/WebAudio @bergie - https://github.com/bergie/noflo @idflood - https://github.com/idflood/ThreeNodes.js @daeken - https://github.com/daeken/Qtzweb
(Feel free to tag other related projects to this conversation.)
Sounds great to me !
What do you mean by "Save to localStorage / gist" ?
Things missing for me :
- connections should be selectable as well
- delete with "del"
- not only should the whole thing be stylable, but you should be able to configure if connectors are horizontal or vertical.
What are you using JQuery UI for ? Backbone.js is great ... let's keep it.
I'm using Backbone.localStorage to save graphs to the browser's key/value localStorage. If you change something on a Meemoo app and click [app], [save local] it will be there for you in the future.
[save public] uses Github's Gist API as a quick way to share graphs publicly. This saves anonymous gists.
These functions should be plugins, to add or change as needed. I'll add that to the to-do list.
The ports could be positioned horizontally, vertical, or anywhere with CSS. The way that I see edges working is a function that takes (x1, y1, x2, y2) of the i/o ports and returns the svg path's d attribute, like M 767 25 L 50 31. This could be a straight line, curved, or fancy edge routing around modules.
I'm using jQueryUI's slider, button, selectable, draggable, droppable, and resizable. I guess it would be lots of work to replace it. TouchPunch got me part of the way to touchscreen usability, but I need to do more work there.
yes! i've been using audiolet quite a bit, but i've been avoiding UI graph related stuff because it's a pain in the ass. i realized meemoo was right for this- but the rest of the meemoo suite is inapplicable. abstracting the graph stuff out would be really helpful. assuming most graphs operated using the same API (add a package manager into the equation and now we're having fun...). my unqualified opinions in no order:
backbone is in every one of my projects, but to be completely honest, removing the backbone and jquery ui dependencies would be nice. if you're willing to support modern browsers exclusively, reimplementing slider/draggable etc. would not be much work since they all have native equivalents.
a note on inputs: if i understand correctly, we're talking about the horizontal/vertical justification of the inputs (along the edges of each node)? if so, horizontal/vertical would not account for absolute inputs, eg. the ims 20. (ignore the fact these are routing internal components- i would like to route between nodes this way as well).
a note on css: just a thought... would be nice to have minimal or no css required at all, and write the most basic styles inline through javascript. requiring css makes it slightly less portable.
Since this is a full-on graph building GUI, I'll shy away from the no-dependency ideal to start. But I'll make the HTML/CSS as minimal as possible.
Absolute-positioned ports for sure. (The Korg example will be tricky, as the SVG for the wires could block click events to the modules below. But I'm sure there is a workaround. WebModular puts everything in a canvas.)
This project will build on the ideas in https://github.com/h5bp/lazyweb-requests/issues/82 ... JSON graph formats are easy enough to translate between, but some version of {nodes: [], edges: []} seems sane.
The big question for me is how to define macros/subgraphs. We need some way to import somebody else's graph (an audio or visual effect) as a subgraph in our own. WebPD has recieve and send nodes as a way to publish inputs and outputs for a particular patch.
I should mention @idflood 's ThreeNodes. ThreeNodes is already fully-featured graph editor, honed to the task of building three.js scenes.
How could we include a ThreeNodes module in a Meemoo graph, defining ins and outs so it could accept textures or numbers from other modules?
Well ... actually, this subgraph problem exists in Pure Data, and is an open question in WebPD. I wouldn't use [receive] to handle that. The way I think I will do is actually make the idea of subgraph disappear ... if you just open the subgraph and make it part of the parent graph there's no problem anymore... which doesn't mean that you should display, and doesn't mean that you should forget there's a subgraph. ... just not handle connections differently. Ex : (I don't know the format you use in dataflow) :
{
"graphs": [
{"nodes": [1, 2, 3], "connections": [[1, 89], ...]},
{"nodes": [89]} # -> a subgraph
]
}
I guess I'm confusing pd's send and receive with inlet and outlet. I think that any graph could define its default ins and outs, so that when it is included in another graph it automatically exposes those. You could then open that subgraph and change the variables that it is exposing.
+1 so interesting to a Web Audio API node editor as well. Something like "blockly to nodes"?
[send] / [receive] are basically just named cables. In PD you can indeed define subgraph's inlets and outlets, which are basically its ins and outs. What about subgraphs being nodes ? Nodes have inlets and outlets right ? So if 'Graph' is just a subclass of 'Node', then that could work !?
@automata : WebPD will soon be a Web Audio API node editor :) I'm planning to rebuild it fully on Web Audio API. Some objects (Oscillators, filters, ...) will only add a very very thin layer on top of Web Audio API nodes.
@sebpiq so let's work together because we are doing the same thing on Meemoo :-)
you mean that you plan on doing a web audio API editor for Meemoo ? Anyways ... now about the graph editor, I'm talking a bit for nothing ... I'd need to read the code first and then we can talk.
yes! sorry about the OT... https://github.com/automata/iframework/tree/gh-pages/src/nodes (the audio*.js files) I think we can have WebPd as Meemoo main "audio engine" instead of rewriting everything from scratch.
Well ... that would be awesome I say ! One of the most important things to me with WebPD was to build it as a DSP library, so there shouldn't be any problem for using it as an audio engine (at least no more problem than with any other JavaScript DSP library).
It makes sense to think of "graph" as a subclass of "node." In this case, if you import two instances of graph B into graph A and then edit one, the other won't edit. Sometimes this is what you want, and sometimes you want the two instances to be the same.
Maybe it could work both ways...
Unsynced:
{
nodes: [
{id: "b", graph: {nodes:[], edges:[]}},
{id: "b2", graph: {nodes:[], edges:[]}}
],
edges: []
}
Synced:
{
nodes: [
{id: "b", subgraph: "b"},
{id: "b2", subgraph: "b"}
],
edges: [],
subgraphs: [
{id:"b", nodes:[], edges:[]},
]
}
Then the question would be how to represent the difference in adding/editing the different kinds of subgraphs in the UX.
Why not something like :
{
root: {
type: "graph",
nodes: [
{id: "b", type: "graph", nodes: [], edges: []},// normal subgraph
{id: "b2", type: "abstraction", template: "mySubGraph"},// Subgraphs synced to the same template
{id: "b3", type: "abstraction", template: "mySubGraph"}
],
edges: []
},
templates: [
{id: "mySubGraph", nodes: [], edges: []}
]
}
"template" seems good to me. I don't think we need to separate the "root" graph out like that, what the advantage be?
No ... seriously that way the root graph is just a graph like any other, same attributes and so on. So then you leave the top namespace free for putting other stuff (like "templates"), which don't really belong to the graph itself. ... and it is just cleaner.
The is a very cool idea, and I'd definitely be interested. For it to work with Seriously.js, it would have to have a lot of hooks in and out with pretty fine-grained control. I would probably want to build some kind of wrapper between the graph UI and a Seriously node graph. And would want to be able to customize what goes in the view for each node. For example, I'd like to be able to put a thumbnail canvas to preview output for each node.
It'd be helpful to have good controls for inputs of different types, or at least the ability to build them and plug 'em in. For example:
- Number (with or without optional min/max/step)
- Color (RGBA, HSLA)
- Enum
- Boolean
- vector (2d, 3d, etc)
@brianchirls : I might be wrong ... but the way I see it, the graph editor would be completely naive, in the sense that it doesn't care what actually goes through the wires.
Sounds good! I've written some of my thoughts about the requirements for a flow-based programming editor in bergie/noflo#1
For what it is worth, here is the NoFlo JSON format: https://github.com/bergie/noflo/blob/master/examples/linecount/count.json
Corresponding "FBP notation" file in https://github.com/bergie/noflo/blob/master/examples/linecount/count.fbp
That sounds reasonably cool. Have you thought much about event triggers for modifications? I'm thinking...
- new node
- deleted node
- node connected
- node disconnected
- input modified (maybe node connected/disconnected is included in this one)
Each trigger should include only the data that was modified so you don't have to go traverse the whole graph again to look for changes.
@brianchirls you could go even further, and do eventing for things like:
- node crashed
- packet transmitted in a connection
Especially for the packets a "step debugger" mode would be useful where you would see and confirm each packet before it gets received by an inport.
@bergie That's interesting. But I would think most or all of these frameworks handle the packet transmission internally.
If you think of it as an MVC, then (Seriously.js|WebPD|ThreeNode.js|etc) is the Model and dataflow is the C. V is both dataflow and the canvas, speakers, etc.
But I could see where it'd be helpful to allow the model to tell dataflow when it's passing packets and allow an outside listener to be alerted when that happens. Or is that what you're getting at?
@brianchirls yep, not actually touching how data flows inside a FBP framework, but instead hooking to it using some sort of observer-kind of API.
Yep ... same here ! I'd need to update the visual of a node when data goes through. For example blinking light. But I think this is part of the bigger problem of "how is a dataflow node (node view) tied to an actual Computing node (model)" ?