corona
corona copied to clipboard
coronalabs
Custom objects, step #3: effect data types and shell transforms
This is PR 3 as first described here, with some additions.
As with PR 2, this again modifies Rtt_ApplePlatform.mm and Main.cpp, since it also extends the API (quite a bit!). Those new functions, along with some improvements, have been added to CoronaGraphics.*.
These changes supersede those of PR 2, but it still ought to be fine applying these PRs in sequence.
Under the hood, Solar builds up geometry that gets drawn in batches based on some related state. At the moment, this always consists of vertices in one specific format.
Any such batch always has a couple shaders associated with it. These describe how to transform the vertices and then paint the space they cover. Solar developers are able to write vertex and / or fragment kernels to customize this process.
As the "kernel" terminology suggests, there is also a "shell" in which these are embedded. This is a bit like a common require() for the kernels, providing some well-defined structure and capabilities.
Comparing the aforementioned vertex format and these shells, one can see that bits go unused. The position's z-coordinate, for example, might as well not exist. If we're not using distorted rects, ditto the texture coordinate's q component. Often we don't need the vertex userdata, either.
The general idea behind shell transforms is to allow some rewriting of these to tap some of this unexplored power. (Other possibilities might open up later, such as customizing sampler types.) This is unfortunately a bit of an expert's domain, but I figure a few well-defined transforms (those in the test plugins, say) will become standbys and be gathered into modules.
This is available in Lua through the graphics.defineShellTransform() function, which provides find-and-replace and find-and-insert-after options for the shell text. For more esoteric needs, the more low-level CoronaShaderRegisterShellTransform() native API is also available.
After a transform is defined, we can provide its name to graphics.defineEffect() and it will be used during compilation.
Effect data types come at things from a different angle.
This largely corresponds to the object created when we do object.fill.effect = "filter.custom.myEffect": we have something with properties we can read and write, the associated object will be drawn a certain way, etc.
When we define an effect data type, we give it some set of out-of-the-ordinary behaviors: properties not among the vertexData or uniform userdata, for instance, so effectively built-ins of "this sort of effect". We can also adjust the draw behavior, say to only do so conditionally, do so multiple times, modify the geometry batch, etc. Some other steps along the way are also available. We can share resources among these steps, too, e.g. to let a property drive the draw.
The CoronaShaderRawDraw() API is provided to do our own draws if we override the built-in draw behavior.
The CoronaGeometry* APIs are intended where draw calls want to modify the to-be-drawn vertices.
Data types are created by CoronaShaderRegisterEffectDataType() and installed by name with graphics.defineEffect() (they are orthogonal to shell transforms, so it may take either or both). Very little of what these do makes sense in Lua, so this is strictly a native API.
The CoronaCommandBufferWriteNamedUniform() API gives us some support for vertex kernel uniforms beyond the u_UniformUserdata* set. Mainly, these were intended to open up some possibilities with arrays, e.g. for contiguous data like curves or things like skinning weights. These don't participate in batching, so also allow a modicum of sharing among objects sharing the same effect. This builds on a previous attempt of mine, but now much more naturally piggybacks on the effect data type mechanisms.
Calling system.getInfo("maxUniformVectorsCount") will tell us (roughly) how many vectors are still free for our own use in a vertex kernel. This queries OpenGL's limit, then pessimistically assumes every uniform found in the shell gets used and the compiler completely fails to pack them together. 😃 The result is the limit minus that count.
The CoronaCommandBufferGetBaseAddress() API can be used in command writer callbacks to remember where we wrote, but making this robust against reallocation by then finding the relative offset. This way, we can reliably refer back to written resources in reader callbacks and the like. Named uniform arrays were the inspiration here but certainly aren't the only use case.
Some matrix APIs are also provided. These really target the next PR, but this way the two platform files might not need to be changed again.
As before, the comments in the native API should about represent the future docs. The system.getInfo() and graphics.defineShellTransform() bits are still pending.
I tested using this plugin, with three samples.
The first of these, "basic", uses both the shell transform and effect data type machinery.
After assigning the sample's effect to an object, we can set an instances property to some integer > 1. In that case, the overridden draw logic will show the object several times, although in between the z-coordinate of every vertex gets updated: 0 on the first draw, 1 on the second, etc.
The above is provided by the data type, whereas the shell transform makes the z-coordinate available in the shader.
The sample sets 3 instances, so the draw operation adds our rect geometry (with minor alterations) three times to the same batch. The shader reads the instance index—z—to modify each rect's position slightly, and likewise their colors.
When defining the effect we also pass the array { supportsInstancing = "z" } to the its details key. Our callbacks will see these values and can respond to them in some way. For instance, in this plugin we could overwrite some other unused member than z, like q or uy.
In the second sample, "shared", you can click or tap to lay down points. Once four or more of them exist, curves will be drawn passing between them (n.b. this is a Catmull-Rom curve, so the first and last points are just controls).
Again, this uses the effect data type feature, quite heavily in fact.
We can create a memory buffer via the plugin, add it like object.fill.effect.buffer = buffer, and populate it with buffer:setVector(index, x, y, z, w) calls. This is first used to fill a fixed part of the buffer with some shaping information. The rest is then packed with position and tangent information, gathered from our points.
For each segment, we make a new grid-shaped mesh and assign the effect to it. We also give it an instance index, although here we simply use boring old vertex userdata. :smile:
A lot of work goes into checking if the buffer itself is out of date or the object's effect is in sync with it. (This should also account for adding / removing masks too, although I haven't tested that yet.) If the object is caught up, nothing need be done. Otherwise, we issue a command to record the buffer contents; any subsequent syncs using this buffer would need this same data, so we just save its relative offset and pass that around. An end-of-frame handler clears some of the bookkeeping bits, after rendering finishes.
When we do update the buffer, we're writing to a uniform array, as declared in the vertex kernel. Writes are done in a bind callback, once we know we have the proper effect in place.
The gist here is that curves are continuous: a Hermite curve segment will access two position / tangent pairs: the start and end points. However, the next segment will start at this end position and end with the next pair in line. Arrays, together with instance indices, give us a way to take advantage of this sort of adjacency, grabbing the pairs we need.
Since adding a new point is likely to require a redraw, the CoronaRendererInvalidate() API is used to ensure a refresh of the display, even though the object's display properties haven't been changed.
The third sample, "unshared", has some surface similarities to "shared". We put down points in the same way and draw curves as a result.
However, we now simply append points, so we'll eventually have too many to fit in the available uniforms. (Well, for testing, it's just kept artifically low. 😄)
Since we can't rely on having everything comfortably loaded, keeping in sync goes out of the window, along with sharing. In this case, the buffer comes hard-wired into the effect and we read it back from the buffer property.
A set of points that would overflow the uniforms is split up into several draw calls. This is done using the same instancing idea as the "basic" sample, and also adopts its shell transform. These calls are also done as separate batches, since the proper subset of uniforms must accompany each draw.
There's a little subtlety in performing the uniform writes, too, since the shader might already be bound (same shader used by the previous object in the hierarchy), rather than during the first draw, whereas with the second draw and thereafter it's a done deal. :smile:
I added undefine features to the effect data types and shell transforms, in keeping with the idea of undefining effects (although orthogonal to that PR). Since the data structures are "plain old data" the ShaderResource simply clones the value on assignment and takes ownership of this copy.
Some work on PR 5 (WIP here) obviates a couple small points above. The introduction of "dirty state" subsumes the end-of-frame handler; it also gets handled in the renderer's Insert() logic, so some of the timing concerns about uniform writes are moot.
I don't quite know the best way to update this, e.g. drop what I can in PR 1 and PR 2, or just do it all in PR 5? At any rate, I'm not expecting the PR to be pulled any time soon, so I did update the test plugin.
I'm wondering why the vertexData is not per vertex different. If so we just can use it to custom our per vertex data.
@zero-meta Well, obviously the per-object case is convenient for all the built-in effects, and indeed most user-defined ones. So it grew out of that need, and the more general way didn't have any immediate use cases.
I did consider what you mention back when I implemented per-vertex colors, but never pursued the idea.
With this PR you can achieve it through effect data types. The plugins I include should give some idea of what this entails.
With #5 you can also augment an object's vertex format. Then you'd just feed another stream of inputs in. It resembles, say, Love's attachAttributes, though I only found about that later.
@Shchvova and I have several of these PRs rolled up into an experimental branch. Just waiting on a build now.