CesiumGS
Cesium Native PinBuilder / billboard support
One of the goals we've been aiming at with our work on #1154 is being able to load and display the data from the iTwin Feature Service, as shown in CesiumJS in this sandcastle. With #1163, we're tackling the polylines and polygons component of this task, but the points component remains unfulfilled.
To fulfill this, we need some way of:
- Loading points from GeoJSON (Done)
- Fitting those points to the terrain (Done)
- Putting some sort of billboarded graphic, selected based on data from the GeoJSON file, on the terrain at those points (TODO)
Ideas
To fulfill requirement number three, the remaining requirement, I see three different approaches we could take.
Do Nothing
This is the easiest approach. We just don't provide any particular support for billboarded graphics on the terrain. Users will find whatever way that works best for them to display graphics at these positions. With the blueprint support for GeoJSON data in Unreal, this is already possible to do. But this hardly seems like a good solution to me - every person who wants to accomplish this task (which seems to be a fairly common one outside of iTwin Feature Service as well) will have to reinvent the wheel.
User-Pluggable Point Prefab
We give the user some way of specifying that they'd like to spawn a given prefab, actor, or similar at the points obtained from a GeoJSON file. We could implement some way of passing the metadata from each point to each instance of the prefab/actor/etc (perhaps a base class in Unreal, or attaching a custom component in Unity) so the user can handle styling themselves. This still involves some degree of reinventing the wheel, but we could at least provide our own default implementations of these prefabs to help the users out. This is sort of a compromise between the "Do Nothing" solution, and...
Full Billboard Support
We take a cue from CesiumJS and provide an equivalent to the PinBuilder. The user tells us how we should display each point (for example, using an icon derived from a metadata property), and we generate textures from them to display on the map. This would probably involve implementing the KHR_billboard extension and generating a glTF out of the vector points. This would have the advantage of letting us use the existing texture loading paths for our game engine integrations, and we could benefit from instancing as the same point will likely be duplicated a number of times.
However, this solution doesn't feel very native runtimes. We usually lean on the side of giving the users the tools to leverage what the game engines already offer, rather than providing our own bespoke solutions to each issue. The number of people who want the exact same Cesium pin design for whatever tool they're making is probably far smaller than the number of people who would like the ability to plug in a prefab and go to town.
Conclusion
Out of these three options, I think the second is probably the way to go - it might involve more work for the user, but it'll let users do exactly what they want instead of fighting our tool to get it to do something close. We can help by providing basic implementations of these prefabs. It's also a lot less work than the third option! But I'm interested in hearing more about what people think of these options and whether there's alternatives I haven't considered.
Something I should add: while Unreal does have its own billboard component, it's not adequate for our needs - you can't change the origin that it's affixed to, so half of whatever you attach to the billboard tends to get stuck in the ground. You can fix this by applying an offset to the position, but as the billboard scales with your distance from it to remain at a constant screen size, a constant offset is not enough. We'd want some sort of custom billboard component for Unreal, at the very least.
Some additional context here: while currently we're only querying the first 1,000 results of the feature service and displaying that, what we'll want to do going forward is using the bbox parameter of the Get Features API to refine features in a similar way to raster overlay tiles. This is something we can currently do with the VectorDocumentRasterOverlay (or will soon be able to - I'm working on integrating it with the Get Features API now), but how we can tie this in with billboards is more of an open question. Some way to handle creating and destroying billboards in accordance with loading and unloading feature tiles is needed, I think.
The "User-Pluggable Point Prefab" makes sense to me. I think the only thing I might add is that this use-case is very likely to benefit from instancing. So you probably don't want a separate Actor / prefab for each point, but rather you want to add each point to an InstancedStaticMesh or some kind of "bunch of billboards Actor". At the Cesium Native level, it make just be a callback. Or maybe we don't need anything special, users just iterate over features.
At the Unreal level... well, we should provide some building blocks, like an extension to CesiumGlobeAnchorComponent that automatically keeps itself on terrain as the terrain refinement changes. A similar component that works on instanced meshes would be useful as well. A decent billboard rendering component would be very helpful to our users if it's not too difficult for us to do.
I don't think something like CesiumJS's PinBuilder, with its fixed set of icons, is worth doing at all. Emojis happened, making that whole thing painfully obsolete. But giving users an easy way to put text on a pin (including emojis) is very useful.
