f2e-spec
f2e-spec copied to clipboard
alibaba
Text rendering: some relatively serious issues.
Do you want to request a feature or report a bug?
Bug
What did you do?
Sometime ago, people on Discord brought to our attention the fact they wanted to have highlighted lyrics change outline color instead of just fill color. This is a very reasonable request. This doesn't work and at the time I dismissed it as a relatively minor oversight but in trying to tackle the issue now I see (IMO) it's a pretty deep issue going to the core of how we render text.
From what I understand... we initialize OpenGLText objects, which constructs the actual text using cairo, then makes a bitmap and passes that to our surface loader. Now, when we sing, this doesn't happen again. Instead, we grab the bitmap, stretch it artificially at the syllable level (which is the cause of my having to implement a hacky mechanism to sort-of account for the increase in size and prevent syllables from colliding onto each other) and then use OpenGL to change its color. Since we already have our text made from before, we currently have no way to change the stroke color of the text.
I'm thinking maybe we need to refactor this so highlighted lyrics are actually rendered again with appropriate size, color and stroke, or if that would cause a a significant blow to performance, maybe find a better way to use OpenGL to accomplish this.
I'm not sure I'm savvy enough to get into this myself though... anybody care to weigh in and/or help?
text rendering code is indeed pretty messy, I once tried to add some asian style scrolling effect but I only got as far as filling an entire letter at a time in stead of gradually filling it like this: https://youtu.be/mNjif02658w?t=333
Basically what I'm saying is, we need to figure out a better way to compose/render text.
This looks... Interesting? https://github.com/rougier/freetype-gl/blob/master/README.md
it does...
lol the japanese text on their example reads: "I can eat glass but I can't find it".
summons @basisbit Do you have any experience/ideas from others projects you might be able to contribute to the discussion with?
Also... somebody pointed me to this: https://github.com/ocornut/imgui
Using that would be a MAJOR refactor of the code but it looks impressive IMO.
° I was summoned ° Not entirely sure what would be the best way for performous, but for another game I am working on, I was considering this: For every new sentence coming into view: use freetype-gl (or similar) to render each syllable to a separate texture for each desired style, then at every frame only fade in / out the currently sung syllable depending on percentage of how much of the syllable was sung. That will result into something similar to what can be seen in that above mentioned youtube video, and will be quite lightweight regarding memory / cpu usage.
Might be worth trying it. What we currently have looks like some very badly placed optimizations. Not to mention the rendering code now is much cleaner than it used to be back then.
@Lord-Kamina should this issue still be open? If so please clarify what still needs to be done, Thanks in advance!
No, I think I solved the larger issue. However, I still want to try my hand at a basic texture batching system at some point in the future. I was thinking, Even if we could draw all notes in just a single drawcall, we'd probably get a significant performance boost.
There is still too much space between syllables, especially so when using western style where they used to be full size normally and oversized when sung, before the asian mode was implemented.
Overall, I'd like to see some changes to this but cannot quite say what and won't be able to invest any time in development. The Pango-based rendering has always been too slow (not sure if still so on modern hardware and improved libs) and it never got the spacing exactly right. Also, there are problems with letter borders especially on more complex fonts.
The distance field method (which I gather is now implemented in freetype-gl) is probably still the best way to render high quality text in OpenGL. I've known this for a very long time but never had the time to implement it on Performous because back in the day the only existing implementation I knew of was a commercial Unity plugin.
I thinl I still need to correct oversized syllables potentially going off-screen. However, the spacing between syllables is fully intentional. Before I implemented that hack, syllables would all clip one another and it'd become an unreadable mess.
Re: pango drawing; yes it's kinda slow owing to the fact every syllable is a draw call but I wouldn't say it's something to really worry about in modern hardware. Once the syllables are on screen, enlarging and recoloring them in a shader as I'm currently doing is definitely the best method with regards to performance; if not to quality (we could re-render everything as we sing but that woule be ridiculous).
The somewhat reduced edge quality was a necessary casualty in order to implement being able to change the outline color as well as the font itself.
Re: distance fields; I looked into them also and the results are definitely great, but I don’t think they're a feasible option for us because IIRC, they require a previously generated texture atlas for the font and that's not a great idea for a game relying on user content.
Mind you, it might be the actual reason why I didn't try them was something else and I can't remember though.
Maybe something like this would work: https://github.com/Chlumsky/msdfgen
But I'm not entirely sure whether I'm up to the task of implementing that.
Still( even if we can do it in colors with this technique though, would it be feasible to implement the recoloring needed for the singing effect?
Rasterisation (rendering text or SVG to texture) used to be very slow and a few draw calls are really no concern compared to that. I gather that syllables are rasterised a few seconds before they appear, so that the game can run at smooth 60 FPS (rendering a whole line of lyric would take more than 16 ms but rendering one syllable was fast enough).
I suppose that distance maps could be generated runtime for each character found in content. However, a serious flaw with distance maps is that colour emoji cannot be supported and presumably singing game lyrics are likely to contain those if not now then in near future.