MizunagiKB
Proposal for Improvements in GDCubism next version.
As we develop GDCubism v0.7, there are two areas we are considering for improvement.
If you are using GDCubism or are interested in it, please feel free to comment.
Comments can be in either Japanese or English.
Specifying the Size of Mask Rendering Textures
GDCubism focuses solely on displaying Live2D models, which results in significant GPU memory consumption. Especially when displaying at 4K (3840 x 2160) resolution, the memory usage quickly reaches several gigabytes.
This is because the mask images created during the rendering of Live2D models are generated at the same size as the final output, and since the mask images are realized with SubViewport, the more masks used in a single rendering, the more GPU memory is consumed (the back buffer cannot be reused).
While higher quality masks result in smoother edges, masks with a resolution higher than the actual output resolution do not seem to add much value. Therefore, we are considering allowing users to change the size of the mask images.
For example, even if rendering at a size of 4096x4096, it would be possible to set the mask to 256x256.
As an item, we plan to prepare a Vector2i item where (0, 0) would maintain the current behavior, and sizes of (1, 1) or larger would generate the mask image at that size.
Rendering Without Using SubViewport
The GDCubismUserModel used to render Live2D models in GDCubism is a class that inherits from SubViewport. This is to handle multiple MeshInstance2D as a single image, but we are considering whether it might be more convenient to separate this.
To achieve the goal of "displaying Live2D models," there seems to be a somewhat high barrier, and if the user understands SubViewport, they could place SubViewport as the parent of GDCubismUserClass themselves. (Combining purpose-specific Nodes rather than creating a highly functional and large Node seems to align more with the Godot Engine's philosophy.)
If SubViewport is not used, it would likely be necessary to adjust the mask to screen coordinates, which may require shader modifications.
GDCubism v0.7 を開発するにあたり、改善を考えている箇所が二箇所あります。
もしGDCubismを利用されていたり、興味がもっているのでしたらお気軽にコメントをお願いします。
日本語・英語、どちらでもコメント可能です。
マスク描画用テクスチャのサイズ指定
GDCubismはLive2Dモデルを表示する事だけに注力しているため、GPUメモリをかなり消費します。特に4K（3840 x 2160）解像度での表示を行うとすぐに数GBに達してしまいます。
これはLive2Dモデルの描画時に作成するマスク画像を最終出力サイズと同じサイズで生成していることと、マスク画像をSubViewportで実現しているため、一度の描画で使用されるマスクが多いほどGPUメモリを消費するためです。（バックバッファを使いまわせない）
マスクの品質が高い方がエッジが滑らかになるのですが、実際の解像度以上のマスクは出力結果にあまり意味をなさない気がするため、マスク画像のサイズを利用者側で変更出来る様にすることを考えています。
例えば4096x4096のサイズで描画していても、マスクは256x256にするといった事が可能になります。
項目としては、Vector2iの項目を用意して(0, 0)の場合は現状と互換の動作、サイズが(1, 1)以上であれば、そのサイズでマスク画像を生成するといったものにしようとしています。
SubViewportを使用せずに描画を行う方法
GDCubismでLive2Dモデルを描画するのに使用しているGDCubismUserModelは、SubViewportを継承したクラスにしています。これは複数のMeshInstance2Dから生成されたものを一枚の画像として扱うためなのですが、これを分離してしまう方が便利なのではないかと考えています。
「Live2Dモデルを表示したい」という目的を達成するには、やや高い壁がある様な気がしており、SubViewportを理解しているのであれば、利用者自身でGDCubismUserClassの親にSubViewportを配置すれば解決出来るだろうと考えます。（高機能で巨大なNodeを作るよりも目的特化Nodeを組み合わせる方がGodot Engineの流儀にもあっている気もします）
SubViewportを使用しない場合は、マスクをスクリーン座標にあわせて用意する必要があるため恐らくシェーダーの修正が必要になりそうです。直接描画時は描画結果も若干変わってしまうかもしれません。
I tried it on Android, it freeze the game.
I used the GDCubism at 4.3 is worked on window platform, and I want to run at android, Then, it caused the freeze on android platform. I'm not sure what parts made game can't be runnable. I know the android support is experimental, but still want to get a try on android platform.
When I build for android platform
Basically, I never try the godot-cpp old version to build for android platform, it always caused "Required toolchain not found". So... I build it after godot 4.3 updated. godot-cpp have a update, and then I can build android platform now.
The issues when use
I try to install android template and output, it have some errors i got (they solved I guess... or not?). The first of error called "GDCubismUserModel not found", I resolved it by built android output. (Yup, this problem is caused by me never build the android libs) Seconds of error called "No suitable library found for GDExtension", I resolved it on changing gd_cubism.gdextension's path,
# fixed to full path and it finally found.
android.debug.arm64 = "res://addons/gd_cubism/bin/libgd_cubism.android.debug.arm64.so"
android.release.arm64 = "res://addons/gd_cubism/bin/libgd_cubism.android.release.arm64.so"
Finally, it should be work if I'm not wrong... It freeze the viewer scene and never moving on.
I can't handle the issue, hope it can be easy to imporve...
@ALOBugTea
This issue pertains to the features to be implemented in version 0.7. The comment you made is a new issue, so I have prepared a separate issue for it elsewhere.
https://github.com/MizunagiKB/gd_cubism/issues/104
It is now possible to make the viewport size for masks adjustable. This behavior is compatible with version 0.6, so it has been implemented as version 0.6.
https://github.com/MizunagiKB/gd_cubism/tree/0.6
This is an example where the SubViewport is used at 4096x4096, while the mask viewport is specified at 1024x1024 and 512x512.
When I try to use Mask Viewport, it stops rendering the eyes (and other facial features such as blush), and I am not sure why.
UPD: "Adjust Scale" option is the culprit, if you have the scale less than 1, the mask viewport will be shifted towards (0;0) in scale. Mask Viewport does not follow the Scale properly.
@cjmaxik I have created a new issue to address this bug. https://github.com/MizunagiKB/gd_cubism/issues/108
About Rendering Without Using SubViewport
We have started experimenting with rendering methods that do not use SubViewport. The current experimental implementation allows for direct rendering but comes with certain limitations.
If direct rendering is feasible, inheriting from SubViewport will no longer be necessary, and knowledge of SubViewport will be required. This will likely result in a lighter class.
However, this approach changes the way nodes are assembled, so it may be necessary to switch options at build time or provide two nodes.
Nonetheless, having two nearly identical nodes might be confusing for users.
In order to provide a more practical demonstration rather than just an explanation, I have committed the current test version. https://github.com/MizunagiKB/gd_cubism/tree/feature_direct_rendering
Please note that this is intended for functionality verification, and it is not yet decided whether this will become the new version.
The implementation is as follows:
- There are some incompatibilities with the currently released GDCubismUserModel.
- This version has only been tested with demo_simple.tscn.
- GDCubismUserModel has been changed from inheriting SubViewport to inheriting Node2D.
-
AutoScale, which was intended for SubViewport, does not function correctly (it only works with
AutoScale = False). -
MaskViewport Size is only valid when
x = 0, y = 0(it generates a mask the same size as the screen, consuming a lot of memory). - Transform is also applied to the SubViewport prepared as a Mask, so it correctly inherits the Transform of the parent node.
Hello [MizunagiKB] and fellow contributors! Thanks for porting the engine to Godot, you did a great job!
I would like to point out that feature_direct_rendering is a must. When I try to use the master build, the model always has a nasty black outline (i.e. premultiplied alpha), while feature_direct_rendering does not have this problem.
If it has any chances of being merged, I'd contribute a renderer that uses the RenderingDevice.
@Vadimledyaev Thank you very much. I ported the Live2D framework out of personal interest, and it seems that more people than I expected are using it. The somewhat strange rendering results are probably due to my lack of understanding of shaders.
@mr11stein Although I have not been able to confirm it yet, I have already received a pull request separate from my implementation.
https://github.com/MizunagiKB/gd_cubism/pull/129
The approach of rendering without using SubViewport is seen as a significant change since it alters the rendering approach.
For those who want to run Live2D, having to build it themselves is quite a burden, so at least I want to make it easy to use.
You should be able to have all 3 renderers without changes for the users (Other than providing an option to choose it). Rendering with the RenderingDevice should theoretically provide the best performance and lowest memory usage. It also allows to implement #7 without problem. The downside is that it doesn't support the Godot Compatibility renderer, so it's good to have multiple options.
レンダラーの選択以外に変更はありません。理論的には、RenderingDevice を使用すると、パフォーマンスとメモリ使用量が最大限になります。#7 の解決は問題ありません。欠点は、Godot Compatibility レンダラーを使用できないことです。
You should be able to have all 3 renderers without changes for the users (Other than providing an option to choose it). Rendering with the RenderingDevice should theoretically provide the best performance and lowest memory usage. It also allows to implement #7 without problem. The downside is that it doesn't support the Godot Compatibility renderer, so it's good to have multiple options.
レンダラーの選択以外に変更はありません。理論的には、RenderingDevice を使用すると、パフォーマンスとメモリ使用量が最大限になります。#7 の解決は問題ありません。欠点は、Godot Compatibility レンダラーを使用できないことです。
Totally agree, looking forward to using this great implementation on the Nintendo Switch where the mobile renderer is the only option. まったく同感です。モバイル レンダラーが唯一の選択肢である Nintendo Switch でこの優れた実装を使用するのを楽しみにしています。
The contents described in this proposal have been achieved in version 0.9.x, although in a different form.
Since leaving it open may cause misunderstandings regarding its completion status, I am closing this issue as an announcement.
Specifying the Size of Mask Rendering Textures
Rendering a Live2D model involves an internal process of generating a mask image.
In GDCubism, this process is implemented using SubViewport. Initially, the same size as the screen was used, but later it was made adjustable.
This functionality remains effective in the current implementation, though it was introduced more as a trade-off between memory efficiency and quality rather than purely reducing unnecessary memory usage.
There is room for improvement in the texture generation for mask rendering. However, as long as no rendering issues arise, the current implementation should be sufficient.
Rendering Without Using SubViewport
Over six months ago, I conducted an experimental implementation where rendering was done directly instead of through SubViewport.
https://github.com/MizunagiKB/gd_cubism/tree/feature_direct_rendering
Ultimately, this approach was fully realized through a pull request by erodozer.
This pull request introduced significant changes, and numerous minor refinements were made along the way. While the behavior of GDCubismUserModel has slightly changed, the functionality remains consistent when accessed via GDScript.
In fact, my sample code runs equally well in both GDScript and C#!
Additionally, another approach to rendering without SubViewport was proposed by mr11stein, utilizing Godot Engine's RenderingDevice.
The implementation of GDCubism is designed to use Node, which are commonly used by developers creating applications in Godot Engine.
However, this is simply my approach, and Live2D model users may not necessarily be concerned with this aspect.
I have not merged this pull request due to my own lack of knowledge, but I would like to introduce it as a reference for handling Live2D models efficiently and for utilizing RenderingDevice.
Initially, rendering using SubViewport was a convenient approach for development.
However, it required users to have knowledge of SubViewport, and achieving the intended quality of Live2D models resulted in excessive GPU memory consumption.
Acknowledgments
This issue has existed since the inception of GDCubism, but I am pleased that we have finally reached this point. It was made possible thanks to the support and efforts of many contributors.
Thank you all!
こちらのプロポーザルに記載されていた内容については、形は異なりますが 0.9.x にて達成する事が出来ました。
開いたままだと未達成と誤解される可能性があるため、発表も兼ねてクローズいたします。
マスク描画用テクスチャのサイズ指定
Live2Dモデルの描画をするには、内部でマスク画像の生成という処理が発生します。
この処理は GDCubism では SubViewport を使用して実現しています。当初は画面と同じサイズを使用していたのですが、途中から可変可能としました。
こちらは現在の実装でも有効ですが、メモリの無駄を省くというよりも品質とのトレードオフを行うために用意されています。
マスク描画用のテクスチャ生成については改良の余地はありますが、描画が崩れたりしなければ、現状のままで問題ないと考えています。
SubViewportを使用せずに描画を行う方法
半年以上前に私が試験的に実装したものがありましたが、この実装は SubViewport に対して行っていたものを、ひとまず直接描画にしてしまうというものでした。
https://github.com/MizunagiKB/gd_cubism/tree/feature_direct_rendering
最終的にこの実装は、 erodozer さんのプルリクエストにより実現されました。
このプルリクエストはかなり大きな変更であり、ここに至るまでに細かい修正が地道に行われました。
GDCubismUserModel の振る舞いはやや変わってはいるものの GDScript からの操作については同じ様に利用できるものが提供されました。
実際に私が用意しているサンプルコードは GDScript, C# どちらでも動きます！
なお、SubViewport を使用しない方法として Godot Engine の RenderingDevice を使用するというアプローチのプルリクエストを mr11stein さんより頂きました。
GDCubismの実装は、Godot Engineでアプリを開発する際、多くの人が触れるNodeを用いる形で設計されています。しかし、これは私の考え方であり、Live2Dモデルの利用者にとってはあまり関係のないことかもしれません。
こちらをマージはしていない理由は、私の知識不足に依るものなのですが、 Live2Dモデルを軽量に取り扱う手法、そして RenderingDevice を使用する手法の記録として紹介致します。
謝辞
当初 SubViewport を使用した描画は、プログラムをする上で都合がよいものでしたが、利用する人にとってはSubViewportの知識が必要なものでした。
そして、Live2Dモデルが本来持っている品質を引き出そうとするとGPUメモリが過剰に必要なものでした。
この課題はGDCubismが誕生した時からの課題でしたが、ここまで辿り着く事が出来ました。これは様々な人の応援があり辿り着けたものです。
ありがとうございます！