Support runtime loading of ply splat format?

[tufeixp]

It'd be nice if this repo can support runtime loading of ply splat format, like this repo have done: https://github.com/clarte53/GaussianSplattingVRViewerUnity

Also, I find the ply models generated by using: https://github.com/alvinliu0/HumanGaussian/tree/main?tab=readme-ov-file#pretrained-models and https://github.com/Microsoft/TRELLIS will both report vertex size mismatch issue while importing in editor Gaussian splat creator. Please look into it.

Here the header: ply format binary_little_endian 1.0 element vertex 675392 property float x property float y property float z property float nx property float ny property float nz property float f_dc_0 property float f_dc_1 property float f_dc_2 property float opacity property float scale_0 property float scale_1 property float scale_2 property float rot_0 property float rot_1 property float rot_2 property float rot_3 end_header

and I tried to modify the InputSplatData to this: public struct InputSplatDataS { public Vector3 pos; public Vector3 nor; public Vector3 dc0; public float opacity; public Vector3 scale; public Quaternion rot; } However, the error 's gone and loaded without correct rendering, the points will show with the debug points option choosen in the render mode.

[Hiverlab-Ray]

I've tried reconfiguring the editor scripts into run-time (even for those downloaded via web URLs). With some code understanding and knowledge, it's definitely possible which I have successfully made it working.

See video: https://github.com/user-attachments/assets/3c03977d-c8c6-4731-86b8-91afee2fddf7

[aras-p]

Please look into it. Here the header:

Yeah, so their PLY file format is different from the original 3DGS paper format. This repository right now only supports the file format as it was in the original gaussian splatting paper, and not any of the later modifications to the format done by others.

Someday I might implement support for other PLY format variations, but it is not very likely (I stopped working on gaussian splatting a year ago). If anyone wants to contribute support, I'll gladly look at it.

[akutterxressential]

@aras-p It is a pity that you stopped working on it. We want to use this project for our jobs but I don`t have good enough programmers so we could contribute. I hope this changes some day...

[YusukeKobayashiAB]

I've tried reconfiguring the editor scripts into run-time (even for those downloaded via web URLs). With some code understanding and knowledge, it's definitely possible which I have successfully made it working.

See video: https://github.com/user-attachments/assets/3c03977d-c8c6-4731-86b8-91afee2fddf7

Hello, this script is exactly what I need. I don't know which part to modify to achieve this. Could you share the code with me?

[akutterxressential]

I've tried reconfiguring the editor scripts into run-time (even for those downloaded via web URLs). With some code understanding and knowledge, it's definitely possible which I have successfully made it working.

See video: https://github.com/user-attachments/assets/3c03977d-c8c6-4731-86b8-91afee2fddf7

Yes yes me too! Can you please contribute this to the code?

[Megumin61]

It'd be nice if this repo can support runtime loading of ply splat format, like this repo have done: https://github.com/clarte53/GaussianSplattingVRViewerUnity

Also, I find the ply models generated by using: https://github.com/alvinliu0/HumanGaussian/tree/main?tab=readme-ov-file#pretrained-models and https://github.com/Microsoft/TRELLIS will both report vertex size mismatch issue while importing in editor Gaussian splat creator. Please look into it.

Here the header: ply format binary_little_endian 1.0 element vertex 675392 property float x property float y property float z property float nx property float ny property float nz property float f_dc_0 property float f_dc_1 property float f_dc_2 property float opacity property float scale_0 property float scale_1 property float scale_2 property float rot_0 property float rot_1 property float rot_2 property float rot_3 end_header

and I tried to modify the InputSplatData to this: public struct InputSplatDataS { public Vector3 pos; public Vector3 nor; public Vector3 dc0; public float opacity; public Vector3 scale; public Quaternion rot; } However, the error 's gone and loaded without correct rendering, the points will show with the debug points option choosen in the render mode.

I solved the 3dgs vertex size mismatch generated by Trellis by modifying GaussianFileReader.cs. Following the format of the data it generates, I added to the code to take into account the 68-byte case. The code can be referred:

public static unsafe void ReadFile(string filePath, out NativeArray<InputSplatData> splats)
        {
            if (isPLY(filePath))
            {
                NativeArray<byte> verticesRawData;
                PLYFileReader.ReadFile(
                    filePath,
                    out var splatCount,
                    out var splatStride,
                    out List<string> headerInfo,
                    out verticesRawData
                );
                int expectedSize = UnsafeUtility.SizeOf<InputSplatData>();

                if (splatStride == expectedSize)
                {
                    NativeArray<float> floatData = verticesRawData.Reinterpret<float>(1);
                    ReorderSHs(splatCount, (float*)floatData.GetUnsafePtr());
                    splats = verticesRawData.Reinterpret<InputSplatData>(1);
                    LinearizeData(splats);
                    return;
                }
                else if (splatStride == 68)
                {
                    var newSplats = new NativeArray<InputSplatData>(
                        splatCount,
                        Allocator.Persistent
                    );
                    NativeArray<float> floatData = verticesRawData.Reinterpret<float>(1);
                    for (int i = 0; i < splatCount; i++)
                    {
                        int baseIndex = i * 17;
                        InputSplatData splat = new InputSplatData();
                        splat.pos = new Vector3(
                            floatData[baseIndex + 0],
                            floatData[baseIndex + 1],
                            floatData[baseIndex + 2]
                        );
                        splat.nor = new Vector3(
                            floatData[baseIndex + 3],
                            floatData[baseIndex + 4],
                            floatData[baseIndex + 5]
                        );
                        splat.dc0 = new Vector3(
                            floatData[baseIndex + 6],
                            floatData[baseIndex + 7],
                            floatData[baseIndex + 8]
                        );
                        splat.opacity = floatData[baseIndex + 9];
                        splat.scale = new Vector3(
                            floatData[baseIndex + 10],
                            floatData[baseIndex + 11],
                            floatData[baseIndex + 12]
                        );
                        splat.rot = new Quaternion(
                            floatData[baseIndex + 13],
                            floatData[baseIndex + 14],
                            floatData[baseIndex + 15],
                            floatData[baseIndex + 16]
                        );
                        splat.sh1 =
                            splat.sh2 =
                            splat.sh3 =
                            splat.sh4 =
                            splat.sh5 =
                            splat.sh6 =
                            splat.sh7 =
                            splat.sh8 =
                            splat.sh9 =
                            splat.shA =
                            splat.shB =
                            splat.shC =
                            splat.shD =
                            splat.shE =
                            splat.shF =
                                Vector3.zero;
                        newSplats[i] = splat;
                    }
                    splats = newSplats;
                    LinearizeData(splats);
                    return;
                }
                else
                {
                    throw new IOException(
                        $"PLY vertex size mismatch, expected {expectedSize} or 68 but file has {splatStride}"
                    );
                }
            }
            if (isSPZ(filePath))
            {
                SPZFileReader.ReadFile(filePath, out splats);
                return;
            }
            throw new IOException($"File {filePath} is not a supported format");
        }

[Megumin61]

I've implemented runtime creation of assets and assignment of values by simulating the inputs to the window through reflection. You can refer to it:

if (string.IsNullOrEmpty(plyFilePath))
        {
            Debug.LogError("Set Path！");
            return;
        }
        EditorPrefs.SetString("nesnausk.GaussianSplatting.CreatorOutputFolder", outputFolder);
        EditorPrefs.SetInt("nesnausk.GaussianSplatting.CreatorQuality", quality);

        var creator = EditorWindow.CreateInstance<GaussianSplatAssetCreator>();
        SetField(creator, "m_InputFile", plyFilePath);
        SetField(creator, "m_ImportCameras", importCameras);
        MethodInfo onEnableMethod = creator.GetType().GetMethod("OnEnable", BindingFlags.NonPublic | BindingFlags.Instance);
        onEnableMethod?.Invoke(creator, null);

        MethodInfo createAssetMethod = creator.GetType().GetMethod("CreateAsset", BindingFlags.NonPublic | BindingFlags.Instance);
        if (createAssetMethod == null)
        {
            Debug.LogError("CreateAsset not found！");
            return;
        }
        createAssetMethod.Invoke(creator, null);

        string baseName = Path.GetFileNameWithoutExtension(FilePickerControl.PathToDisplayString(plyFilePath));
        string assetPath = Path.Combine(outputFolder, baseName + ".asset").Replace("\\", "/");

        var createdAsset = AssetDatabase.LoadAssetAtPath<GaussianSplatAsset>(assetPath);
        if (createdAsset == null)
        {
            Debug.LogError("can not create asset：" + assetPath);
        }
        else
        {
            var renderer = GetComponent<GaussianSplatRenderer>();
            if (renderer == null)
            {
                Debug.LogError("Can not find GaussianSplatRenderer！");
            }
            else
            {
                renderer.m_Asset = createdAsset;
                Debug.Log("Success！");
            }
        }
        creator.Close();

[YusukeKobayashiAB]

Hello, where should this code be embedded? Also, this code seems to work only in the Editor—would it be possible to make it work in a built application? What I want to create is a model viewer where users can freely change the background.

I've implemented runtime creation of assets and assignment of values by simulating the inputs to the window through reflection. You can refer to it:

[alex-vienne]

Hi, I've extended this code in order to load PLY file on runtime. I think it's that you're looking for. I've tested it on Windows, it's still on work for WebGPU. The source code is there: https://github.com/alex-vienne/UnityGaussianSplatting

[YusukeKobayashiAB]

@alex-vienne It worked perfectly! I also appreciate the UI to adjust parameters! Thank you!

[akutterxressential]

@alex-vienne

I imported your version into Unity and created a WebGPU Build. Your flower splat works like a charmm, but as soon as I create a splat with the wizard from an own PLY it looks like this. I tried several PLY that worked before :-/ is there something I am doing wrong?

Here the splat I used: https://cloud.xr-essential.com/s/sXJffPiXkARLC8Z

[soumi-akira]

Hi @aras-p

First of all, I'd like to express my sincere gratitude for creating and sharing this amazing repository. It's a very useful project, and I think you've done a fantastic job.

Regarding the "runtime PLY file loading" feature discussed in this issue (#157), I've implemented it in my fork:

https://github.com/soumi-akira/UnityGaussianSplatting

Here's a summary of my main approach to implementing this feature:

• I moved some editor utility classes related to asset generation to a runtime namespace so they can be used at runtime.
• I extracted asset generation-related methods used in other editor scripts and ported them to the runtime side.
• Considering that Unity assets (like TextAsset) cannot always be created or handled in the same way at runtime, I defined interfaces for the necessary operations on these assets (e.g., a method like GetData). I then implemented these interfaces for runtime use.
• Based on these interfaces, I implemented classes to create the actual asset instances at runtime.
• I also adjusted the structure of existing assets generated on the editor side to conform to these new interfaces.
• I added properties to the renderer to support dynamic switching to assets loaded from PLY files at runtime.

I would like to create a pull request with this implementation and contribute to this project. If there are any project-specific contribution guidelines, other points to note, or any feedback on this approach before I create a PR, I would appreciate it if you could let me know.

Thank you for your time and consideration.

[FirepadCN]

@soumi-akira I also want to try reading the ply file to use it. How can I use your project? Is there an example?

[FirepadCN]

@soumi-akira I also want to try reading the ply file to use it. How can I use your project? Is there an example?

    [ContextMenu("LoadPlayGS")]
    private void LoadPlayGS()
    {
        string path = "E:\\point_cloud.ply";
        GaussianSplatRuntimeAssetCreator creator = new GaussianSplatRuntimeAssetCreator();
        var asset = creator.CreateAsset("point_cloud", path);
        var render = FindObjectOfType<GaussianSplatRenderer>();
        render.InjectAsset(asset);
    }

[soumi-akira]

@soumi-akira I also want to try reading the ply file to use it. How can I use your project? Is there an example?

[ContextMenu("LoadPlayGS")]
private void LoadPlayGS()
{
    string path = "E:\\point_cloud.ply";
    GaussianSplatRuntimeAssetCreator creator = new GaussianSplatRuntimeAssetCreator();
    var asset = creator.CreateAsset("point_cloud", path);
    var render = FindObjectOfType<GaussianSplatRenderer>();
    render.InjectAsset(asset);
}

Hi @FirepadCN

Thanks for reaching out! Your understanding and the code snippet you shared for loading the PLY file are spot on. My apologies for not including a sample directly in the main branch or making its location clearer initially. I was trying to keep the main branch focused on the core functionality.

You can find a sample implementation on the check branch in my repository: https://github.com/soumi-akira/UnityGaussianSplatting/tree/check

Here is sample code: https://github.com/soumi-akira/UnityGaussianSplatting/blob/check/projects/GaussianExample-URP/Assets/Test.cs

[FirepadCN]

Hi @soumi-akira When I tried to build it, several editor methods failed. I wrapped them in #if UNITY_EDITOR. Is this a viable solution?

[soumi-akira]

Hi @FirepadCN

Thanks for the report and the #if UNITY_EDITOR suggestion. You're right, EditorUtility.CompressTexture is editor-only.

For runtime, Unity docs suggest Texture2D.Compress.

https://docs.unity3d.com/6000.1/Documentation/ScriptReference/EditorUtility.CompressTexture.html

I've updated the code accordingly:

#if UNITY_EDITOR
        EditorUtility.CompressTexture(tex, GraphicsFormatUtility.GetTextureFormat(gfxFormat), 100);
#else
        tex.Compress(true); 
#endif

I've pushed this fix to the main branch.

[diandiandevelop]

@soumi-akira Hi， m_GpuSHData = new GraphicsBuffer(GraphicsBuffer.Target.Raw, splatCount * 12 * 16, 4) { name = "GaussianSHData" }; GraphicsBuffer may exceed 2GB，Are there any optimization methods?