TripoSR-for-Unity
TripoSR-for-Unity copied to clipboard
Generate 3D meshes from 2D images and retexture them directly in Unity.
TripoSR for Unity
Seamlessly generate high-quality 3D meshes from 2D images and retexture them using text-based texture generation directly in Unity.
Overview
This project integrates TripoSR, a "a state-of-the-art open-source model for fast feedforward 3D reconstruction from a single image" by StabilityAI and TripoAI, and triposr-texture-gen directly into Unity Editor.
Features
- Transformation of 2D images into textured 3D meshes, available both in Editor and in Playmode.
- Text-based texture generation, applied directly to the generated 3D meshes.
The generated 3D meshes are imported using a modified vertex color importer (based on Andrew Raphael Lukasik's importer) and auto-assigned to a base Material with custom shader to utilize and display the vertex colors correctly (without surface normals, though).
Tested with Unity 2022.3.22f1 running on Windows 11 and Ubuntu 22.04.
Demo
The first demo clip shows the creation of a 3D mesh based on a 2D Texture within the Editor.
The generated meshes are colored (vertex colors), react to light, and (optionally) automatically use MeshCollider
and Rigidbody
for physics interaction.
https://github.com/mapluisch/TripoSR-for-Unity/assets/31780571/f27f62e0-00e3-4c14-8458-97302a82e76d
This clip shows the text-based retexturing of a TripoSR-generated 3D mesh:
https://github.com/mapluisch/TripoSR-for-Unity/assets/31780571/3e2fbb54-6828-49f0-80da-310bcedfc4c9
This demo clip shows 3D mesh generation in Playmode:
https://github.com/mapluisch/TripoSR-for-Unity/assets/31780571/d6b85653-a672-495f-b268-f4996075a4c1
Setup - TripoSR
Integrating the .unitypackage into your project
- Download the latest release
.unitypackage
and import it into your project (Assets > Import Package
). -
cd
into theAssets
folder of your Unity project (using Command Prompt, Terminal, ...) and clone the latest repo of TripoSR:git clone https://github.com/VAST-AI-Research/TripoSR.git
. - After cloning TripoSR,
cd
into the TripoSR folder that you just created by cloning the repo. - Run
pip install --upgrade setuptools
,pip install torch
(in case you don't have PyTorch installed) andpip install -r requirements.txt
. - Add the
TripoSR
Prefab (found inAssets > Prefabs
) to your scene. - Configure the path to your python executable in the
TripoSR
GameObject within your scene: For Windows, runwhere python
within Command Prompt. For Unix, runwhich python
within Terminal. - Configure the other public variables in the Inspector as needed.
Using this repo's Unity project
- Clone this repo.
- Ensure you have Python installed on your system.
- Run
pip install --upgrade setuptools
,pip install torch
(in case you don't have PyTorch installed) andpip install -r requirements.txt
from within this project'sAssets/TripoSR
folder. - In Unity, add the
TripoSR
Prefab to the scene (or simply open up mySampleScene
). - Configure the path to your python executable in the
TripoSR
GameObject within your scene: For Windows, runwhere python
within Command Prompt. For Unix, runwhich python
within Terminal. - Configure the other public variables in the Inspector as needed.
Setup - TextureGenerator
I've adapted the text2texture.py
script from triposr-texture-gen, so it doesn't open up the 3D viewer and works in-situ. When you clone the repo on your own, this won't be included.
When using the .unitypackage in your project
- Follow the general setup for TripoSR from above
-
cd
into theAssets
folder and clone the latest triposr-texture-gen. - Run
pip install -r -requirements.txt
from within theAssets/triposr-texture-gen
directory. - Run
pip install huggingface-hub --upgrade
to fix some compatibility problems. - Add the
TextureGenerator
Prefab to your scene. - Configure the path to your python executable.
- Configure the public vars to your liking.
When using this repo's Unity project
- Follow the general setup from above
- Run
pip install -r -requirements.txt
from within theAssets/triposr-texture-gen
directory. - Run
pip install huggingface-hub --upgrade
to fix some compatibility problems. - Add the
TextureGenerator
Prefab to your scene (or use mySampleScene
). - Configure the path to your python executable.
- Configure the public vars to your liking.
Usage - TripoSR
Once you have set up your scene with the TripoSRForUnity
component and configured the parameters, you can run the process by clicking the Run TripoSR
button in the inspector.
When you run TripoSR for the first time, the model weights will be downloaded and cached - this only occurs once; subsequent runs use the cached model.
General Settings
-
autoAddMesh
: When enabled, the generated mesh is automatically added to the Unity scene as GameObject. -
autoAddPhysicsComponents
: Whether or not to automatically add physics components (i.e., convexMeshCollider
andRigidbody
) to the generated mesh. -
autoFixRotation
: If enabled, will automatically correct the wrong object rotation after adding the mesh to the scene. -
moveAndRename
: Moves and renames the output.obj
file based on the input image's filename if enabled. -
moveAndRenamePath
: The directory to which the.obj
file will be moved. Must start withAssets/
. -
showDebugLogs
: Enables the display of debug outputs from therun.py
script in the Unity console.
TripoSR Parameters
All TripoSR parameters are exposed by my script. Feel free to change them as you see fit.
I've made some of them ReadOnly
within the Inspector, since you shouldn't really change those vars (e.g. model name, device to use). You can still change them within the script of course.
Usage - TextureGenerator
Likewise, you can run the texture generator by clicking on Generate Textured Object
within the TextureGenerator
-component in the inspector.
When you run the texture generator for the first time, model weights will also be downloaded and cached, which also only occurs the first time.
Set the model file in the inspector to your .obj
from Assets > Models
, describe the design you would like as model description, and start the generator.
Known Issues
TripoSR .obj
s only consist of v
s and f
s, surface normals are not calculated. When Unity calculates the normals upon import, they are not smoothed correctly (even when using high smoothing angles).
For now, I've disabled normal calculation - feel free to create a PR if you know how to correctly handle this issue.
License
This project is licensed under the MIT License. See LICENSE
for more information.
Acknowledgments
Special thanks to StabilityAI, TripoAI, and contributors of the TripoSR project.
Please cite their work adequately in case you use it in your own publications:
@article{TripoSR2024,
title={TripoSR: Fast 3D Object Reconstruction from a Single Image},
author={Tochilkin, Dmitry and Pankratz, David and Liu, Zexiang and Huang, Zixuan and and Letts, Adam and Li, Yangguang and Liang, Ding and Laforte, Christian and Jampani, Varun and Cao, Yan-Pei},
journal={arXiv preprint arXiv:2403.02151},
year={2024}
}
Also special thanks to @ejones for triposr-texture-gen.