Increase texture resolution

[rkfg]

Is there any way to improve the texture? It's very low res and often buggy. The mesh quality is fine though.

I played with foreground ratio but can't make it any better. I also changed the chunk size and I don't see any noticeable effect.

[mrbid]

It's projecting to Vertex Colors, you could easily re-map it manually in Blender for example by UV Unwrapping it and then using the original source image as a texture.

Although you'd probably want to project the existing vertex colors to texture first, maybe you could run that though some kind of machine learning refinement, maybe an upscale just to downscale again so that you have the original size image of refined detail or just plug it back into a diffuser - there are many options.

This process would require little manual work other than just executing some automatic UV Unwrapping, you could use one of the rendering engines in cycles to to the projection etc.

[al3dv2]

For increase the details of the vertex color it's just need to increase the level of subdivision of the object someone know where in the script we can change the resolution of the mesh ?

[mrbid]

For increase the details of the vertex color it's just need to increase the level of subdivision of the object someone know where in the script we can change the resolution of the mesh ?

Correct, I am not sure just yet as the vertex count of the model varies with no changes to the settings, I have had some models come out at 30,000 triangles and some at 130,000 triangles. So I don't think there is a specific parameter but I will be looking into this and may get back to you with an answer in coming days.

[nagolinc]

I've just been projecting the original image onto the front of the mesh

def projectImg(mesh, img_path, save_path="static/samples/",foreground_ratio=0.85):
    modified_img_path img_path
    
    # Load the modified image to get its size for UV mapping calculations
    modified_image = Image.open(modified_img_path)
    width, height = modified_image.size
    
    # Calculate UVs for the transparent area
    # Assuming the transparent area is at the bottom right corner
    uv_transparent = np.array([(width - 100) / width, (height - 100) / height])
    
    # Identify front and back faces based on their normals
    forward_normal = np.array([0, 0, 1])
    dot_product = mesh.face_normals.dot(forward_normal)
    front_faces = dot_product < 0
    back_faces = ~front_faces
    
    # Compute UV coordinates for all vertices based on their XY positions
    uv_bounds_min, uv_bounds_max = -0.5/foreground_ratio, 0.5/foreground_ratio
    uv_coords = (mesh.vertices[:, [0, 1]] - uv_bounds_min) / (uv_bounds_max - uv_bounds_min)
    
    # Adjust UV coordinates for back faces to map to the transparent area
    for face in mesh.faces[back_faces]:
        uv_coords[face] = uv_transparent  # Map back face vertices to the transparent area
    
    # Apply the texture and UV mapping
    texture = trimesh.visual.texture.TextureVisuals(image=modified_image)
    mesh.visual = texture
    mesh.visual.uv = uv_coords
    
    # Export the mesh as GLB
    #mesh_filename = save_path + 'textured_mesh_with_transparency.glb'
    mesh_filename = getFilename(save_path, "glb")
    mesh.export(mesh_filename, file_type='glb')
    
    return mesh_filename

Obviously not optimal, and doesn't generate colors for the back/side.

Does anyone have a solution for dumping the vertex colors to a uv texture (then maybe we could try and enhance them with stableDiffusion or something)?

[CyberTimon]

I would also like to know how to increase the generated texture resolution automatically. +1

[al3dv2]

@nagolinc you can unwrap the model in blender then use Stable Projectorz for project hd texture on the model but the problem you get bad result in some angles the best could be to just generate models with more geometry

For dumping the vertex colors to a uv texture use Meshlab :

[mrbid]

Please refer to my findings in this answer concerning increased mesh vertex count: https://github.com/VAST-AI-Research/TripoSR/issues/12#issuecomment-1981009404

I don't think increasing the mesh size is going to help much. Probably best to focus on using something like: https://huggingface.co/spaces/liuyuan-pal/SyncDreamer https://huggingface.co/spaces/flamehaze1115/Wonder3D-demo https://huggingface.co/spaces/sudo-ai/zero123plus-demo-space https://github.com/bytedance/MVDream

To generate the consistent multi-view images of the original texture.

And then upscale those 256^2 or 320^2 outputs by x2 or x4: https://huggingface.co/spaces/bookbot/Image-Upscaling-Playground

And then to use those to construct a new texture map with new UV mapping.

[al3dv2]

@mrbid Yes good idea thanks for the links

[al3dv2]

@mrbid The best for upscaling is to use controlnet Tile, there's also SUPIR

[al3dv2]

Something that could works too is to duplicate the mesh in Blender make a clean unwrap of the duplicate for get a great uv texture, baking the vertex color on the clean unwrap and then upscale with controlnet tile or SUPIR the uv texture

[CyberTimon]

Maybe a combination of projecting and the generated will work good together for the front and the back?

[thatname]

I fixed this, you can try #68

[mrbid]

I fixed this, you can try #68

In CPU mode I can't set the marching cubes resolution to more than 96 without this error:

Traceback (most recent call last):
  File "/home/r/.local/lib/python3.11/site-packages/gradio/queueing.py", line 495, in call_prediction
    output = await route_utils.call_process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/gradio/route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/gradio/blocks.py", line 1561, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/gradio/blocks.py", line 1179, in call_function
    prediction = await anyio.to_thread.run_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/gradio/utils.py", line 678, in wrapper
    response = f(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
  File "/home/r/Desktop/TripoSR/gradio_app.py", line 59, in generate
    mesh = model.extract_mesh(scene_codes, resolution=mc_resolution)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/Desktop/TripoSR/tsr/system.py", line 166, in extract_mesh
    v_pos, color, t_pos_idx = self.renderer.block_based_marchingcube(self.decoder.to(scene_codes.device),
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/Desktop/TripoSR/tsr/models/nerf_renderer.py", line 91, in block_based_marchingcube
    v_color = self.query_triplane(decoder, v_pos, triplane, False)["color"]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/Desktop/TripoSR/tsr/models/nerf_renderer.py", line 142, in query_triplane
    net_out = chunk_batch(_query_chunk, self.chunk_size, positions)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/Desktop/TripoSR/tsr/utils.py", line 169, in chunk_batch
    out_chunk = func(
                ^^^^^
  File "/home/r/Desktop/TripoSR/tsr/models/nerf_renderer.py", line 138, in _query_chunk
    net_out: Dict[str, torch.Tensor] = decoder(out)
                                       ^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/r/Desktop/TripoSR/tsr/models/network_utils.py", line 121, in forward
    features = features.reshape(*inp_shape, -1)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous

I have not tested on the GPU yet as my main workstation is a thread ripper with a dinky 2060 in it, it's more of a CPU load machine.

[al3dv2]

I fixed this, you can try #68 @thatname I have the same problem than mrbid I can't go above 96 and I'm using gpu mode :

Traceback (most recent call last): File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\queueing.py", line 501, in call_prediction output = await route_utils.call_process_api( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\route_utils.py", line 258, in call_process_api output = await app.get_blocks().process_api( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\blocks.py", line 1710, in process_api result = await self.call_function( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\blocks.py", line 1250, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio_backends_asyncio.py", line 851, in run result = context.run(func, *args) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\utils.py", line 693, in wrapper response = f(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\gradio_app.py", line 63, in generate mesh = model.extract_mesh(scene_codes, resolution=mc_resolution)[0] File "C:\Users\Shadow\Desktop\TripoSR\tsr\system.py", line 166, in extract_mesh v_pos, color, t_pos_idx = self.renderer.block_based_marchingcube(self.decoder.to(scene_codes.device), File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 91, in block_based_marchingcube v_color = self.query_triplane(decoder, v_pos, triplane, False)["color"] File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 142, in query_triplane net_out = chunk_batch(_query_chunk, self.chunk_size, positions) File "C:\Users\Shadow\Desktop\TripoSR\tsr\utils.py", line 169, in chunk_batch out_chunk = func( File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 138, in _query_chunk net_out: Dict[str, torch.Tensor] = decoder(out) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\network_utils.py", line 121, in forward features = features.reshape(*inp_shape, -1) RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous

[thatname]

I fixed this, you can try #68 @thatname I have the same problem than mrbid I can't go above 96 and I'm using gpu mode :

Traceback (most recent call last): File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\queueing.py", line 501, in call_prediction output = await route_utils.call_process_api( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\route_utils.py", line 258, in call_process_api output = await app.get_blocks().process_api( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\blocks.py", line 1710, in process_api result = await self.call_function( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\blocks.py", line 1250, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio_backends_asyncio.py", line 851, in run result = context.run(func, *args) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\utils.py", line 693, in wrapper response = f(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\gradio_app.py", line 63, in generate mesh = model.extract_mesh(scene_codes, resolution=mc_resolution)[0] File "C:\Users\Shadow\Desktop\TripoSR\tsr\system.py", line 166, in extract_mesh v_pos, color, t_pos_idx = self.renderer.block_based_marchingcube(self.decoder.to(scene_codes.device), File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 91, in block_based_marchingcube v_color = self.query_triplane(decoder, v_pos, triplane, False)["color"] File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 142, in query_triplane net_out = chunk_batch(_query_chunk, self.chunk_size, positions) File "C:\Users\Shadow\Desktop\TripoSR\tsr\utils.py", line 169, in chunk_batch out_chunk = func( File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 138, in _query_chunk net_out: Dict[str, torch.Tensor] = decoder(out) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\network_utils.py", line 121, in forward features = features.reshape(*inp_shape, -1) RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous

I changed the color sampling code to fix it, please try again? @mrbid @al3dv2

[al3dv2]

I changed the color sampling code to fix it, please try again?

It's works thanks! I have another question do you know how to add your code to the comfyui version ? https://github.com/flowtyone/ComfyUI-Flowty-TripoSR Because the comfyui version generate 3d models faster on my computer

[mrbid]

I fixed this, you can try #68 @thatname I have the same problem than mrbid I can't go above 96 and I'm using gpu mode :

Traceback (most recent call last): File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\queueing.py", line 501, in call_prediction output = await route_utils.call_process_api( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\route_utils.py", line 258, in call_process_api output = await app.get_blocks().process_api( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\blocks.py", line 1710, in process_api result = await self.call_function( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\blocks.py", line 1250, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio\to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\anyio_backends_asyncio.py", line 851, in run result = context.run(func, *args) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\gradio\utils.py", line 693, in wrapper response = f(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\gradio_app.py", line 63, in generate mesh = model.extract_mesh(scene_codes, resolution=mc_resolution)[0] File "C:\Users\Shadow\Desktop\TripoSR\tsr\system.py", line 166, in extract_mesh v_pos, color, t_pos_idx = self.renderer.block_based_marchingcube(self.decoder.to(scene_codes.device), File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 91, in block_based_marchingcube v_color = self.query_triplane(decoder, v_pos, triplane, False)["color"] File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 142, in query_triplane net_out = chunk_batch(_query_chunk, self.chunk_size, positions) File "C:\Users\Shadow\Desktop\TripoSR\tsr\utils.py", line 169, in chunk_batch out_chunk = func( File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\nerf_renderer.py", line 138, in _query_chunk net_out: Dict[str, torch.Tensor] = decoder(out) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\tripo\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl return forward_call(*args, **kwargs) File "C:\Users\Shadow\Desktop\TripoSR\tsr\models\network_utils.py", line 121, in forward features = features.reshape(*inp_shape, -1) RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1] because the unspecified dimension size -1 can be any value and is ambiguous

I changed the color sampling code to fix it, please try again? @mrbid @al3dv2

Looks good, took 664 seconds for 1024 resolution on a 3995WX 256 vs 1024 resolution

Regular 256 resolution from original codebase:

Download: gnome_obj.zip gnome_256_regular_obj.zip

[thatname]

I changed the color sampling code to fix it, please try again?

It's works thanks! I have another question do you know how to add your code to the comfyui version ? https://github.com/flowtyone/ComfyUI-Flowty-TripoSR Because the comfyui version generate 3d models faster on my computer Copy the 'tsr' folder from my repo may work. But their code used another marching cube package, maybe it can generate 1024 directly.

[al3dv2]

@thatname I tried but it's doesn't works I got a error message : ERROR:root:* TripoSRSampler 274: ERROR:root: - Return type mismatch between linked nodes: model, TRIPOSR_MODEL != ERROR:root:Output will be ignored

[mrbid]

It's hard to say if one is better than the other, we need more comprehensive comparisons and performance benchmarks posted here.

[al3dv2]

Someone have release a script for increase the resolution of the texture https://github.com/ejones/triposr-texture-gen?tab=readme-ov-file

[mrbid]

Someone have release a script for increase the resolution of the texture https://github.com/ejones/triposr-texture-gen?tab=readme-ov-file

Hey did you get it working in Comfy, you can do it I am sure of it, just look at the changes to the file via the git and copy those changes over to the respective modified Comfy files - the Comfy files can't have been changed that much?

[mrbid]

I changed the color sampling code to fix it, please try again?

It's works thanks! I have another question do you know how to add your code to the comfyui version ? https://github.com/flowtyone/ComfyUI-Flowty-TripoSR Because the comfyui version generate 3d models faster on my computer Copy the 'tsr' folder from my repo may work. But their code used another marching cube package, maybe it can generate 1024 directly.

It's about 1.3-1.5 times faster from my benchmarks with default settings resolution=256 etc:

Nice work bro.

[mrbid]

I fixed this, you can try #68

I've been using this in my recent project https://huggingface.co/datasets/tfnn/HeadsNet

I have made sure to credit you for your work :bow: