zero123plus
zero123plus copied to clipboard
SUDO-AI-3D
How to complete 3D reconstruction
How to complete 3D reconstruction？ Image results cannot be reconstructed using colmap。 Can you mention some of the fake gestures?
What do you mean by fake gestures? Classical methods for 3D reconstruction usually cannot work well with the sparse views. Another model would usually be needed to reconstruct mesh. For example, you can utilize narrow baseline 0123 + SparseNeus from One-2-3-45 https://github.com/One-2-3-45/One-2-3-45 for 3D reconstruction.
Test showing what results can be achieved using only 7 images in Instant-NGP:
https://github.com/SUDO-AI-3D/zero123plus/assets/12137233/005bb9db-1d83-4c5f-ae14-495b7b83e27f
https://github.com/SUDO-AI-3D/zero123plus/assets/12137233/40fa3472-c002-406d-9950-e5f20730f459
https://github.com/SUDO-AI-3D/zero123plus/assets/12137233/126b7139-5a6d-4e95-ab12-4f203022eb9d
https://github.com/SUDO-AI-3D/zero123plus/assets/12137233/1cbbc395-f590-41e1-8ae7-34bdcc3dfb78
Obviously, more images are needed to achieve better and more accurate results. But considering that this is only 7 images, the result is impressive.
Here are the contents of the "transforms.json" file that I got (In case anyone wants to try):
{
"camera_angle_x": 0.03812197129202137,
"camera_angle_y": 0.0140774200158162,
"fl_x": 320.719333182715,
"fl_y": 320.1815001140895,
"k1": -0.01956388203737949,
"k2": 0.110182871165539e-05,
"p1": -0.05669583599777802,
"p2": -0.0175356537000764e-05,
"cx": 160.9954396730627,
"cy": 160.2710416469586,
"w": 320.0,
"h": 320.0,
"aabb_scale": 16,
"scale": 2.7,
"frames": [
{
"file_path": "./images/image0.png",
"sharpness": 27.732116033649934,
"transform_matrix": [
[0.9999500513076782, 0.004810694605112076, 0.008763475343585014, 0.010515931993722916],
[0.008762343786656857, 0.0002564578899182379, -0.9999614953994751, -1.1999539136886597],
[-0.004812757018953562, 0.9999883770942688, 0.00021429210028145462, 0.00025705056032165885],
[0.0, 0.0, 0.0, 1.0]
]
},
{
"file_path": "./images/image1.png",
"sharpness": 27.794309310423316,
"transform_matrix": [
[0.861971378326416, -0.24816764891147614, 0.442061185836792, 0.5305169224739075],
[0.5069567561149597, 0.42257794737815857, -0.751280665397644, -0.9016107320785522],
[-0.0003617570619098842, 0.8716884255409241, 0.49006035923957825, 0.5878257155418396],
[0.0, 0.0, 0.0, 1.0]
]
},
{
"file_path": "./images/image2.png",
"sharpness": 27.52245648391413,
"transform_matrix": [
[-0.008050278760492802, 0.34757664799690247, 0.9376169443130493, 1.1250077486038208],
[0.9999675750732422, 0.0030544146429747343, 0.007453335449099541, 0.008942931890487671],
[-0.00027326561394147575, 0.9376466274261475, -0.3475899398326874, -0.41733309626579285],
[0.0, 0.0, 0.0, 1.0]
]
},
{
"file_path": "./images/image3.png",
"sharpness": 27.58521978950844,
"transform_matrix": [
[-0.8660249710083008, -0.2501213252544403, 0.4329434335231781, 0.5195764899253845],
[0.4999733865261078, -0.42414551973342896, 0.7550675868988037, 0.9061554074287415],
[-0.005227487534284592, 0.870367705821991, 0.49237459897994995, 0.5906028747558594],
[0.0, 0.0, 0.0, 1.0]
]
},
{
"file_path": "./images/image4.png",
"sharpness": 27.95186379777407,
"transform_matrix": [
[-0.8700066804885864, -0.1741916537284851, -0.46124354004859924, -0.5534254312515259],
[-0.4930136799812317, 0.2977108061313629, 0.8174996376037598, 0.9808858036994934],
[-0.0050844247452914715, 0.9386296272277832, -0.34488925337791443, -0.4140926003456116],
[0.0, 0.0, 0.0, 1.0]
]
},
{
"file_path": "./images/image5.png",
"sharpness": 27.08037798388144,
"transform_matrix": [
[-0.008049866184592247, 0.4948376417160034, -0.8689481019973755, -1.0428248643875122],
[-0.9999675750732422, -0.003890044754371047, 0.007048365660011768, 0.008458686992526054],
[0.00010754966933745891, 0.8689767122268677, 0.4948529005050659, 0.5935773849487305],
[0.0, 0.0, 0.0, 1.0]
]
},
{
"file_path": "./images/image6.png",
"sharpness": 27.732116033649934,
"transform_matrix": [
[0.866024911403656, -0.17087915539741516, -0.4698948264122009, -0.5638080835342407],
[-0.49997571110725403, -0.3053785562515259, -0.8104123473167419, -0.9723791480064392],
[-0.005013220012187958, 0.9367733001708984, -0.3499008119106293, -0.4201057553291321],
[0.0, 0.0, 0.0, 1.0]
]
}
]
}
Be sure to replace the background in the pictures with transparency.
Test showing what results can be achieved using only 7 images in Instant-NGP:
instant-ngp_.x.00176.mp4 instant-ngp.x.00178.mp4 instant-ngp.x.00179.mp4 instant-ngp.x._00180.mp4 Obviously, more images are needed to achieve better and more accurate results. But considering that this is only 7 images, the result is impressive.
Here are the contents of the "transforms.json" file that I got (In case anyone wants to try):
{ "camera_angle_x": 0.03812197129202137, "camera_angle_y": 0.0140774200158162, "fl_x": 320.719333182715, "fl_y": 320.1815001140895, "k1": -0.01956388203737949, "k2": 0.110182871165539e-05, "p1": -0.05669583599777802, "p2": -0.0175356537000764e-05, "cx": 160.9954396730627, "cy": 160.2710416469586, "w": 320.0, "h": 320.0, "aabb_scale": 16, "scale": 2.7, "frames": [ { "file_path": "./images/image0.png", "sharpness": 27.732116033649934, "transform_matrix": [ [0.9999500513076782, 0.004810694605112076, 0.008763475343585014, 0.010515931993722916], [0.008762343786656857, 0.0002564578899182379, -0.9999614953994751, -1.1999539136886597], [-0.004812757018953562, 0.9999883770942688, 0.00021429210028145462, 0.00025705056032165885], [0.0, 0.0, 0.0, 1.0] ] }, { "file_path": "./images/image1.png", "sharpness": 27.794309310423316, "transform_matrix": [ [0.861971378326416, -0.24816764891147614, 0.442061185836792, 0.5305169224739075], [0.5069567561149597, 0.42257794737815857, -0.751280665397644, -0.9016107320785522], [-0.0003617570619098842, 0.8716884255409241, 0.49006035923957825, 0.5878257155418396], [0.0, 0.0, 0.0, 1.0] ] }, { "file_path": "./images/image2.png", "sharpness": 27.52245648391413, "transform_matrix": [ [-0.008050278760492802, 0.34757664799690247, 0.9376169443130493, 1.1250077486038208], [0.9999675750732422, 0.0030544146429747343, 0.007453335449099541, 0.008942931890487671], [-0.00027326561394147575, 0.9376466274261475, -0.3475899398326874, -0.41733309626579285], [0.0, 0.0, 0.0, 1.0] ] }, { "file_path": "./images/image3.png", "sharpness": 27.58521978950844, "transform_matrix": [ [-0.8660249710083008, -0.2501213252544403, 0.4329434335231781, 0.5195764899253845], [0.4999733865261078, -0.42414551973342896, 0.7550675868988037, 0.9061554074287415], [-0.005227487534284592, 0.870367705821991, 0.49237459897994995, 0.5906028747558594], [0.0, 0.0, 0.0, 1.0] ] }, { "file_path": "./images/image4.png", "sharpness": 27.95186379777407, "transform_matrix": [ [-0.8700066804885864, -0.1741916537284851, -0.46124354004859924, -0.5534254312515259], [-0.4930136799812317, 0.2977108061313629, 0.8174996376037598, 0.9808858036994934], [-0.0050844247452914715, 0.9386296272277832, -0.34488925337791443, -0.4140926003456116], [0.0, 0.0, 0.0, 1.0] ] }, { "file_path": "./images/image5.png", "sharpness": 27.08037798388144, "transform_matrix": [ [-0.008049866184592247, 0.4948376417160034, -0.8689481019973755, -1.0428248643875122], [-0.9999675750732422, -0.003890044754371047, 0.007048365660011768, 0.008458686992526054], [0.00010754966933745891, 0.8689767122268677, 0.4948529005050659, 0.5935773849487305], [0.0, 0.0, 0.0, 1.0] ] }, { "file_path": "./images/image6.png", "sharpness": 27.732116033649934, "transform_matrix": [ [0.866024911403656, -0.17087915539741516, -0.4698948264122009, -0.5638080835342407], [-0.49997571110725403, -0.3053785562515259, -0.8104123473167419, -0.9723791480064392], [-0.005013220012187958, 0.9367733001708984, -0.3499008119106293, -0.4201057553291321], [0.0, 0.0, 0.0, 1.0] ] } ] }Be sure to replace the background in the pictures with transparency.
Hi, how could you generate this "transforms.json" file, especially the camera intrinsic parameters?
Hi, how could you generate this "transforms.json" file, especially the camera intrinsic parameters?
In this particular case, this file was made partially by hand. I took one "transforms.json" file generated using colmap2nerf.py and started changing it.
First I generated a "transform matrix" using blender following the instructions https://github.com/SUDO-AI-3D/zero123plus#camera-poses.
I simply rotated the plane as was written in the link above and then got the matrix using this command:
bpy.context.selected_objects[0].matrix_world
Well, then I began to select the appropriate remaining parameters ("fl_x", "fl_y", "cx", "cy", "w", "h", "scale"). Most likely they are not chosen perfectly, but they are enough for it to work.
try blendernerf. saves a lot of trouble. https://github.com/maximeraafat/BlenderNeRF
