d3fields Confusing about optimize

Thank you for your excellent work! I am confused about one part, Is it feasible to optimize without dynamic model? And whether the cost function can be interpreted as the pixel difference between key points？

Oct 17 '23 15:10 Sjey-Lyn

Hi, thank you for your interest in our work! Yes, it is possible to optimize without dynamic models. If the action is only pick and place, you can assume the dynamics model is just a 3D rigid transformation. For your second question, yes it is correct.

Oct 17 '23 17:10 WangYixuan12

Thank you for your answer, is it convenient to provide the code for the robotic arm simulation？

Oct 18 '23 00:10 Sjey-Lyn

I may consider that if many people want it, but it may take a while. (pls leave a thumbup if you want this feature).

Oct 18 '23 01:10 WangYixuan12

Okay, thanks.

Oct 18 '23 01:10 Sjey-Lyn

I will leave this issue open so that people could comment if they want this feature.

Oct 18 '23 01:10 WangYixuan12

I have another question about how to choose a reference camera to project 3D keypoints into 2D images？

Oct 18 '23 01:10 Sjey-Lyn

Currently, we are manually setting the reference camera pose. But it is possible to do it automatically.

Oct 18 '23 01:10 WangYixuan12

Thank you for your excellent work! May I ask how to train my own model?

Oct 26 '23 02:10 Gloryseven

Hi, actually our work does not need to train the model. You only need the off-shelf foundation models.

Oct 26 '23 03:10 WangYixuan12

Hi, actually our work does not need to train the model. You only need the off-shelf foundation models.

Thank you for your timely reply！ What are the GPU requirements for running this model？ And how can I get the robotic simulation in the paper?

Oct 26 '23 06:10 Gloryseven

We use Nvidia 3090 to run the model. For the simulation. I may need to find some time to organize the code. We use OmniGibson for simulation

Get Outlook for iOShttps://aka.ms/o0ukef

From: Gloryseven @.> Sent: Thursday, October 26, 2023 1:14:46 AM To: WangYixuan12/d3fields @.> Cc: Wang, Yixuan @.>; State change @.> Subject: Re: [WangYixuan12/d3fields] Confusing about optimize (Issue #3)

Hi, actually our work does not need to train the model. You only need the off-shelf foundation models.

Thank you for your timely reply！ What are the GPU requirements for running this model？ And how can I get the robotic simulation in the paper?

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/WangYixuan12/d3fields/issues/3*issuecomment-1780476724__;Iw!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPkmcbKmKzg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AHWV3D5G7SLAS3RVFPAZ2PDYBH5VNAVCNFSM6AAAAAA6ECQZUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQGQ3TMNZSGQ__;!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPknX4UGVrA$. You are receiving this because you modified the open/close state.Message ID: @.***>

Oct 26 '23 06:10 WangYixuan12

May I ask where these datasets come from? How do I create my own dataset?

Oct 27 '23 06:10 Gloryseven

I collect the dataset on my own using four RGBD cameras. You could use RGBD cameras and create data using a similar file structure.

Oct 27 '23 17:10 WangYixuan12

We use Nvidia 3090 to run the model. For the simulation. I may need to find some time to organize the code. We use OmniGibson for simulation Get Outlook for iOShttps://aka.ms/o0ukef … ________________________________ From: Gloryseven @.> Sent: Thursday, October 26, 2023 1:14:46 AM To: WangYixuan12/d3fields @.> Cc: Wang, Yixuan @.>; State change @.> Subject: Re: [WangYixuan12/d3fields] Confusing about optimize (Issue #3) Hi, actually our work does not need to train the model. You only need the off-shelf foundation models. Thank you for your timely reply！ What are the GPU requirements for running this model？ And how can I get the robotic simulation in the paper? — Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/WangYixuan12/d3fields/issues/3*issuecomment-1780476724__;Iw!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPkmcbKmKzg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AHWV3D5G7SLAS3RVFPAZ2PDYBH5VNAVCNFSM6AAAAAA6ECQZUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQGQ3TMNZSGQ__;!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPknX4UGVrA$. You are receiving this because you modified the open/close state.Message ID: @.***>

Hello! I'm very interested in this work. I want ask some question:

Does the OmniGibson for simulation means the 'planning' part in the paper? And are all the experiments conducted in a simulation environment? So I do not need a real robotic arm.
And is the part of the robotic code, along with the already open source code, is the entire code about the paper? Is the robotic arm simulation code writed by OmniGibson?

Nov 10 '23 02:11 Gloryseven

Yes, the sim code corresponds to the planning part of the paper. But we also have real robot experiments.
Yes (+ planning code for real-world experiments)

Nov 10 '23 02:11 WangYixuan12

Thank you for your reply！ If I use simulation robot experiments, will the dataset (RGBD images) be captured and generated by the OmniGibson simulation platform?

该邮件从移动设备发送

------------------ 原始邮件 ------------------ 发件人: "WangYixuan12/d3fields" @.>; 发送时间: 2023年11月10日(星期五) 上午10:43 @.>; @.@.>; 主题: Re: [WangYixuan12/d3fields] Confusing about optimize (Issue #3)

Yes, the sim code corresponds to the planning part of the paper. But we also have real robot experiments.

Yes (+ planning code for real-world experiments)

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Nov 10 '23 03:11 Gloryseven

Yep

Nov 10 '23 03:11 WangYixuan12

Hi, actually our work does not need to train the model. You only need the off-shelf foundation models.

What about the pretrained model in the readme 'bash scripts/download_ckpts.sh'. Does it mean the Optimization process in the OmniGibson? Is the pretrained model conducted by OmniGibson?

Nov 10 '23 03:11 Gloryseven

Omnigibson is a simulation platform, while the pre-trained model is to construct our representation

Nov 10 '23 03:11 WangYixuan12

What do you see as the limitations of your work？

Nov 25 '23 03:11 Bailey-24

I think the control part could be more advanced.

Dec 04 '23 03:12 WangYixuan12

How many sets of matching points are needed to calculate the transformation matrix?

Jan 11 '24 07:01 Sjey-Lyn

More matching points lead to a more stable transformation matrix. A typical choice is 100 points

Jan 15 '24 17:01 WangYixuan12

the knife blade is not match the goal image

Feb 26 '24 03:02 Bailey-24

Since DINOv2 cannot distinguish the sides of blades, it is expected that these two cannot match

Mar 02 '24 02:03 WangYixuan12

I found the similar using DINO to find correspond feature in this repo. but he need the goal image's depth to find the transformation. When we use the goal image using AI to generate image, it does have depth information, what is you method? I just find in the code compare two images difference, https://github.com/WangYixuan12/d3fields/blob/5158f48ac6314bd9fbfea532b4c0e40a11493c17/fusion.py#L1729 , but i have no idea how to implement to get the transformation? Do you use the learned MPC to directly get the action?

https://gist.github.com/normandipalo/fbc21f23606fbe3d407e22c363cb134e

Mar 05 '24 13:03 Bailey-24

Actually, our method does not need a depth image to obtain the goal image. We assume that there is a floating reference camera in the workspace. The projected 2D image will be compared with goal image without the need for depth image

Mar 18 '24 06:03 WangYixuan12

d3fields d3fields copied to clipboard

Confusing about optimize

d3fields
d3fields copied to clipboard