关于camera motion的prompt

[aulaywang]

我试着在prompt里增加camera motion相关的表述（比如zoom in），发现效果不佳。同时，我发现Gradio下有关于camera motion的选项链接，然而我没有在作者的command案例里找到类似的设置方式链接。有人了解怎么设置吗？

[JThh]

I think it is being auto-added during inference without having to manually add to configs. See: https://github.com/hpcaitech/Open-Sora/blob/ee909a7d6611bcc9c5cf1ac055a7e9cc74157e09/docs/report_03.md?plain=1#L121

[aulaywang]

I think it is being auto-added during inference without having to manually add to configs. See:

https://github.com/hpcaitech/Open-Sora/blob/ee909a7d6611bcc9c5cf1ac055a7e9cc74157e09/docs/report_03.md?plain=1#L121

Can you give me an example? Btw, I tried to add camera_motion = 'zoom in' in my config (configs/opensora-v1-2/inference/sample.py), it still didn't work.

[FrankLeeeee]

We use this function https://github.com/hpcaitech/Open-Sora/blob/main/opensora/utils/inference_utils.py#L55 to append camera motion to prompts.

Just to double check if the camera_motion value is actually there, you may want to print out the value by inserting the following line to https://github.com/hpcaitech/Open-Sora/blob/main/scripts/inference.py#L232.

print("Camera motion: ", cfg.get("camera_motion", None))

[aulaywang]

We use this function https://github.com/hpcaitech/Open-Sora/blob/main/opensora/utils/inference_utils.py#L55 to append camera motion to prompts.

Just to double check if the camera_motion value is actually there, you may want to print out the value by inserting the following line to https://github.com/hpcaitech/Open-Sora/blob/main/scripts/inference.py#L232.

print("Camera motion: ", cfg.get("camera_motion", None))

I hope it’s my mistake, but based on the debug results, the model indeed got camera motion and did not produce the correct output. I hope you can take a look at this command for generating the video from image.

python scripts/inference.py configs/opensora-v1-2/inference/sample.py --num-frames 32 --image-size 360 640 --sample-name image-cond --prompt 'The waves are surging inside the house.'  --reference-path 'assets/images/condition/wave.png' --mask-strategy "0"  --camera-motion "zoom in"

[JThh]

What is your result printed from print("Camera motion: ", cfg.get("camera_motion", None))?

[github-actions[bot]]

This issue is stale because it has been open for 7 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 7 days since being marked as stale.