blender System is out of GPU memory

[tianxiaguixin002]

Hi author, I try the Generate image(s) in one command and success generate 'hello_world' folder in outputs:

then i run run_pipeline.sh, the gpu memory error occurs:

In bleander webser, it says:

My gpu is 3080 10G, memory is 64G, how to adjust the blender render Device in run_pipeline.sh, what is the default device in run_pipeline.sh? Thank you !

[bssrdf]

Sorry, this is a question, not the solution to your problem.

How did you enable GPU rendering of Blender? According to the author, only terrain generation step can use GPUs.

See here.

[badgids]

This is how I did it. I have a RTX 4090 with 24GB of Ram, and I had to set manage_datagen_jobs.num_concurrent=1 and LocalScheduleHandler.jobs_per_gpu=1 or else I got out of memory errors. I'm pretty sure this was meant to run on a render farm. lol

Edit the enable_gpu.gin in infinigen\worldgen\tools\pipeline_configs to this:

enable_gpu.gin

include 'tools/pipeline_configs/local_256GB.gin'

queue_fine_terrain.gpus = 1
queue_combined.gpus = 1
queue_fine_terrain.exclude_gpus = ['k80']
queue_combined.exclude_gpus = ['k80']
manage_datagen_jobs.num_concurrent=1
LocalScheduleHandler.use_gpu=True
LocalScheduleHandler.jobs_per_gpu=1

And if you want to customize the world generation/render settings, those are found in infinigen\worldgen\config. You can copy one to a new file and play around to see what you get. Here is a config that changes the render/output resolution. I've named it dev_small.gin.

dev_small.gin

compose_scene.generate_resolution = (1280, 720)

full/render_image.min_samples = 32
full/render_image.num_samples = 512

OpaqueSphericalMesher.pixels_per_cube = 4
TransparentSphericalMesher.pixels_per_cube = 4
target_face_size.global_multiplier = 2

compose_scene.ground_creatures_chance = 0.0
compose_scene.flying_creatures_chance = 0.0

compose_scene.inview_distance = 35
placement.populate_all.dist_cull = 35
compose_scene.near_distance = 10
compose_scene.center_distance = 20

If you would like to adjust somehing like the monoculer settings, like to output a 5 scond video, you can do that like this:

monocular_video_5_sec.gin

# frame segment to render. This example outputs the first 5 seconds of
# video at 30 fps.
iterate_scene_tasks.frame_range = [1, 150]
# Default 16, change this to as large as your GPU can handle. 
# This is the tile size that blender renders at a time.
iterate_scene_tasks.view_block_size = 2048

iterate_scene_tasks.cam_id_ranges = [1, 1]

iterate_scene_tasks.global_tasks = [
    {'name': 'coarse', 'func': @queue_coarse}, 
    {'name': "populate", 'func': @queue_populate},
    {'name': 'backuppopulate', 'func': @backup/queue_populate, 'condition': 'prev_failed'}
]
iterate_scene_tasks.view_dependent_tasks = [
    {'name': "fineterrain", 'func': @queue_fine_terrain}, 
]

# Choosing to use blendergt ground truth
iterate_scene_tasks.camera_dependent_tasks = [
    {'name': 'short', 'func': @short/queue_render}, 
    {'name': 'backup', 'func': @backup/queue_render, 'condition': 'prev_failed'}, 
    {'name': 'blendergt', 'func': @ground_truth/queue_render}
]

You can call the newly created settings from the cli, like so:

python -m tools.manage_datagen_jobs --output_folder outputs/hello_world --num_scenes 1 
--pipeline_configs enable_gpu monocular_video_5_sec blender_gt --specific_seed 0 --configs desert dev_small

Cheers!

[araistrick]

@bssrdf "Only uses gpu for terrain" was an incorrect statement by one of us, we use the GPU heavily for rendering, and CPU rendering is not really a viable option.

@badgids iterate_scene_tasks.view_block_size = 2048 actually controls how many frames of video will be rendered from each .blend file before making a new fine mesh blend file, essentially how often will we remesh the terrain in accordance with new views. I recommend leaving it at 16. There is another setting in blender for cycles block size but this is separate.

We actually use 3090s/4090s for almost all our rendering :) so your spec is pretty much maxed out. jobs_per_gpu=1 should be the default iirc, it used to be set to 2 early in development when the scenes were less detailed and 2 could fit within 24GB of VRAM. But this is no longer the case.

Also, thanks for all your help! really thrilled to see people digging into the system and customizing it

[araistrick]

Regarding the title of this issue, if the GPU is being used and you are running out of VRAM, there is likely very little we can do. In the current system, many scenes are too big for 10GB of VRAM. We are actively working on bringing down these performance requirements. For now, you could try keeping some elements of the simple config or otherwise reducing scene complexity until it fits your GPU.

[cyfex]

I have only a 4G GPU memory. Please make it possible to run on that because currently the price of graphics cards is insane... I can accept slower speed but please at least accept a GPU instead of having to run on a CPU...

Solution

Well, this problem can be solved by using CUDA instead of OPTIX in rendering, because CUDA allows you to use CPU and RAM along with your GPU and VRAM.

To be specific in infinigen, edit rendering/render.py at line 106: change

for gpu_type in ['OPTIX', 'CUDA']:#, 'METAL']:

to

for gpu_type in ['CUDA']:
``

[tianxiaguixin002]

I have only a 4G GPU memory. Please make it possible to run on that because currently the price of graphics cards is insane... I can accept slower speed but please at least accept a GPU instead of having to run on a CPU...

Solution

Well, this problem can be solved by using CUDA instead of OPTIX in rendering, because CUDA allows you to use CPU and RAM along with your GPU and VRAM.

To be specific in infinigen, edit rendering/render.py at line 106: change

for gpu_type in ['OPTIX', 'CUDA']:#, 'METAL']:

to

for gpu_type in ['CUDA']:
``

I also have used same method, it works. Great !