Is real-time visualization possible in Google Colab? Attempted frame-by-frame image conversion as an alternative

[ardkyer]

trafficstars

Issue Description

When running Genesis in Google Colab environment, real-time visualization is not possible due to OpenGL restrictions. Using show_viewer=True option results in an error.

Alternative Approach

I implemented visualization by saving images frame by frame. Here's a minimal working example:

import genesis as gs
import numpy as np
from PIL import Image
import os

gs.init(backend=gs.cuda)

# Create a directory to save the results
save_dir = "/content/simulation_frames"
os.makedirs(save_dir, exist_ok=True)

scene = gs.Scene(show_viewer=False)

# Add objects
plane = scene.add_entity(gs.morphs.Plane())
box = scene.add_entity(
    gs.morphs.Box(
        pos=(0, 0, 2),
        size=(0.5, 0.5, 0.5)
    )
)

# Camera settings
cam = scene.add_camera(
    res=(320, 240),
    pos=(2.0, 2.0, 2.0),
    lookat=(0, 0, 0),
    fov=30,
)

scene.build()

# Render and save images
for i in range(30):
    scene.step()
    if i % 5 == 0:
        print(f"Step {i}/30")
        rgb_data = cam.render(rgb=True)
        
        # rgb_data[0] contains the actual image data
        img_array = rgb_data[0]
        img = Image.fromarray((img_array * 255).astype(np.uint8))
        img.save(f"{save_dir}/frame_{i:03d}.png")
        print(f"Frame {i} saved to {save_dir}")

Results

While real-time visualization is not possible, frame-by-frame image saving works successfully FPS varies by environment, but achieves about 10-15 FPS on A100 GPU Saved images can be found in the '/content/simulation_frames' directory

Questions

Are there any other methods to achieve real-time visualization in Colab? #166 Are there ways to improve the performance of the current approach? Is real-time visualization possible in other cloud environments?

Environment Details

Google Colab GPU: NVIDIA A100-SXM4-40GB