[Question] Debug Drawing in ManiSkill

[hesic73]

trafficstars

Isaac Sim provides debug drawing utilities. Does ManiSkill offer a similar feature for visual debugging?

[StoneT2000]

Do you have a picture of what these look like? We probably support it since SAPIEN's GUI viewer supports some visual annotations (e.g. the xyz axes when u click an object and try to transform it).

[hesic73]

The DebugDraw class in Isaac Sim allows users to draw points and lines for visual debugging. The interface looks like this:

from torch import Tensor
from omni.isaac.debug_draw import _debug_draw

from mani_skill.envs.sapien_env import BaseEnv


class MyEnv(BaseEnv):
    def __init__(self, *args, **kwargs):
        self.draw = _debug_draw._debug_draw.acquire_debug_draw_interface()
        self.central_env_idx = 0

        super().__init__(*args, **kwargs)

    def _initialize_episode(self, env_idx: Tensor, options: dict):
        if (env_idx == self.central_env_idx).any():
            self.draw.clear_lines()
            self.draw.clear_points()

            line_start_points = [
                [0, 0, 0],
            ]
            line_end_points = [
                [0, 0, 1],
            ]
            line_colors = [(1, 0, 0, 1)] * 1
            line_sizes = [1.0] * 1
            self.draw.draw_lines(line_start_points,
                                 line_end_points, line_colors, line_sizes)

            points = [
                [0, 0.5, 0],
            ]
            point_colors = [(0, 1, 0, 1)] * 1
            point_sizes = [3.0] * 1
            self.draw.draw_points(points, point_colors, point_sizes)

I also have a video demonstrating this in OmniDrones:

https://github.com/user-attachments/assets/4fbe60e1-162b-4fc3-b554-996dfbb88e80

[StoneT2000]

I understand now. currently there isn't documentation on it. It does exist though. It only works on the "default" shader pack option.

The code to do so

import numpy as np
import mani_skill.envs
import gymnasium as gym
env = gym.make("PickCube-v1", viewer_camera_configs=dict(shader_pack="default"))
env.reset()
viewer = env.render_human()
viewer.draw_aabb(np.array([0, 0, 0]), np.array([1, 1, 1]), np.array([1, 0, 0]))
while True:
  env.step(env.action_space.sample())
  env.render_human()

(note this is using the new shader pack configuration system in ManiSkill, on the latest git commit, shaders for cameras are customized per camera now instead of globally which was done with the shader_dir=... option in gym.make, explanation here: https://maniskill.readthedocs.io/en/latest/user_guide/tutorials/sensors/index.html#cameras).

[StoneT2000]

This is the code in the viewer code that was used for the bounding box.

    def add_bounding_box(self, pose, half_size, color):
        vertices = np.array(
            [
                [1, -1, -1],
                [1, 1, -1],
                [-1, 1, -1],
                [-1, -1, -1],
                [1, -1, 1],
                [1, 1, 1],
                [-1, 1, 1],
                [-1, -1, 1],
            ]
        )
        lines = [0, 1, 1, 2, 2, 3, 3, 0, 4, 5, 5, 6, 6, 7, 7, 4, 0, 4, 1, 5, 2, 6, 3, 7]
        vertices = vertices[lines]
        colors = np.ones((vertices.shape[0], 4)) * [*color[:3], 1]
        lineset = self.renderer_context.create_line_set(vertices, colors)

        # render_scene: R.Scene = self.system._internal_scene
        box = self.render_scene.add_line_set(lineset)
        box.set_position(pose.p)
        box.set_rotation(pose.q)
        box.set_scale(half_size)

        return box

    def update_bounding_box(self, box, pose, half_size):
        box.set_position(pose.p)
        box.set_rotation(pose.q)
        box.set_scale(half_size)

    def remove_bounding_box(self, box):
        # render_scene: R.Scene = self.system._internal_scene
        self.render_scene.remove_node(box)

    def draw_aabb(self, lower, upper, color):
        pose = sapien.Pose((lower + upper) / 2)
        half_size = (upper - lower) / 2
        return self.add_bounding_box(pose, half_size, color)

    def update_aabb(self, aabb, lower, upper):
        pose = sapien.Pose((lower + upper) / 2)
        half_size = (upper - lower) / 2
        self.update_bounding_box(aabb, pose, half_size)

self here is the Viewer object which you can fetch via viewer = env.render_human(). You can customize this further to draw other kinds of shapes via lines.