IsaacGymEnvs
IsaacGymEnvs copied to clipboard
Published
20 hours ago •
NVIDIA-Omniverse
NVIDIA-Omniverse
Factory environment on remote server, Segmentation fault (core dumped)
Thanks for the great job on IsaacGymEnvs! When I ran the demo 'python train.py task=FactoryTaskNutBoltScrew', I met 'Segmentation fault (core dumped)', The output is as follows:
Importing module 'gym_37' (/home/slc/env/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/slc/env/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
train.py:49: UserWarning:
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
@hydra.main(config_name="config", config_path="./cfg")
/home/slc/miniconda3/envs/rlgpu/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'config': Defaults list is missing `_self_`. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
warnings.warn(msg, UserWarning)
/home/slc/miniconda3/envs/rlgpu/lib/python3.7/site-packages/hydra/_internal/defaults_list.py:415: UserWarning: In config: Invalid overriding of hydra/job_logging:
Default list overrides requires 'override' keyword.
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/defaults_list_override for more information.
deprecation_warning(msg)
/home/slc/miniconda3/envs/rlgpu/lib/python3.7/site-packages/hydra/_internal/hydra.py:127: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
configure_logging=with_log_configuration,
/home/slc/miniconda3/envs/rlgpu/lib/python3.7/site-packages/torch/utils/cpp_extension.py:3: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
PyTorch version 1.8.1
Device count 2
/home/slc/env/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/slc/.cache/torch_extensions as PyTorch extensions root...
Emitting ninja build file /home/slc/.cache/torch_extensions/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
/home/slc/env/isaacgym/python/isaacgym/torch_utils.py:135: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
def get_axis_params(value, axis_idx, x_value=0., dtype=np.float, n_dims=3):
2022-11-04 16:54:09,740 - INFO - logger - logger initialized
<unknown>:6: DeprecationWarning: invalid escape sequence \*
Error: FBX library failed to load - importing FBX data will not succeed. Message: No module named 'fbx'
FBX tools must be installed from https://help.autodesk.com/view/FBX/2020/ENU/?guid=FBX_Developer_Help_scripting_with_python_fbx_installing_python_fbx_html
/home/slc/miniconda3/envs/rlgpu/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py:3: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if not hasattr(tensorboard, '__version__') or LooseVersion(tensorboard.__version__) < LooseVersion('1.15'):
task:
name: FactoryTaskNutBoltScrew
physics_engine: physx
sim:
use_gpu_pipeline: True
up_axis: z
dt: 0.016667
gravity: [0.0, 0.0, -9.81]
disable_gravity: False
env:
numEnvs: 128
numObservations: 32
numActions: 12
randomize:
franka_arm_initial_dof_pos: [0.0015178, -0.19651, -0.0014364, -1.9761, -0.00027717, 1.7796, 0.78556]
nut_rot_initial: 30.0
rl:
pos_action_scale: [0.1, 0.1, 0.1]
rot_action_scale: [0.1, 0.1, 0.1]
force_action_scale: [1.0, 1.0, 1.0]
torque_action_scale: [1.0, 1.0, 1.0]
unidirectional_rot: True
unidirectional_force: False
clamp_rot: True
clamp_rot_thresh: 1e-06
add_obs_finger_force: False
keypoint_reward_scale: 1.0
action_penalty_scale: 0.0
max_episode_length: 4096
far_error_thresh: 0.1
success_bonus: 0.0
ctrl:
ctrl_type: operational_space_motion
all:
jacobian_type: geometric
gripper_prop_gains: [100, 100]
gripper_deriv_gains: [1, 1]
gym_default:
ik_method: dls
joint_prop_gains: [40, 40, 40, 40, 40, 40, 40]
joint_deriv_gains: [8, 8, 8, 8, 8, 8, 8]
gripper_prop_gains: [500, 500]
gripper_deriv_gains: [20, 20]
joint_space_ik:
ik_method: dls
joint_prop_gains: [1, 1, 1, 1, 1, 1, 1]
joint_deriv_gains: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]
joint_space_id:
ik_method: dls
joint_prop_gains: [40, 40, 40, 40, 40, 40, 40]
joint_deriv_gains: [8, 8, 8, 8, 8, 8, 8]
task_space_impedance:
motion_ctrl_axes: [1, 1, 1, 1, 1, 1]
task_prop_gains: [40, 40, 40, 40, 40, 40]
task_deriv_gains: [8, 8, 8, 8, 8, 8]
operational_space_motion:
motion_ctrl_axes: [0, 0, 1, 0, 0, 1]
task_prop_gains: [1, 1, 1, 1, 1, 100]
task_deriv_gains: [1, 1, 1, 1, 1, 1]
open_loop_force:
force_ctrl_axes: [0, 0, 1, 0, 0, 0]
closed_loop_force:
force_ctrl_axes: [0, 0, 1, 0, 0, 0]
wrench_prop_gains: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1]
hybrid_force_motion:
motion_ctrl_axes: [1, 1, 0, 1, 1, 1]
task_prop_gains: [40, 40, 40, 40, 40, 40]
task_deriv_gains: [8, 8, 8, 8, 8, 8]
force_ctrl_axes: [0, 0, 1, 0, 0, 0]
wrench_prop_gains: [0.1, 0.1, 0.1, 0.1, 0.1, 0.1]
train:
params:
seed: 42
algo:
name: a2c_continuous
model:
name: continuous_a2c_logstd
network:
name: actor_critic
separate: False
space:
continuous:
mu_activation: None
sigma_activation: None
mu_init:
name: default
sigma_init:
name: const_initializer
val: 0
fixed_sigma: True
mlp:
units: [256, 128, 64]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
load_checkpoint: True
load_path: /home/slc/env/IsaacGymEnvs-main/isaacgymenvs/runs/FactoryTaskNutBoltScrew/nn/FactoryTaskNutBoltScrew.pth
config:
name: FactoryTaskNutBoltScrew
full_experiment_name: FactoryTaskNutBoltScrew
env_name: rlgpu
multi_gpu: False
ppo: True
mixed_precision: True
normalize_input: True
normalize_value: True
value_bootstrap: True
num_actors: 128
reward_shaper:
scale_value: 1.0
normalize_advantage: True
gamma: 0.99
tau: 0.95
learning_rate: 0.0001
lr_schedule: fixed
schedule_type: standard
kl_threshold: 0.016
score_to_win: 20000
max_epochs: 1024
save_best_after: 50
save_frequency: 100
print_stats: True
grad_norm: 1.0
entropy_coef: 0.0
truncate_grads: False
e_clip: 0.2
horizon_length: 32
minibatch_size: 512
mini_epochs: 8
critic_coef: 2
clip_value: True
seq_len: 4
bounds_loss_coef: 0.0001
device: cuda:0
task_name: FactoryTaskNutBoltScrew
experiment:
num_envs:
seed: 42
torch_deterministic: False
max_iterations:
physics_engine: physx
pipeline: gpu
sim_device: cuda:0
rl_device: cuda:0
graphics_device_id: 0
num_threads: 4
solver_type: 1
num_subscenes: 4
test: False
checkpoint: /home/slc/env/IsaacGymEnvs-main/isaacgymenvs/runs/FactoryTaskNutBoltScrew/nn/FactoryTaskNutBoltScrew.pth
multi_gpu: False
wandb_activate: False
wandb_group:
wandb_name: FactoryTaskNutBoltScrew
wandb_entity:
wandb_project: isaacgymenvs
capture_video: False
capture_video_freq: 1464
capture_video_len: 100
force_render: True
headless: False
Setting seed: 42
self.seed = 42
Started to train
Exact experiment name requested from command line: FactoryTaskNutBoltScrew
/home/slc/miniconda3/envs/rlgpu/lib/python3.7/site-packages/gym/spaces/box.py:112: UserWarning: WARN: Box bound precision lowered by casting to float32
logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
Not connected to PVD
+++ Using GPU PhysX
Physics Engine: PhysX
Physics Device: cuda:0
GPU Pipeline: enabled
WARNING: lavapipe is not a conformant vulkan implementation, testing use only.
/home/slc/env/IsaacGymEnvs-main/isaacgymenvs/tasks/factory/factory_base.py:507: UserWarning: WARN: Please be patient: SDFs may be generating, which may take a few minutes. Terminating prematurely may result in a corrupted SDF cache.
logger.warn('Please be patient: SDFs may be generating, which may take a few minutes. Terminating prematurely may result in a corrupted SDF cache.')
Using SDF cache directory '/home/slc/.isaacgym/sdf_V100'
~!~!~! Loaded/Cooked SDF triangle mesh 0 @ 0x55b23f95ae80, resolution=256, spacing=0.000108
~!~! Bounds: (-0.012000, 0.012000) (-0.013856, 0.013856) (0.016000, 0.029000)
~!~! Extents: (0.024000, 0.027712, 0.013000)
~!~! Resolution: (222, 256, 121)
~!~!~! Loaded/Cooked SDF triangle mesh 1 @ 0x55b245831410, resolution=512, spacing=0.000080
~!~! Bounds: (-0.012000, 0.012000) (-0.012000, 0.012000) (0.000000, 0.041000)
~!~! Extents: (0.024000, 0.024000, 0.041000)
~!~! Resolution: (300, 300, 512)
~!~!~! Loaded/Cooked SDF triangle mesh 2 @ 0x55b23e473a10, resolution=256, spacing=0.000108
~!~! Bounds: (-0.012000, 0.012000) (-0.013856, 0.013856) (0.016000, 0.029000)
~!~! Extents: (0.024000, 0.027712, 0.013000)
~!~! Resolution: (222, 256, 121)
~!~!~! Loaded/Cooked SDF triangle mesh 3 @ 0x55b249372f10, resolution=512, spacing=0.000080
~!~! Bounds: (-0.012000, 0.012000) (-0.012000, 0.012000) (0.000000, 0.041000)
~!~! Extents: (0.024000, 0.024000, 0.041000)
~!~! Resolution: (300, 300, 512)
Box(-1.0, 1.0, (12,), float32) Box(-inf, inf, (32,), float32)
current training device: cuda:0
build mlp: 32
RunningMeanStd: (1,)
RunningMeanStd: (32,)
=> loading checkpoint '/home/slc/env/IsaacGymEnvs-main/isaacgymenvs/runs/FactoryTaskNutBoltScrew/nn/FactoryTaskNutBoltScrew.pth'
/home/slc/env/IsaacGymEnvs-main/isaacgymenvs/tasks/factory/factory_control.py:145: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
task_wrench = task_wrench + torch.tensor(cfg_ctrl['motion_ctrl_axes'], device=device).unsqueeze(0) * task_wrench_motion
Unhandled descriptor set 433
Unhandled descriptor set 1176522960
Unhandled descriptor set 1177337120
Segmentation fault (core dumped)
If I use the 'headless=True', the error will disappear. I am aware that the remote server needs the graphic display, so I use X11. And I used the method in other demos like Ant and ShadowHand, It seems to work well. so I am confused why the error will appear in the factory environment. I tried the method like #22 , updating the Nvidia driver, but it doesn't work, either.
