ray [RLlib] Custom RLModule import fails using absolute imports (with non-zero `env

[RLlib] Custom RLModule import fails using absolute imports (with non-zero `env_runners`)

Open mantasu opened this issue 3 weeks ago • 2 comments

What happened + What you expected to happen

Following the official tutorial, it is possible to specify custom RLModules. However, after importing them from a custom local package, I get errors about that custom package not being found. The custom RLModule can only be found if it is defined within the same file or a relative import is used.

Versions / Dependencies

rllib==2.24.0, python=3.11

Reproduction script

File structure:

└── my_project
    ├── notebooks
    │   └── main.ipynb
    │
    └── src/my_module
        ├── __init__.py
        └── my_ppo_torch_rl_module.py

my_ppo_torch_rl_module.py content:

from ray.rllib.algorithms.ppo.torch.ppo_torch_rl_module import PPOTorchRLModule

class MyPPOTorchRLModule(PPOTorchRLModule):
    pass

main.ipynb content (single cell):

import sys

sys.path.append("../src")

from ray.rllib.algorithms.ppo.ppo import PPOConfig
from ray.rllib.core.rl_module.rl_module import SingleAgentRLModuleSpec
from ray.rllib.algorithms.ppo.torch.ppo_torch_rl_module import PPOTorchRLModule

from my_module.my_ppo_torch_rl_module import MyPPOTorchRLModule

config = (
    PPOConfig()
    .api_stack(enable_rl_module_and_learner=True)
    .rl_module(
        rl_module_spec=SingleAgentRLModuleSpec(module_class=MyPPOTorchRLModule)
    )
    .environment("CartPole-v1")
    .training(train_batch_size=32, sgd_minibatch_size=16, num_sgd_iter=1)
)

config.build()

Error traceback

---------------------------------------------------------------------------
ActorDiedError                            Traceback (most recent call last)
File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:169, in EnvRunnerGroup.__init__(self, env_creator, validate_env, default_policy_class, config, num_env_runners, local_env_runner, logdir, _setup, num_workers, local_worker)
    [168](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:168) try:
--> [169](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:169)     self._setup(
    [170](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:170)         validate_env=validate_env,
    [171](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:171)         config=config,
    [172](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:172)         num_env_runners=num_env_runners,
    [173](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:173)         local_env_runner=local_env_runner,
    [174](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:174)     )
    [175](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:175) # EnvRunnerGroup creation possibly fails, if some (remote) workers cannot
    [176](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:176) # be initialized properly (due to some errors in the EnvRunners's
    [177](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:177) # constructor).

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:239, in EnvRunnerGroup._setup(self, validate_env, config, num_env_runners, local_env_runner)
    [238](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:238) # Create a number of @ray.remote workers.
--> [239](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:239) self.add_workers(
    [240](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:240)     num_env_runners,
    [241](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:241)     validate=config.validate_env_runners_after_construction,
    [242](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:242) )
    [244](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:244) # If num_env_runners > 0 and we don't have an env on the local worker,
    [245](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:245) # get the observation- and action spaces for each policy from
    [246](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:246) # the first remote worker (which does have an env).

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:754, in EnvRunnerGroup.add_workers(self, num_workers, validate)
    [753](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:753) if not result.ok:
--> [754](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:754)     raise result.get()

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/utils/actor_manager.py:500, in FaultTolerantActorManager._fetch_result(self, remote_actor_ids, remote_calls, tags, timeout_seconds, return_obj_refs, mark_healthy)
    [499](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/utils/actor_manager.py:499) try:
--> [500](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/utils/actor_manager.py:500)     result = ray.get(ready)
    [501](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/utils/actor_manager.py:501)     remote_results.add_result(actor_id, ResultOrError(result=result), tag)

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/auto_init_hook.py:21, in wrap_auto_init.<locals>.auto_init_wrapper(*args, **kwargs)
     [20](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/auto_init_hook.py:20) auto_init_ray()
---> [21](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/auto_init_hook.py:21) return fn(*args, **kwargs)

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/client_mode_hook.py:103, in client_mode_hook.<locals>.wrapper(*args, **kwargs)
    [102](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/client_mode_hook.py:102)         return getattr(ray, func.__name__)(*args, **kwargs)
--> [103](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/client_mode_hook.py:103) return func(*args, **kwargs)

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/worker.py:2613, in get(object_refs, timeout)
   [2612](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/worker.py:2612) # TODO(ujvl): Consider how to allow user to retrieve the ready objects.
-> [2613](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/worker.py:2613) values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
   [2614](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/worker.py:2614) for i, value in enumerate(values):

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/worker.py:863, in Worker.get_objects(self, object_refs, timeout)
    [862](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/worker.py:862)         else:
--> [863](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/worker.py:863)             raise value
    [864](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/worker.py:864) return values, debugger_breakpoint

ActorDiedError: The actor died because of an error raised in its creation task, ray::RolloutWorker.__init__() (pid=345329, ip=172.29.231.252, actor_id=98c6b7d296129f2c6c58738401000000, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x7f2009463690>)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RaySystemError: System error: No module named 'my_module'
traceback: Traceback (most recent call last):
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          ^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'my_module'

During handling of the above exception, another exception occurred:

RaySystemError                            Traceback (most recent call last)
Cell In[17], [line 22](vscode-notebook-cell:?execution_count=17&line=22)
      [8](vscode-notebook-cell:?execution_count=17&line=8) # class MyPPOTorchRLModule(PPOTorchRLModule):
      [9](vscode-notebook-cell:?execution_count=17&line=9) #     pass
     [12](vscode-notebook-cell:?execution_count=17&line=12) config = (
     [13](vscode-notebook-cell:?execution_count=17&line=13)     PPOConfig()
     [14](vscode-notebook-cell:?execution_count=17&line=14)     .api_stack(enable_rl_module_and_learner=True)
   (...)
     [19](vscode-notebook-cell:?execution_count=17&line=19)     .training(train_batch_size=32, sgd_minibatch_size=16, num_sgd_iter=1)
     [20](vscode-notebook-cell:?execution_count=17&line=20) )
---> [22](vscode-notebook-cell:?execution_count=17&line=22) config.build()

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm_config.py:868, in AlgorithmConfig.build(self, env, logger_creator, use_copy)
    [865](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm_config.py:865) if isinstance(self.algo_class, str):
    [866](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm_config.py:866)     algo_class = get_trainable_cls(self.algo_class)
--> [868](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm_config.py:868) return algo_class(
    [869](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm_config.py:869)     config=self if not use_copy else copy.deepcopy(self),
    [870](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm_config.py:870)     logger_creator=self.logger_creator,
    [871](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm_config.py:871) )

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:554, in Algorithm.__init__(self, config, env, logger_creator, **kwargs)
    [540](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:540) # Initialize common evaluation_metrics to nan, before they become
    [541](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:541) # available. We want to make sure the metrics are always present
    [542](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:542) # (although their values may be nan), so that Tune doesn't complain
    [543](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:543) # when we use these as stopping criteria.
    [544](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:544) self.evaluation_metrics = {
    [545](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:545)     EVALUATION_RESULTS: {
    [546](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:546)         ENV_RUNNER_RESULTS: {
   (...)
    [551](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:551)     },
    [552](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:552) }
--> [554](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:554) super().__init__(
    [555](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:555)     config=config,
    [556](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:556)     logger_creator=logger_creator,
    [557](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:557)     **kwargs,
    [558](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:558) )

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/tune/trainable/trainable.py:158, in Trainable.__init__(self, config, logger_creator, storage)
    [154](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/tune/trainable/trainable.py:154)     logger.debug(f"StorageContext on the TRAINABLE:\n{storage}")
    [156](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/tune/trainable/trainable.py:156) self._open_logfiles(stdout_file, stderr_file)
--> [158](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/tune/trainable/trainable.py:158) self.setup(copy.deepcopy(self.config))
    [159](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/tune/trainable/trainable.py:159) setup_time = time.time() - self._start_time
    [160](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/tune/trainable/trainable.py:160) if setup_time > SETUP_TIME_THRESHOLD:

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:640, in Algorithm.setup(self, config)
    [637](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:637)     self.config.off_policy_estimation_methods = ope_dict
    [639](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:639) # Create a set of env runner actors via a EnvRunnerGroup.
--> [640](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:640) self.workers = EnvRunnerGroup(
    [641](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:641)     env_creator=self.env_creator,
    [642](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:642)     validate_env=self.validate_env,
    [643](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:643)     default_policy_class=self.get_default_policy_class(self.config),
    [644](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:644)     config=self.config,
    [645](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:645)     num_env_runners=self.config.num_env_runners,
    [646](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:646)     local_env_runner=True,
    [647](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:647)     logdir=self.logdir,
    [648](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:648) )
    [650](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:650) # Ensure remote workers are initially in sync with the local worker.
    [651](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py:651) self.workers.sync_weights(inference_only=True)

File ~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:191, in EnvRunnerGroup.__init__(self, env_creator, validate_env, default_policy_class, config, num_env_runners, local_env_runner, logdir, _setup, num_workers, local_worker)
    [178](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:178) except RayActorError as e:
    [179](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:179)     # In case of an actor (remote worker) init failure, the remote worker
    [180](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:180)     # may still exist and will be accessible, however, e.g. calling
    [181](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:181)     # its `sample.remote()` would result in strange "property not found"
    [182](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:182)     # errors.
    [183](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:183)     if e.actor_init_failed:
    [184](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:184)         # Raise the original error here that the EnvRunners raised
    [185](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:185)         # during its construction process. This is to enforce transparency
   (...)
    [189](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:189)         # - e.args[0].args[2]: The original Exception (e.g. a ValueError due
    [190](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:190)         # to a config mismatch) thrown inside the actor.
--> [191](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:191)         raise e.args[0].args[2]
    [192](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:192)     # In any other case, raise the RayActorError as-is.
    [193](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:193)     else:
    [194](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/mantasu/projects/symrl/notebooks/~/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py:194)         raise e

RaySystemError: System error: No module named 'my_module'
traceback: Traceback (most recent call last):
  File "/home/mantasu/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/serialization.py", line 412, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mantasu/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/serialization.py", line 271, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mantasu/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/serialization.py", line 226, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mantasu/programs/anaconda/envs/sddl/lib/python3.11/site-packages/ray/_private/serialization.py", line 216, in _deserialize_pickle5_data
    obj = pickle.loads(in_band)
          ^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'my_module'

Issue Severity

Medium: It is a significant difficulty but I can work around it.

Jun 10 '24 22:06 mantasu

ray ray copied to clipboard

[RLlib] Custom RLModule import fails using absolute imports (with non-zero `env_runners`)

What happened + What you expected to happen

Versions / Dependencies

Reproduction script

Issue Severity

ray
ray copied to clipboard