maddpg-rllib icon indicating copy to clipboard operation
maddpg-rllib copied to clipboard

fixes import from rllib

Open sanjeevanahilan opened this issue 5 years ago • 14 comments

maddpg is currently in rllib.contrib rather than rllib.agents

see https://github.com/ray-project/ray/issues/6476

sanjeevanahilan avatar Dec 13 '19 11:12 sanjeevanahilan

Hey, when I try running this PR to reproduce the results of the paper, I still get related issues, error log below. Given that NameError: Please run contrib/MADDPG instead is included three times, I assume it's related to this.

2019-12-14 17:57:42,143	INFO resource_spec.py:205 -- Starting Ray with 39.11 GiB memory available for workers and up to 2.79 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2019-12-14 17:57:42,516	WARNING sample.py:31 -- DeprecationWarning: wrapping <function main.<locals>.<lambda> at 0x7f115303cd90> with tune.function() is no longer needed
2019-12-14 17:57:42,599	WARNING worker.py:348 -- WARNING: Falling back to serializing objects of type <class 'numpy.dtype'> by using pickle. This may be inefficient.
2019-12-14 17:57:42,608	WARNING worker.py:348 -- WARNING: Falling back to serializing objects of type <class 'mtrand.RandomState'> by using pickle. This may be inefficient.
(pid=12579) 2019-12-14 17:57:44,655	INFO trainer.py:345 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2019-12-14 17:57:45,381	ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 515, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
    result = ray.get(trial_future[0])
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 2121, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AttributeError): ray_MADDPG:train() (pid=12579, host=Prophet)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
    Trainer.__init__(self, config, env, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 372, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 492, in _setup
    self._init(self.config, self.env_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 109, in _init
    self.config["num_workers"])
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 537, in _make_workers
    logdir=self.logdir)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 64, in __init__
    RolloutWorker, env_creator, policy, 0, self._local_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 220, in _make_worker
    _fake_sampler=config.get("_fake_sampler", False))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 348, in __init__
    self._build_policy_map(policy_dict, policy_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 764, in _build_policy_map
    policy_map[name] = cls(obs_space, act_space, merged_conf)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 158, in __init__
    scope="actor"))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 368, in _build_actor_network
    sampler = tfp.distributions.RelaxedOneHotCategorical(
AttributeError: 'NoneType' object has no attribute 'distributions'
2019-12-14 17:57:45,386	INFO trial_runner.py:615 -- Attempting to recover trial state from last checkpoint.
2019-12-14 17:57:46,464	ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 515, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
    result = ray.get(trial_future[0])
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 2121, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(NameError): ray__SeeContrib:train() (pid=12582, host=Prophet)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 372, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/__init__.py", line 58, in _setup
    "Please run `contrib/{}` instead.".format(name))
**NameError: Please run `contrib/MADDPG` instead.**
2019-12-14 17:57:46,468	INFO trial_runner.py:615 -- Attempting to recover trial state from last checkpoint.
(pid=12582) 2019-12-14 17:57:46,461	INFO trainer.py:345 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2019-12-14 17:57:47,576	ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 515, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
    result = ray.get(trial_future[0])
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 2121, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(NameError): ray__SeeContrib:train() (pid=12576, host=Prophet)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 372, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/__init__.py", line 58, in _setup
    "Please run `contrib/{}` instead.".format(name))
**NameError: Please run `contrib/MADDPG` instead.**
2019-12-14 17:57:47,579	INFO trial_runner.py:615 -- Attempting to recover trial state from last checkpoint.
(pid=12576) 2019-12-14 17:57:47,572	INFO trainer.py:345 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2019-12-14 17:57:48,644	ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 515, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
    result = ray.get(trial_future[0])
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 2121, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(NameError): ray__SeeContrib:train() (pid=12578, host=Prophet)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 372, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/__init__.py", line 58, in _setup
    "Please run `contrib/{}` instead.".format(name))
**NameError: Please run `contrib/MADDPG` instead.**
Traceback (most recent call last):
  File "run_maddpg.py", line 182, in <module>
    main(args)
  File "run_maddpg.py", line 177, in main
    }, verbose=0)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/tune.py", line 334, in run_experiments
    return_trials=True)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/tune.py", line 269, in run
    raise TuneError("Trials did not complete", errored_trials)
ray.tune.error.TuneError: ('Trials did not complete', [MADDPG_mpe_0])
(pid=12578) 2019-12-14 17:57:48,639	INFO trainer.py:345 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution

jkterry1 avatar Dec 14 '19 23:12 jkterry1

Ahh sorry, you also need to change:

run_experiments({
        "MADDPG_RLLib": {
            "run": "contrib/MADDPG",
            },

I've updated the PR but can't test this right now unfortunately, so let me know if there are still problems. I was trying to help out with this issue: https://github.com/ray-project/ray/issues/6476

sanjeevanahilan avatar Dec 14 '19 23:12 sanjeevanahilan

I got to this PR from having the same error as in that bug actually. Thanks the help with this too. There are still errors with that fix:

justinkterry@Prophet:~/Documents/maddpg-rllib$ python3 run_maddpg.py 
2019-12-14 18:33:21,718	INFO resource_spec.py:205 -- Starting Ray with 38.43 GiB memory available for workers and up to 2.79 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2019-12-14 18:33:22,053	WARNING sample.py:31 -- DeprecationWarning: wrapping <function main.<locals>.<lambda> at 0x7fdeb389ad90> with tune.function() is no longer needed
2019-12-14 18:33:22,127	WARNING worker.py:348 -- WARNING: Falling back to serializing objects of type <class 'numpy.dtype'> by using pickle. This may be inefficient.
2019-12-14 18:33:22,136	WARNING worker.py:348 -- WARNING: Falling back to serializing objects of type <class 'mtrand.RandomState'> by using pickle. This may be inefficient.
(pid=14615) 2019-12-14 18:33:24,168	INFO trainer.py:345 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2019-12-14 18:33:24,860	ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 515, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
    result = ray.get(trial_future[0])
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 2121, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AttributeError): ray_MADDPG:train() (pid=14615, host=Prophet)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
    Trainer.__init__(self, config, env, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 372, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 492, in _setup
    self._init(self.config, self.env_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 109, in _init
    self.config["num_workers"])
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 537, in _make_workers
    logdir=self.logdir)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 64, in __init__
    RolloutWorker, env_creator, policy, 0, self._local_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 220, in _make_worker
    _fake_sampler=config.get("_fake_sampler", False))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 348, in __init__
    self._build_policy_map(policy_dict, policy_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 764, in _build_policy_map
    policy_map[name] = cls(obs_space, act_space, merged_conf)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 158, in __init__
    scope="actor"))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 368, in _build_actor_network
    sampler = tfp.distributions.RelaxedOneHotCategorical(
AttributeError: 'NoneType' object has no attribute 'distributions'
2019-12-14 18:33:24,865	INFO trial_runner.py:615 -- Attempting to recover trial state from last checkpoint.
(pid=14604) 2019-12-14 18:33:25,922	INFO trainer.py:345 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2019-12-14 18:33:26,632	ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 515, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
    result = ray.get(trial_future[0])
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 2121, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AttributeError): ray_MADDPG:train() (pid=14604, host=Prophet)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
    Trainer.__init__(self, config, env, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 372, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 492, in _setup
    self._init(self.config, self.env_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 109, in _init
    self.config["num_workers"])
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 537, in _make_workers
    logdir=self.logdir)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 64, in __init__
    RolloutWorker, env_creator, policy, 0, self._local_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 220, in _make_worker
    _fake_sampler=config.get("_fake_sampler", False))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 348, in __init__
    self._build_policy_map(policy_dict, policy_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 764, in _build_policy_map
    policy_map[name] = cls(obs_space, act_space, merged_conf)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 158, in __init__
    scope="actor"))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 368, in _build_actor_network
    sampler = tfp.distributions.RelaxedOneHotCategorical(
AttributeError: 'NoneType' object has no attribute 'distributions'
2019-12-14 18:33:26,637	INFO trial_runner.py:615 -- Attempting to recover trial state from last checkpoint.
(pid=14605) 2019-12-14 18:33:27,692	INFO trainer.py:345 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2019-12-14 18:33:28,390	ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 515, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
    result = ray.get(trial_future[0])
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 2121, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AttributeError): ray_MADDPG:train() (pid=14605, host=Prophet)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
    Trainer.__init__(self, config, env, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 372, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 492, in _setup
    self._init(self.config, self.env_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 109, in _init
    self.config["num_workers"])
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 537, in _make_workers
    logdir=self.logdir)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 64, in __init__
    RolloutWorker, env_creator, policy, 0, self._local_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 220, in _make_worker
    _fake_sampler=config.get("_fake_sampler", False))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 348, in __init__
    self._build_policy_map(policy_dict, policy_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 764, in _build_policy_map
    policy_map[name] = cls(obs_space, act_space, merged_conf)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 158, in __init__
    scope="actor"))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 368, in _build_actor_network
    sampler = tfp.distributions.RelaxedOneHotCategorical(
AttributeError: 'NoneType' object has no attribute 'distributions'
2019-12-14 18:33:28,394	INFO trial_runner.py:615 -- Attempting to recover trial state from last checkpoint.
(pid=14603) 2019-12-14 18:33:29,456	INFO trainer.py:345 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2019-12-14 18:33:30,136	ERROR trial_runner.py:569 -- Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trial_runner.py", line 515, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/ray_trial_executor.py", line 351, in fetch_result
    result = ray.get(trial_future[0])
  File "/usr/local/lib/python3.7/dist-packages/ray/worker.py", line 2121, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(AttributeError): ray_MADDPG:train() (pid=14603, host=Prophet)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 90, in __init__
    Trainer.__init__(self, config, env, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 372, in __init__
    Trainable.__init__(self, config, logger_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/trainable.py", line 96, in __init__
    self._setup(copy.deepcopy(self.config))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 492, in _setup
    self._init(self.config, self.env_creator)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer_template.py", line 109, in _init
    self.config["num_workers"])
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/agents/trainer.py", line 537, in _make_workers
    logdir=self.logdir)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 64, in __init__
    RolloutWorker, env_creator, policy, 0, self._local_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/worker_set.py", line 220, in _make_worker
    _fake_sampler=config.get("_fake_sampler", False))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 348, in __init__
    self._build_policy_map(policy_dict, policy_config)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/evaluation/rollout_worker.py", line 764, in _build_policy_map
    policy_map[name] = cls(obs_space, act_space, merged_conf)
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 158, in __init__
    scope="actor"))
  File "/usr/local/lib/python3.7/dist-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 368, in _build_actor_network
    sampler = tfp.distributions.RelaxedOneHotCategorical(
AttributeError: 'NoneType' object has no attribute 'distributions'
Traceback (most recent call last):
  File "run_maddpg.py", line 182, in <module>
    main(args)
  File "run_maddpg.py", line 177, in main
    }, verbose=0)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/tune.py", line 334, in run_experiments
    return_trials=True)
  File "/usr/local/lib/python3.7/dist-packages/ray/tune/tune.py", line 269, in run
    raise TuneError("Trials did not complete", errored_trials)
ray.tune.error.TuneError: ('Trials did not complete', [contrib_MADDPG_mpe_0])

jkterry1 avatar Dec 14 '19 23:12 jkterry1

tfp is Tensorflow Probability which I think you just need to pip install.

https://www.tensorflow.org/probability

Hopefully that works. Also, if you are able to benchmark this code it would be great. This project is unrelated to me, but in my brief tests I experienced some training issues when using Tensoflow 2.0.

sanjeevanahilan avatar Dec 14 '19 23:12 sanjeevanahilan

Thank you. I'm using tensorflow v1.14 at the moment, because stable baselines is not compatible with tf2. I get the exact same error after installing tensorflow probability. Here's all my packages for reference:

justinkterry@Prophet:~/Documents/maddpg-rllib$ pip3 list
Package                 Version     Location                                     
----------------------- ----------- ---------------------------------------------
absl-py                 0.8.1       
altair                  3.2.0       
apturl                  0.5.2       
asn1crypto              0.24.0      
astor                   0.8.0       
atari-py                0.2.6       
attrs                   19.3.0      
backcall                0.1.0       
bleach                  3.1.0       
blinker                 1.4         
box2d-py                2.3.8       
Brlapi                  0.6.7       
certifi                 2019.9.11   
cffi                    1.12.3      
chardet                 3.0.4       
chrome-gnome-shell      0.0.0       
Click                   7.0         
cloudpickle             1.1.1       
colorama                0.4.3       
colour                  0.1.5       
command-not-found       0.3         
cryptography            2.3         
cupshelpers             1.0         
cycler                  0.10.0      
dash                    1.4.1       
dash-core-components    1.3.1       
dash-html-components    1.0.1       
dash-renderer           1.1.2       
dash-table              4.4.1       
decorator               4.4.0       
defer                   1.0.6       
defusedxml              0.6.0       
distro                  1.3.0       
distro-info             0.21ubuntu2 
entrypoints             0.3         
enum34                  1.1.6       
filelock                3.0.12      
flake8                  3.7.8       
Flask                   1.1.1       
Flask-Compress          1.4.0       
funcsigs                1.0.2       
future                  0.18.1      
gast                    0.2.2       
google-pasta            0.1.8       
GPy                     1.9.8       
grpcio                  1.24.3      
gym                     0.15.3      
h5py                    2.10.0      
httplib2                0.11.3      
idna                    2.8         
imageio                 2.5.0       
importlib-metadata      0.23        
ipykernel               5.1.2       
ipython                 7.8.0       
ipython-genutils        0.2.0       
ipywidgets              7.5.1       
itsdangerous            1.1.0       
jedi                    0.15.1      
Jinja2                  2.10.1      
joblib                  0.14.0      
jsonschema              3.1.1       
jupyter                 1.0.0       
jupyter-client          5.3.3       
jupyter-console         6.0.0       
jupyter-core            4.6.0       
Keras-Applications      1.0.8       
Keras-Preprocessing     1.1.0       
keyboard                0.13.4      
keyring                 17.1.1      
keyrings.alt            3.1.1       
kiwisolver              1.1.0       
language-selector       0.1         
launchpadlib            1.10.6      
lazr.restfulclient      0.14.2      
lazr.uri                1.0.3       
louis                   3.8.0       
lz4                     2.2.1       
macaroonbakery          1.2.1       
Markdown                3.1.1       
MarkupSafe              1.1.1       
matplotlib              3.1.1       
mccabe                  0.6.1       
mistune                 0.8.4       
mock                    3.0.5       
more-itertools          7.2.0       
mpi4py                  3.0.3       
multiagent              0.0.1       /home/justinkterry/Documents/maddpg-rllib/MPE
nbconvert               5.6.0       
nbformat                4.4.0       
netifaces               0.10.4      
networkx                2.3         
notebook                6.0.1       
numexpr                 2.7.0       
numpy                   1.16.2      
numpy-stl               2.10.1      
oauthlib                2.1.0       
olefile                 0.46        
opencv-python           4.1.1.26    
packaging               19.2        
pandas                  0.25.1      
pandocfilters           1.4.2       
paramz                  0.9.5       
parso                   0.5.1       
pexpect                 4.6.0       
pickleshare             0.7.5       
Pillow                  5.4.1       
Pillow-SIMD             6.0.0.post0 
pip                     18.1        
plotly                  4.2.0       
pluggy                  0.13.1      
prometheus-client       0.7.1       
prompt-toolkit          2.0.9       
protobuf                3.11.1      
psutil                  5.6.7       
psycopg2                2.8.3       
ptyprocess              0.6.0       
py                      1.8.0       
pycairo                 1.16.2      
pycodestyle             2.5.0       
pycparser               2.19        
pycrypto                2.6.1       
pycups                  1.9.73      
pyflakes                2.1.1       
pygame                  1.9.6       
pyglet                  1.3.2       
Pygments                2.4.2       
PyGObject               3.32.0      
PyJWT                   1.7.0       
pymacaroons             0.13.0      
pymunk                  5.5.0       
PyNaCl                  1.3.0       
pynput                  1.4.5       
pyparsing               2.4.2       
pyRFC3339               1.1         
pyrsistent              0.15.4      
pytest                  5.3.1       
python-apt              1.8.4       
python-dateutil         2.7.3       
python-debian           0.1.34      
python-utils            2.3.0       
python-xlib             0.25        
pytz                    2019.3      
PyWavelets              1.0.3       
pyxdg                   0.25        
PyYAML                  3.13        
pyzmq                   18.1.0      
qtconsole               4.5.5       
ray                     0.7.6       
redis                   3.3.11      
reportlab               3.5.18      
requests                2.22.0      
requests-unixsocket     0.1.5       
retrying                1.3.3       
scikit-image            0.15.0      
scikit-learn            0.21.3      
scipy                   1.2.1       
screen-resolution-extra 0.0.0       
seaborn                 0.9.0       
SecretStorage           2.3.1       
selenium                3.141.0     
Send2Trash              1.5.0       
setproctitle            1.1.10      
setuptools              41.4.0      
simplejson              3.16.0      
six                     1.12.0      
sklearn                 0.0         
stable-baselines        2.8.0       
system-service          0.3         
systemd-python          234         
tables                  3.5.2       
tensorboard             1.14.0      
tensorflow              1.14.0      
tensorflow-estimator    1.14.0      
tensorflow-probability  0.8.0       
termcolor               1.1.0       
terminado               0.8.2       
testpath                0.4.2       
toolz                   0.10.0      
torch                   1.2.0       
torchvision             0.4.0       
tornado                 6.0.3       
traitlets               4.3.3       
ubuntu-advantage-tools  19.2        
ubuntu-drivers-common   0.0.0       
ufw                     0.36        
unattended-upgrades     0.1         
urllib3                 1.25.6      
wadllib                 1.3.3       
wcwidth                 0.1.7       
webencodings            0.5.1       
Werkzeug                0.16.0      
wheel                   0.32.3      
widgetsnbextension      3.5.1       
wrapt                   1.11.2      
xkit                    0.0.0       
zipp                    0.6.0       

jkterry1 avatar Dec 14 '19 23:12 jkterry1

@justinkterry have you try to install tensorflow-probability==0.7.0 which tensorflow-probability version 0.7.0?

dragon28 avatar Dec 16 '19 16:12 dragon28

That fixed it. That needs to be included in the documentation somewhere.

On Mon, Dec 16, 2019 at 11:54 AM dragon28 [email protected] wrote:

@justinkterry https://github.com/justinkterry have you try to install tensorflow-probability==0.7.0 which tensorflow-probability version 0.7.0?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wsjeon/maddpg-rllib/pull/7?email_source=notifications&email_token=AEUF33AEJVYGXSHHAKBQX5TQY6XDHA5CNFSM4J2L2BUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEG7LMDA#issuecomment-566146572, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEUF33GKGMHA2ENZYRL7IV3QY6XDHANCNFSM4J2L2BUA .

-- Thank you for your time, Justin Terry

jkterry1 avatar Dec 29 '19 05:12 jkterry1

@sanjeevanahilan @wsjeon @justinkterry

I think I found some errors related to memory:

$ python3 -Wall ./run_maddpg.py --scenario simple_tag
/home/dragon/.local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/home/dragon/.local/lib/python3.6/site-packages/tensorflow/python/keras/backend.py:5506: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/dragon/.keras/keras.json' mode='r' encoding='UTF-8'>
  _config = json.load(open(_config_path))
2020-01-04 18:19:05,520	INFO resource_spec.py:216 -- Starting Ray with 1.17 GiB memory available for workers and up to 0.93 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:593: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/redis-shard_0.err' mode='a' encoding='UTF-8'>
  self.start_redis()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:593: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/redis-shard_0.out' mode='a' encoding='UTF-8'>
  self.start_redis()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:593: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/redis.err' mode='a' encoding='UTF-8'>
  self.start_redis()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:593: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/redis.out' mode='a' encoding='UTF-8'>
  self.start_redis()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:594: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/monitor.out' mode='a' encoding='UTF-8'>
  self.start_monitor()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:594: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/monitor.err' mode='a' encoding='UTF-8'>
  self.start_monitor()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:595: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/raylet_monitor.out' mode='a' encoding='UTF-8'>
  self.start_raylet_monitor()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:595: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/raylet_monitor.err' mode='a' encoding='UTF-8'>
  self.start_raylet_monitor()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:606: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/plasma_store.out' mode='a' encoding='UTF-8'>
  self.start_plasma_store()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:606: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/plasma_store.err' mode='a' encoding='UTF-8'>
  self.start_plasma_store()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:607: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/raylet.out' mode='a' encoding='UTF-8'>
  self.start_raylet()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:607: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/raylet.err' mode='a' encoding='UTF-8'>
  self.start_raylet()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:609: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/reporter.out' mode='a' encoding='UTF-8'>
  self.start_reporter()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:609: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/reporter.err' mode='a' encoding='UTF-8'>
  self.start_reporter()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:612: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/log_monitor.out' mode='a' encoding='UTF-8'>
  self.start_log_monitor()
/home/dragon/.local/lib/python3.6/site-packages/ray/node.py:612: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/ray/session_2020-01-04_18-19-05_509313_31535/logs/log_monitor.err' mode='a' encoding='UTF-8'>
  self.start_log_monitor()
2020-01-04 18:19:05,925	WARNING sample.py:31 -- DeprecationWarning: wrapping <function main.<locals>.<lambda> at 0x7f246a0c9950> with tune.function() is no longer needed
(pid=31584) 2020-01-04 18:19:08,573	INFO trainer.py:371 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
(pid=31584) 2020-01-04 18:19:08,575	INFO trainer.py:512 -- Current log_level is ERROR. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(pid=31584) 2020-01-04 18:19:13,723	WARNING util.py:45 -- Install gputil for GPU system monitoring.
(pid=31584) /home/dragon/.local/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.
(pid=31584)   out=out, **kwargs)
(pid=31584) /home/dragon/.local/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars
(pid=31584)   ret = ret.dtype.type(ret / rcount)
2020-01-04 19:22:31,599	ERROR trial_runner.py:476 -- Error processing event.
Traceback (most recent call last):
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 420, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 378, in fetch_result
    result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/worker.py", line 1456, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RayOutOfMemoryError): ray::MADDPG.train() (pid=31584, ip=192.168.0.190)
  File "python/ray/_raylet.pyx", line 636, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 619, in ray._raylet.execute_task.function_executor
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 444, in train
    raise e
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 433, in train
    result = Trainable.train(self)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/tune/trainable.py", line 176, in train
    result = self._train()
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 138, in _train
    res = collect_metrics_fn(self)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/rllib/contrib/maddpg/maddpg.py", line 170, in collect_metrics
    result = trainer.collect_metrics()
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 753, in collect_metrics
    selected_workers=selected_workers)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/rllib/optimizers/policy_optimizer.py", line 108, in collect_metrics
    timeout_seconds=timeout_seconds)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/rllib/evaluation/metrics.py", line 75, in collect_episodes
    metric_lists = ray_get_and_free(collected)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/rllib/utils/memory.py", line 33, in ray_get_and_free
    result = ray.get(object_ids)
ray.exceptions.RayTaskError(RayOutOfMemoryError): ray::RolloutWorker (pid=31583, ip=192.168.0.190)
  File "python/ray/_raylet.pyx", line 627, in ray._raylet.execute_task
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/memory_monitor.py", line 130, in raise_if_low_memory
    self.error_threshold))
ray.memory_monitor.RayOutOfMemoryError: More than 95% of the memory on node WorkStation is used (14.83 / 15.61 GB). The top 10 memory consumers are:

PID	MEM	COMMAND
31584	7.03GiB	ray::MADDPG.train()
5436	0.47GiB	/usr/lib/firefox/firefox -new-window
30138	0.33GiB	/usr/lib/x86_64-linux-gnu/opera/opera --type=renderer --field-trial-handle=3240866520022368210,60777
31583	0.33GiB	ray::RolloutWorker
3178	0.29GiB	/usr/bin/gnome-shell
25811	0.28GiB	/usr/bin/python3 -m pyls --host 127.0.0.1 --port 2087 --tcp --check-parent-process
5482	0.24GiB	/usr/lib/x86_64-linux-gnu/opera/opera --type=gpu-process --field-trial-handle=3240866520022368210,60
31535	0.24GiB	python3 -Wall ./run_maddpg.py --scenario simple_tag
25753	0.2GiB	/usr/bin/python3 /home/dragon/.local/bin/spyder3
5432	0.16GiB	/usr/lib/x86_64-linux-gnu/opera/opera --new-window

In addition, up to 1.43 GiB of shared memory is currently being used by the Ray object store. You can set the object store size with the `object_store_memory` parameter when starting Ray, and the max Redis size with `redis_max_memory`. Note that Ray assumes all system memory is available for use by workers. If your system has other applications running, you should manually set these memory limits to a lower value.
Traceback (most recent call last):
  File "./run_maddpg.py", line 187, in <module>
    main(args)
  File "./run_maddpg.py", line 182, in main
    }, verbose=0)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/tune/tune.py", line 408, in run_experiments
    return_trials=True)
  File "/home/dragon/.local/lib/python3.6/site-packages/ray/tune/tune.py", line 327, in run
    raise TuneError("Trials did not complete", errored_trials)
ray.tune.error.TuneError: ('Trials did not complete', [contrib_MADDPG_mpe_a339b59a])

OS: Ubuntu 18.04 64-bits

$ uname -a
Linux WorkStation 5.0.0-37-generic #40~18.04.1-Ubuntu SMP Thu Nov 14 12:06:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Python: Python version 3.6 (CPU only)

$ python3 --version
Python 3.6.9

Redis version 4.0.9

$ redis-server -v
Redis server v=4.0.9 sha=00000000:0 malloc=jemalloc-3.6.0 bits=64 build=9435c3c2879311f3

Tensorflow version 1.14.0

Tensorflow-estimator version 1.14.0

Tensorflow-metadata version 0.14.0

Tensorflow-probability version 0.7.0

RLLib version 0.8.0.dev7

Pip Details:

$ pip3 list
Package                       Version             Location                                   
----------------------------- ------------------- -------------------------------------------
absl-py                       0.7.1               
alabaster                     0.7.12              
apache-beam                   2.16.0              
appdirs                       1.4.3               
apturl                        0.5.2               
argh                          0.26.2              
asn1crypto                    0.24.0              
astor                         0.8.0               
astroid                       2.2.5               
asyncio                       3.4.3               
atari-py                      0.2.6               
atomicwrites                  1.3.0               
attrs                         19.1.0              
autopep8                      1.4.4               
avro-python3                  1.9.0               
Babel                         2.6.0               
backcall                      0.1.0               
backports.csv                 1.0.7               
baselines                     0.1.6               
beautifulsoup4                4.7.1               
bintrees                      2.0.7               
bleach                        3.1.0               
blis                          0.2.4               
boto3                         1.9.240             
botocore                      1.12.240            
Box2D                         2.3.2               
Box2D-kengz                   2.3.3               
Brlapi                        0.6.6               
bs4                           0.0.1               
cachetools                    3.1.1               
certifi                       2018.1.18           
cffi                          1.12.3              
chardet                       3.0.4               
Click                         7.0                 
cloudpickle                   1.2.2               
colorama                      0.4.1               
colour                        0.1.5               
command-not-found             0.3                 
contextlib2                   0.5.5               
crcmod                        1.7                 
cryptography                  2.1.4               
cssselect                     1.0.3               
cupshelpers                   1.0                 
cycler                        0.10.0              
cymem                         2.0.2               
Cython                        0.29.10             
data                          0.4                 
dataclasses                   0.7                 
decorator                     4.4.0               
defer                         1.0.6               
defusedxml                    0.6.0               
devscripts                    2.17.12ubuntu1.1    
diff-match-patch              20181111            
dill                          0.3.0               
diskcache                     3.1.1               
distro-info                   0.18ubuntu0.18.04.1 
dlib                          19.17.99            
dm-sonnet                     1.34                
docopt                        0.6.2               
docutils                      0.14                
dopamine-rl                   2.0.5               /home/dragon/machine_learning/dopamine     
easydict                      1.9                 
editdistance                  0.5.3               
entrypoints                   0.3                 
enum34                        1.1.6               
et-xmlfile                    1.0.1               
fake-useragent                0.1.11              
fastavro                      0.21.24             
fasteners                     0.15                
filelock                      3.0.12              
findspark                     1.3.0               
flake8                        3.7.9               
flatbuffers                   1.10                
funcsigs                      1.0.2               
future                        0.17.1              
gast                          0.2.2               
gin-config                    0.1.4               
gitdb2                        2.0.5               
GitPython                     2.1.11              
glfw                          1.8.1               
glob2                         0.6                 
gluonts                       0.3.3               
google-api-core               1.13.0              
google-api-python-client      1.7.11              
google-apitools               0.5.28              
google-auth                   1.6.3               
google-auth-httplib2          0.0.3               
google-auth-oauthlib          0.4.1               
google-cloud-bigquery         1.6.1               
google-cloud-bigtable         0.32.2              
google-cloud-core             0.29.1              
google-cloud-datastore        1.7.4               
google-cloud-pubsub           0.39.1              
google-pasta                  0.1.7               
google-resumable-media        0.3.2               
googleapis-common-protos      1.6.0               
gpg                           1.10.0              
graphviz                      0.8.4               
grpc-google-iam-v1            0.11.4              
grpcio                        1.20.1              
gym                           0.15.3              
gym-ctc-executioner           0.0.1               
gym-ctc-marketmaker           0.0.1               
gym-test                      0.0.1               /home/dragon/reinforcementlearning/gym-test
h5py                          2.9.0               
handyspark                    0.2.2a1             
hdfs                          2.5.8               
helpdev                       0.6.10              
holidays                      0.9.11              
httplib2                      0.9.2               
icc-rt                        2019.0              
idna                          2.6                 
imageio                       2.5.0               
imagesize                     1.1.0               
importlib-metadata            0.17                
imutils                       0.5.3               
intel-openmp                  2019.0              
intervaltree                  3.0.2               
ipykernel                     5.1.3               
ipython                       7.5.0               
ipython-genutils              0.2.0               
ipywidgets                    7.4.2               
isort                         4.3.20              
jdcal                         1.4.1               
jedi                          0.14.1              
Jinja2                        2.10.1              
jmespath                      0.9.4               
joblib                        0.13.2              
jsonschema                    3.0.1               
jupyter                       1.0.0               
jupyter-client                5.3.4               
jupyter-console               6.0.0               
jupyter-core                  4.6.1               
Keras                         2.3.1               
Keras-Applications            1.0.8               
Keras-Preprocessing           1.0.9               
keras-rl                      0.4.2               
keyring                       10.6.0              
keyrings.alt                  3.0                 
kiwisolver                    1.1.0               
language-selector             0.1                 
latex                         0.7.0               
launchpadlib                  1.10.6              
lazr.restfulclient            0.13.5              
lazr.uri                      1.0.3               
lazy-object-proxy             1.4.1               
lockfile                      0.12.2              
louis                         3.5.0               
lxml                          4.3.3               
lz4                           2.2.1               
macaroonbakery                1.1.3               
maddpg                        0.0.1               
Mako                          1.0.7               
Markdown                      3.1.1               
MarkupSafe                    1.0                 
matplotlib                    3.1.0               
mccabe                        0.6.1               
meld                          3.18.0              
mistune                       0.8.4               
mkl                           2019.0              
mkl-fft                       1.0.6               
mkl-random                    1.0.1.1             
mock                          2.0.0               
monotonic                     1.5                 
more-itertools                7.0.0               
mpi4py                        3.1.0a0             
mpmath                        1.1.0               
multiagent                    0.0.1               
murmurhash                    1.0.2               
mxnet                         1.4.1               
nbconvert                     5.5.0               
nbformat                      4.4.0               
neat-python                   0.92                
netifaces                     0.10.4              
networkx                      2.3                 
nltk                          3.4.5               
notebook                      5.7.8               
numexpr                       2.6.9               
numpy                         1.14.6              
numpy-stl                     2.10.1              
numpydoc                      0.9.1               
oauth                         1.0.1               
oauth2client                  3.0.0               
oauthlib                      3.1.0               
odfpy                         1.4.0               
olefile                       0.45.1              
opencv-contrib-python         4.1.1.26            
opencv-python                 4.1.1.26            
opencv-python-headless        4.1.2.30            
openpyxl                      2.4.11              
opt-einsum                    3.1.0               
ordered-set                   3.1.1               
packaging                     19.0                
pandas                        0.25.2              
pandocfilters                 1.4.2               
parse                         1.12.0              
parso                         0.5.1               
pathtools                     0.1.2               
pbr                           5.4.0               
pexpect                       4.7.0               
pickleshare                   0.7.5               
Pillow                        6.1.0               
pip                           19.3.1              
plac                          0.9.6               
plotly                        3.9.0               
pluggy                        0.12.0              
preshed                       2.0.1               
prometheus-client             0.6.0               
prompt-toolkit                2.0.9               
protobuf                      3.11.0              
psutil                        5.6.2               
ptyprocess                    0.6.0               
py                            1.8.0               
py-spy                        0.3.1               
py4j                          0.10.7              
pyaes                         1.6.1               
pyarrow                       0.14.1              
pyasn1                        0.4.5               
pyasn1-modules                0.2.5               
pycairo                       1.16.2              
pyclipper                     1.1.0.post1         
pycodestyle                   2.5.0               
pycparser                     2.19                
pycrypto                      2.6.1               
pycups                        1.9.73              
pydantic                      0.28                
pydocstyle                    5.0.1               
pydot                         1.2.4               
pydotplus                     2.0.2               
PyDrive                       1.3.1               
pyee                          6.0.0               
pyflakes                      2.1.1               
pygame                        1.9.6               
pyglet                        1.3.2               
Pygments                      2.4.0               
pygobject                     3.26.1              
PyLaTeX                       1.3.1               
pylint                        2.3.1               
pymacaroons                   0.13.0              
pymongo                       3.9.0               
Pympler                       0.8                 
PyNaCl                        1.1.2               
PyOpenGL                      3.1.0               
PyOpenGL-accelerate           3.1.0               
pyparsing                     2.4.0               
pyppeteer                     0.0.25              
pyprob                        0.13.0              
PyQt5                         5.12.2              
PyQt5-sip                     4.19.17             
PyQtWebEngine                 5.12.1              
pyquery                       1.4.0               
pyRFC3339                     1.0                 
pyrsistent                    0.15.2              
pyspark                       2.4.3               
pytesseract                   0.3.0               
pytest                        4.6.0               
pytexit                       0.3.3               
python-apt                    1.6.4               
python-dateutil               2.8.0               
python-debian                 0.1.32              
python-jsonrpc-server         0.3.2               
python-language-server        0.31.2              
python-magic                  0.4.16              
python-telegram               0.10.0              
python-utils                  2.3.0               
pytz                          2018.3              
PyWavelets                    1.0.3               
pyxdg                         0.26                
PyYAML                        3.12                
pyzmq                         18.0.1              
QDarkStyle                    2.7                 
QtAwesome                     0.5.7               
qtconsole                     4.6.0               
QtPy                          1.7.1               
ray                           0.8.0.dev7          
records                       0.5.3               
redis                         3.3.11              
regex                         2019.8.19           
reportlab                     3.4.0               
requests                      2.22.0              
requests-html                 0.10.0              
requests-oauthlib             1.3.0               
requests-unixsocket           0.1.5               
retrying                      1.3.3               
rl                            3.0                 
rope                          0.14.0              
rsa                           4.0                 
s3transfer                    0.2.1               
sacremoses                    0.0.35              
scikit-image                  0.15.0              
scikit-learn                  0.21.1              
scipy                         1.3.0               
scons                         3.0.5               
screen-resolution-extra       0.0.0               
seaborn                       0.9.0               
SecretStorage                 2.3.1               
selenium                      3.141.0             
semantic-version              2.6.0               
Send2Trash                    1.5.0               
sentencepiece                 0.1.83              
setproctitle                  1.1.10              
setuptools                    41.0.1              
shutilwhich                   1.1.0               
simplejson                    3.13.2              
six                           1.13.0              
smmap2                        2.0.5               
snowballstemmer               1.2.1               
sortedcontainers              2.1.0               
soupsieve                     1.9.1               
spacy                         2.1.4               
Sphinx                        2.0.1               
sphinxcontrib-applehelp       1.0.1               
sphinxcontrib-devhelp         1.0.1               
sphinxcontrib-htmlhelp        1.0.2               
sphinxcontrib-jsmath          1.0.1               
sphinxcontrib-qthelp          1.0.2               
sphinxcontrib-serializinghtml 1.1.3               
spyder                        4.0.0               
spyder-kernels                1.8.1               
SQLAlchemy                    1.3.8               
srsly                         0.0.7               
stable-baselines              2.8.0               
sympy                         1.4                 
system-service                0.3                 
systemd-python                234                 
tables                        3.5.2               
tablib                        0.13.0              
tabulate                      0.8.3               
tbb                           2019.0              
tbb4py                        2019.0              
Telethon                      1.10.6              
tempdir                       0.7.1               
tensorboard                   1.14.0              
tensorflow                    1.14.0              
tensorflow-estimator          1.14.0              
tensorflow-metadata           0.14.0              
tensorflow-probability        0.7.0               
tensorflow-serving-api        1.14.0              
tensorflow-transform          0.14.0              
termcolor                     1.1.0               
terminado                     0.8.2               
testpath                      0.4.2               
testresources                 2.0.1               
tf-agents                     0.2.0rc2            
tfx-bsl                       0.15.0              
thinc                         7.0.4               
torch                         1.2.0+cpu           
torchvision                   0.4.0+cpu           
tornado                       6.0.2               
tqdm                          4.32.1              
trading-env                   0.0.1.dev0          
traitlets                     4.3.2               
transformers                  2.1.0               
trfl                          1.0.1               
typed-ast                     1.3.5               
ubuntu-drivers-common         0.0.0               
ufw                           0.36                
ujson                         1.35                
unattended-upgrades           0.1                 
unicodecsv                    0.14.1              
unidiff                       0.5.4               
uritemplate                   3.0.0               
urllib3                       1.22                
usb-creator                   0.3.3               
virtualenv                    16.6.0              
w3lib                         1.20.0              
wadllib                       1.3.2               
wasabi                        0.2.2               
watchdog                      0.9.0               
wcwidth                       0.1.7               
webencodings                  0.5.1               
websockets                    7.0                 
Werkzeug                      0.15.4              
wget                          3.2                 
wheel                         0.30.0              
widgetsnbextension            3.4.2               
wrapt                         1.11.1              
wurlitzer                     2.0.0               
xkit                          0.0.0               
xlrd                          1.2.0               
xlwt                          1.3.0               
yapf                          0.29.0              
zipp                          0.5.1               
zmq                           0.0.0               
zope.interface                4.3.2 

On the other hand, it also unable to execute simple_world_comm scenario

 python3 -Wall ./run_maddpg.py --scenario simple_world_comm
/home/dragon/.local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/home/dragon/.local/lib/python3.6/site-packages/tensorflow/python/keras/backend.py:5506: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/dragon/.keras/keras.json' mode='r' encoding='UTF-8'>
  _config = json.load(open(_config_path))
usage: MADDPG with OpenAI MPE [-h]
                              [--scenario {simple,simple_speaker_listener,simple_crypto,simple_push,simple_tag,simple_spread,simple_adversary}]
                              [--max-episode-len MAX_EPISODE_LEN]
                              [--num-episodes NUM_EPISODES]
                              [--num-adversaries NUM_ADVERSARIES]
                              [--good-policy GOOD_POLICY]
                              [--adv-policy ADV_POLICY] [--lr LR]
                              [--gamma GAMMA]
                              [--sample-batch-size SAMPLE_BATCH_SIZE]
                              [--train-batch-size TRAIN_BATCH_SIZE]
                              [--n-step N_STEP] [--num-units NUM_UNITS]
                              [--checkpoint-freq CHECKPOINT_FREQ]
                              [--local-dir LOCAL_DIR] [--restore RESTORE]
                              [--num-workers NUM_WORKERS]
                              [--num-envs-per-worker NUM_ENVS_PER_WORKER]
                              [--num-gpus NUM_GPUS]
MADDPG with OpenAI MPE: error: argument --scenario: invalid choice: 'simple_world_comm' (choose from 'simple', 'simple_speaker_listener', 'simple_crypto', 'simple_push', 'simple_tag', 'simple_spread', 'simple_adversary')

dragon28 avatar Jan 04 '20 14:01 dragon28

@dragon28 the real world com scenario isn't one of the currently supported MPE games (read ~line 34 of run_maddpg.py where it parses the --scenario arg).

In terms of the memory error, it's an issue with ray configuration instead of this code. You have to edit the config for how much ram workers, etc. can use in ray (google it). Ray infers defaults based on the system, so my 64GB RAM workstation can run it fine without changes, but my 16GB RAM laptop can't.

jkterry1 avatar Jan 04 '20 22:01 jkterry1

This default config on my workstation works, for reference:

2020-01-04 17:34:56,871 INFO resource_spec.py:216 -- Starting Ray with 39.06 GiB memory available for workers and up to 2.79 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).

jkterry1 avatar Jan 04 '20 22:01 jkterry1

@dragon28

Hey, I'm having the exact same memory issue as you very reproducibly. There appears to be a leak in the number of workers? I'll update this when I figured out what's going on more exactly.

jkterry1 avatar Jan 15 '20 03:01 jkterry1

Hey all, this repo isn't being maintained, so I forked it, made these fixes, and the maintainers of RLlib/ray made it the new official example repo of the maddpg example code. See https://github.com/ray-project/ray/pull/6831 and https://github.com/justinkterry/maddpg-rllib.

jkterry1 avatar Jan 18 '20 21:01 jkterry1

hey, @justinkterry I got the same error as in that bug actually. Thanks the help with this too. There are still errors with that fix:

2020-06-04 15:16:12,567 ERROR trial_runner.py:519 -- Trial MADDPG_mpe_420fc_00000: Error processing event. Traceback (most recent call last): File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 467, in _process_trial result = self.trial_executor.fetch_result(trial) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/worker.py", line 1474, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(AttributeError): ray::MADDPG.train() (pid=148180, ip=192.168.1.106) File "python/ray/_raylet.pyx", line 407, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 442, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 445, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 446, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 400, in ray._raylet.execute_task.function_executor File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 90, in init Trainer.init(self, config, env, logger_creator) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 452, in init super().init(config, logger_creator) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/trainable.py", line 174, in init self._setup(copy.deepcopy(self.config)) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 627, in _setup self._init(self.config, self.env_creator) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 115, in _init self.config["num_workers"]) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 700, in _make_workers logdir=self.logdir) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py", line 59, in init RolloutWorker, env_creator, policy, 0, self._local_config) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py", line 282, in _make_worker extra_python_environs=extra_python_environs) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 378, in init self._build_policy_map(policy_dict, policy_config) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 930, in _build_policy_map policy_map[name] = cls(obs_space, act_space, merged_conf) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 155, in init scope="actor")) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 371, in _build_actor_network sampler = tfp.distributions.RelaxedOneHotCategorical( AttributeError: 'NoneType' object has no attribute 'distributions' Traceback (most recent call last): File "run_maddpg.py", line 183, in main(args) File "run_maddpg.py", line 178, in main }, verbose=0) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/tune.py", line 411, in run_experiments return_trials=True) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/tune.py", line 347, in run raise TuneError("Trials did not complete", incomplete_trials) ray.tune.error.TuneError: ('Trials did not complete', [MADDPG_mpe_420fc_00000]) (tf) aics@aics-Super-Server:~/RayRLlib/maddpg-rllib$

aiguoth avatar Jun 04 '20 07:06 aiguoth

You have to use my fork

On Thu, Jun 4, 2020 at 3:17 AM Guoth [email protected] wrote:

hey, @justinkterry https://github.com/justinkterry I got the same error as in that bug actually. Thanks the help with this too. There are still errors with that fix:

2020-06-04 15:16:12,567 ERROR trial_runner.py:519 -- Trial MADDPG_mpe_420fc_00000: Error processing event. Traceback (most recent call last): File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 467, in _process_trial result = self.trial_executor.fetch_result(trial) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/worker.py", line 1474, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(AttributeError): ray::MADDPG.train() (pid=148180, ip=192.168.1.106) File "python/ray/_raylet.pyx", line 407, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 442, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 445, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 446, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 400, in ray._raylet.execute_task.function_executor File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 90, in init Trainer.init(self, config, env, logger_creator) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 452, in init super().init(config, logger_creator) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/trainable.py", line 174, in init self._setup(copy.deepcopy(self.config)) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 627, in _setup self._init(self.config, self.env_creator) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer_template.py", line 115, in _init self.config["num_workers"]) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/agents/trainer.py", line 700, in _make_workers logdir=self.logdir) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py", line 59, in init RolloutWorker, env_creator, policy, 0, self._local_config) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/evaluation/worker_set.py", line 282, in _make_worker extra_python_environs=extra_python_environs) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 378, in init self._build_policy_map(policy_dict, policy_config) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/evaluation/rollout_worker.py", line 930, in _build_policy_map policy_map[name] = cls(obs_space, act_space, merged_conf) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 155, in init scope="actor")) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/rllib/contrib/maddpg/maddpg_policy.py", line 371, in _build_actor_network sampler = tfp.distributions.RelaxedOneHotCategorical( AttributeError: 'NoneType' object has no attribute 'distributions' Traceback (most recent call last): File "run_maddpg.py", line 183, in main(args) File "run_maddpg.py", line 178, in main }, verbose=0) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/tune.py", line 411, in run_experiments return_trials=True) File "/home/aics/anaconda3/envs/tf/lib/python3.6/site-packages/ray/tune/tune.py", line 347, in run raise TuneError("Trials did not complete", incomplete_trials) ray.tune.error.TuneError: ('Trials did not complete', [MADDPG_mpe_420fc_00000]) (tf) aics@aics-Super-Server:~/RayRLlib/maddpg-rllib$

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wsjeon/maddpg-rllib/pull/7#issuecomment-638653788, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEUF33GDHJAFSUGNECOIOM3RU5DBPANCNFSM4J2L2BUA .

-- Thank you for your time, Justin Terry

jkterry1 avatar Jun 13 '20 23:06 jkterry1