SQL algorithm is not working

[ivan-ji-walmart]

Seems there are some error with SQL algorithm. I can run SAC but not SQL

ERROR trial_runner.py:567 -- Trial id=a3651_00000-seed=1911: Error processing event. Traceback (most recent call last): File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 515, in _process_trial result = self.trial_executor.fetch_result(trial) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 488, in fetch_result result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT) File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ray/worker.py", line 1428, in get raise value.as_instanceof_cause() ray.exceptions.RayTaskError(TypeError): ray::ExperimentRunner.train() (pid=24394, ip=10.22.134.202) File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task File "python/ray/_raylet.pyx", line 438, in ray._raylet.execute_task.function_executor File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ray/tune/trainable.py", line 336, in train result = self.step() File "/home/app/softlearning/examples/development/main.py", line 101, in step diagnostics = next(self.train_generator) File "/home/app/softlearning/softlearning/algorithms/rl_algorithm.py", line 182, in _train timestep=self._total_timestep) File "/home/app/softlearning/softlearning/algorithms/rl_algorithm.py", line 346, in _do_training_repeats for i in range(self._n_train_repeat) File "/home/app/softlearning/softlearning/algorithms/rl_algorithm.py", line 346, in for i in range(self._n_train_repeat) File "/home/app/softlearning/softlearning/algorithms/sql.py", line 356, in _do_training training_diagnostics = self._do_updates(batch) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in call result = self._call(*args, **kwds) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call self._initialize(args, kwds, add_initializers_to=initializers) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize *args, **kwds)) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function capture_by_value=self._capture_by_value), File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn return weak_wrapped_fn().wrapped(*args, **kwds) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3735, in bound_method_wrapper return wrapped_fn(*args, **kwargs) File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper raise e.ag_error_metadata.to_exception(e) TypeError: in user code:

/home/app/softlearning/softlearning/algorithms/sql.py:338 _do_updates  *
    Qs_values, Qs_losses = self._update_Q(
/home/app/softlearning/softlearning/algorithms/sql.py:205 _update_Q  *
    Q_observations = {

TypeError: 'NoneType' object is not iterable

[hartikainen]

Hey @ivan-ji-walmart, thanks for opening this issue! Sorry about this. Some of my recent changes must have broken the SQL implementation and has left unnoticed since I haven't been using SQL in a long time. I'll try to get this fixed in the upcoming weeks, but in the meanwhile, I'd highly recommend using the official reference implementation found at https://github.com/haarnoja/softqlearning.