deepracer-local icon indicating copy to clipboard operation
deepracer-local copied to clipboard

Cannot connect to Redis when using DEEP_CONVOLUTIONAL_NETWORK

Open tunguyenlam opened this issue 5 years ago • 1 comments
trafficstars

when I try using DEEP_CONVOLUTIONAL_NETWORK (before used DEEP_CONVOLUTIONAL_NETWORK_SHALLOW, It run smoothly).

It raises an error in evaluation step:

Training> Name=main_level/agent, Worker=0, Episode=20, Total reward=2.1, Steps=99, Training iteration=0
## agent: Starting evaluation phase
Testing> Name=main_level/agent, Worker=0, Episode=20, Total reward=1.5, Steps=99, Training iteration=0
## agent: Finished evaluation phase. Success rate = 0.0, Avg Total Reward = 1.5
## agent: Starting evaluation phase
Exception in thread Thread-435:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 185, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 3236, in run
    timeout=sleep_time)
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 3135, in get_message
    response = self.parse_response(block=False, timeout=timeout)
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 3036, in parse_response
    return self._execute(connection, connection.read_response)
  File "/usr/local/lib/python3.5/dist-packages/redis/client.py", line 3013, in _execute
    return command(*args)
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 637, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 290, in read_response
    response = self._buffer.readline()
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 224, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.5/dist-packages/redis/connection.py", line 199, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)

tunguyenlam avatar Apr 02 '20 16:04 tunguyenlam

I'm afraid I don't have enough memory on my GPU to test a 5 layer network.

What GPU are you running? Perhaps Redis is being killed due to oom?

mattcamp avatar Jun 03 '20 13:06 mattcamp