Why does the training always converge at Avg Length of 4.0?

[Cheng-tf]

After ruling out issues caused by the PyTorch version, I successfully ran the command python a3c_main.py --num-processes 6 --evaluate 0. However, after running for a while, the Best reward and Avg Ep length remain unchanged, and I am unsure of the reason.

Could you kindly help me understand what might be causing this issue?

Time 19h 23m 13s, Avg Reward -0.023000000000000024, Avg Accuracy 0.16, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 24m 28s, Avg Reward -0.023000000000000024, Avg Accuracy 0.16, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 25m 42s, Avg Reward 0.048999999999999974, Avg Accuracy 0.22, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 27m 01s, Avg Reward 0.12099999999999998, Avg Accuracy 0.28, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 28m 21s, Avg Reward -0.023000000000000024, Avg Accuracy 0.16, Avg Ep length 4.0, Best Reward 0.31299999999999994 Training thread: 4 Num iters: 124K Avg policy loss: -0.03016290833395694 Avg value loss: 0.5059514494419204 Time 19h 29m 37s, Avg Reward -0.023000000000000024, Avg Accuracy 0.16, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 31m 05s, Avg Reward -0.07100000000000001, Avg Accuracy 0.12, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 32m 24s, Avg Reward 0.16899999999999998, Avg Accuracy 0.32, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 33m 43s, Avg Reward 0.024999999999999974, Avg Accuracy 0.2, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 35m 09s, Avg Reward 0.024999999999999974, Avg Accuracy 0.2, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 36m 43s, Avg Reward -0.07100000000000002, Avg Accuracy 0.12, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 38m 01s, Avg Reward 0.0009999999999999775, Avg Accuracy 0.18, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 39m 23s, Avg Reward -0.09500000000000001, Avg Accuracy 0.1, Avg Ep length 4.0, Best Reward 0.31299999999999994 Time 19h 40m 39s, Avg Reward 0.07299999999999998, Avg Accuracy 0.24, Avg Ep length 4.0, Best Reward 0.31299999999999994