[Feature Request]support with offline dataset (D4RL)

[louieworth]

Could you add a feature that supports offline RL research?

eg.

import gym
env_name = 'hopper-medium-v2'
env = gym.make(env_name)

references: d4rl