Goal

Improve the interactions between ReinforcementLearning.jl and other ecosystems in Julia.

Why is it important?

In the early days of developing this package, the main goal is to reproduce some popular (deep) RL algorithms. It's still important to keep adding new emerging algorithms into this package. But as an engineer, I always think the higher impact is achieved only when users really apply those powerful RL algorithms to the problems they are interested in. In recent years, many important packages across different domains were developed in Julia and the whole ecosystem improved a lot. Although the interfaces defined in this package are loose and flexible, people are still unsure how to use this package due to lacking concrete examples. Adding more examples and removing some restricted assumptions will greatly encourage more people to try this package. On the other hand, doing so will also improve the quality of this package.

Potential breaking changes

The most important change would be decoupling the training data generation and policy optimization. The state is assumed to be a tensor by default in many cases. This is the main blocking issue when interacting with many other packages. Besides, the async training pipeline will not only improve the performance of existing algorithms on a single node but also provide the foundation of large scale training in future releases (possibly in v0.12)

Key issues to be addressed

Following are some of the existing issues on the top of my mind. Please raise new ones if you wish to be addressed in the next release.

Environments

• [x] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/268
• [x] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/572 ~~Stil no luck to address this issue, so I have to remove OpenSpiel related part in the next release.~~ https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/691
• [ ] Demonstrate that environments can be implemented on not only CPU but also GPU https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/121
• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/546
• [ ] Take a look at https://github.com/dojo-sim/Dojo.jl
• [ ] Take a look at https://github.com/Lyceum/MuJoCo.jl
• [ ] More examples with DifferentialEquations.jl
• [ ] https://github.com/deepmind/android_env Should be OK through PyCall.jl. Need some examples.
• [ ] Look into https://github.com/corail-research/SeaPearl.jl and see if there could be some improvements.

Refactor Existing Policies

• [x] BasicDQN
• [ ] JuliaRL_BC_CartPole
• [x] JuliaRL_DQN_CartPole https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/650
• [x] JuliaRL_PrioritizedDQN_CartPole #698
• [x] JuliaRL_Rainbow_CartPole #724
• [x] JuliaRL_QRDQN_CartPole #699
• [x] JuliaRL_REMDQN_CartPole #708
• [x] JuliaRL_IQN_CartPole #710
• [ ] JuliaRL_VMPO_CartPole
• [x] JuliaRL_VPG_CartPole #733
• [ ] JuliaRL_BasicDQN_MountainCar
• [ ] JuliaRL_DQN_MountainCar
• [ ] JuliaRL_A2C_CartPole
• [ ] JuliaRL_A2CGAE_CartPole
• [ ] JuliaRL_PPO_CartPole
• [ ] JuliaRL_MAC_CartPole
• [ ] JuliaRL_DDPG_Pendulum
• [ ] JuliaRL_SAC_Pendulum
• [ ] JuliaRL_TD3_Pendulum
• [ ] JuliaRL_PPO_Pendulum
• [ ] JuliaRL_BasicDQN_SingleRoomUndirected

Add New Policies

• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/608

• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/326

• [ ] Improve the code structure and docs on general utils when defining a network https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/139

• [ ] Add alternatives to Flux.jl https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/136

• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/193

• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/206

• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/250

• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/262 and https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/531

• [ ] Revisit https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/347

• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/578

• [ ] Integrate CFR related algos at https://github.com/WhiffleFish/CounterfactualRegret.jl ?

• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/392 Borrow some ideas from https://github.com/facebookresearch/salina

Training pipeline

• [ ] Support async training https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/142
• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/144
• [ ] non-episodic environments https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/613 changes to policies may also be needed.

Documentation

• [x] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/582
• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/580
• [ ] Explain MultiThreadEnv in detail. https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/596

Utils

• [ ] Visualization. Leverage Term.jl based on the suggestion here
• [ ] https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/232
• [ ] Observation. Integrate OpenTelemetry.jl to provide a more unified approach for recording.
• [ ] Setup CI to generate the Docker image

Timeline

I'm not sure I can fix them all. But at least I'll take a deep look into them and then tag a new release at the end of this quarter (around the end of June 2022).

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/61734

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v0.10.1 -m "<description of version>" 2e1de3e5b6b8224f50b3d11bba7e1d2d72c6ef7c
git push origin v0.10.1

@JuliaRegistrator register subdir=src/ReinforcementLearningBase

Registration pull request created: JuliaRegistries/General/63479

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningBase-v0.10.0 -m "<description of version>" b58c7c4e7c7ecb141df135023dd284cab487c6e0
git push origin ReinforcementLearningBase-v0.10.0

@JuliaRegistrator register subdir=src/ReinforcementLearningEnvironments

Registration pull request created: JuliaRegistries/General/63531

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningEnvironments-v0.7.0 -m "<description of version>" 06aa63e06f255654e5ae967ec4737cafd4d77908
git push origin ReinforcementLearningEnvironments-v0.7.0

@JuliaRegistrator register subdir=src/ReinforcementLearningCore

Registration pull request updated: JuliaRegistries/General/63532

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningCore-v0.9.0 -m "<description of version>" 0f00ba42a00aacbf7cd8a31ab6b6e3c4cf14c1ef
git push origin ReinforcementLearningCore-v0.9.0

@JuliaRegistrator register subdir=src/ReinforcementLearningBase

Registration pull request created: JuliaRegistries/General/78892

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningBase-v0.11.0 -m "<description of version>" 107ceec40976d8023bc2b17a82861e4a240b7dee
git push origin ReinforcementLearningBase-v0.11.0

@JuliaRegistrator register subdir=src/ReinforcementLearningBase

Registration pull request updated: JuliaRegistries/General/78892

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningBase-v0.11.0 -m "<description of version>" b38b9f30c5750bc0d522f423d910750efb6f507b
git push origin ReinforcementLearningBase-v0.11.0

@JuliaRegistrator register subdir=src/ReinforcementLearningCore

Registration pull request created: JuliaRegistries/General/78898

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningCore-v0.9.1 -m "<description of version>" a6216b0b6e21fda86b72611228698455ce515b22
git push origin ReinforcementLearningCore-v0.9.1

@JuliaRegistrator register subdir=src/ReinforcementLearningCore

Registration pull request updated: JuliaRegistries/General/78898

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningCore-v0.9.1 -m "<description of version>" d49bdd226ccc7e89ec0ebf8eaffee245a5e94346
git push origin ReinforcementLearningCore-v0.9.1

@JuliaRegistrator register subdir=src/ReinforcementLearningEnvironments

Registration pull request created: JuliaRegistries/General/78919

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningEnvironments-v0.7.1 -m "<description of version>" bbce4f10c1863b3cae4a98f0a0b1595c34606342
git push origin ReinforcementLearningEnvironments-v0.7.1

@JuliaRegistrator register subdir=src/ReinforcementLearningEnvironments

Registration pull request updated: JuliaRegistries/General/78919

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningEnvironments-v0.7.1 -m "<description of version>" 9b1c2737044736d642d54b10917b038cd5f0bd73
git push origin ReinforcementLearningEnvironments-v0.7.1

@JuliaRegistrator register subdir=src/ReinforcementLearningExperiments

Error while trying to register: Version 0.1.4 already exists

@JuliaRegistrator register subdir=src/ReinforcementLearningZoo

Registration pull request created: JuliaRegistries/General/78985

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningZoo-v0.6.0 -m "<description of version>" 589fa9efc8130372bbde0f2733dbaf3ab68f15e0
git push origin ReinforcementLearningZoo-v0.6.0

@JuliaRegistrator register subdir=src/ReinforcementLearningExperiments

Registration pull request created: JuliaRegistries/General/78987

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningExperiments-v0.2.0 -m "<description of version>" d40dde1d4804965265eb9eeabab2fec500f2f490
git push origin ReinforcementLearningExperiments-v0.2.0

@JuliaRegistrator register subdir=src/ReinforcementLearningExperiments

Registration pull request updated: JuliaRegistries/General/78987

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a ReinforcementLearningExperiments-v0.2.0 -m "<description of version>" 25d9349c9fe20956419ea6341baf01a7237361bf
git push origin ReinforcementLearningExperiments-v0.2.0

@JuliaRegistrator register subdir=src/ReinforcementLearningCore branch=patch-v10.1.1

Error while trying to register: Version 0.8.12 already exists