Development | Example driven approach

[zsunberg]

A couple things I've learned when developing packages for research:

1. It helps to have many people to push back on ideas (we seem to be doing good here :smile: )
2. It is impossible to successfully develop an interface like this without concrete examples (and it's much better to have people actively using it)

So, in light of 2, I think we should try to add to the interface primarily as we come upon concrete examples of things we need. I have created the usage-example tag for this purpose. Please add your desired use examples and associated needs so that we can discuss them and add the functionality.

Also, I have fortunately had some extra time lately to help get this package started, but unfortunately that will usually not be the case. I'm hoping that we can keep the package simple and stable enough* that we can all weigh in on decisions, but, for it to be successful, a few people will have to take it and other associated packages under their wing and give it some loving care :smile: , so don't be afraid to take on some part that you want to improve and make it yours!

*Note: In my mind, keeping the package simple means limiting the number and complexity of concepts you have to understand; it doesn't necessarily mean keeping the number of interface functions very small. We should not be too afraid of adding a new interface function if it is needed and does not conflict with the rest of the interface.

[findmyway]

A couple things I've learned when developing packages for research:

1. It helps to have many people to push back on ideas (we seem to be doing good here 😄 )
2. It is impossible to successfully develop an interface like this without concrete examples (and it's much better to have people actively using it)

That's so true.

I'll add some experimental support at https://github.com/JuliaReinforcementLearning/ReinforcementLearningEnvironments.jl/issues/70 so most existing environments should work with just a convert function. And because API in ReinforcementLearningBase.jl is not re-exported in ReinforcementLearningEnvironments.jl, there won't be API conflicts.

[darsnack]

Also, I have fortunately had some extra time lately to help get this package started, but unfortunately that will usually not be the case. I'm hoping that we can keep the package simple and stable enough* that we can all weigh in on decisions, but, for it to be successful, a few people will have to take it and other associated packages under their wing and give it some loving care 😄 , so don't be afraid to take on some part that you want to improve and make it yours!

Unfortunately, my time is also limited to nights and weekends this summer for open-source, and currently I am coordinating some other Julia projects. I want to improve FluxML/Gym.jl. I am going try to get in touch with the maintainers and see if I can turn it into a usage example.

[jbrea]

I want to improve FluxML/Gym.jl

We also have a gym wrapper here. Is there something particular you would like to improve?

Edit: ah, I guess you're not looking for a wrapper, but some envs written in julia. We have some here.

[darsnack]

I haven't worked on that package for some time, but when I was working on it, the point was

• be pure Julia (I don't think JuliaReinforcementLearning supported anything other than a Gym wrapper at the time)
• to support a headless render (e.g. manipulate and return the RGB array for cart pole as an observation)

EDIT: Though at this point, it may not be worth updating that repo

[jbrea]

to support a headless render (e.g. manipulate and return the RGB array for cart pole as an observation)

Ah, that's cool. We have rendering (using GR) for some envs (e.g. cartpole, but the code is a bit ugly (I am to blame :smile:) and there is no option yet to just return the RGB array...

[zsunberg]

I'll add some experimental support at JuliaReinforcementLearning/ReinforcementLearningEnvironments.jl#70 so most existing environments should work with just a convert function.

@findmyway That would be really great! What would actually be more useful for me (in teaching) is the other direction: making all of the algorithms that you guys have developed work with environments implemented using this interface.

[findmyway]

I'll add some experimental support at JuliaReinforcementLearning/ReinforcementLearningEnvironments.jl#70 so most existing environments should work with just a convert function.

@findmyway That would be really great! What would actually be more useful for me (in teaching) is the other direction: making all of the algorithms that you guys have developed work with environments implemented using this interface.

That's definitely not a big problem. For those in https://github.com/JuliaReinforcementLearning/ReinforcementLearningZoo.jl They are not tight with any specific environments. The only work is to extend the run functions here: https://github.com/JuliaReinforcementLearning/ReinforcementLearningCore.jl/blob/master/src/core/run.jl.