Move Mesa examples to another repo with sub classing to this repo?

[jackiekazil]

Move Mesa examples to another repo with sub classing to this repo? Let's explore this more and figure out if this is something pattern that will work with subclassing git repo inside a git repo.

Things to consider

• How pushing to pypi works
• Make sure if someone changes any of the core that the examples still run
  • The tests are based on some of our examples

[dmasad]

I like the way tornado does it, where the examples are part of the repo but not the manifest that gets packaged up for pypi.

[jackiekazil]

Here is what I was talking about: https://github.com/projectmesa/mesa/tree/134.subrepo The tests will need to be updated a little bit, but before I do that, I want to know that we are interested in moving in this direction. @projectmesa/maintainers --- move forward or no?

[jackiekazil]

When I was getting folks setup to run models in Mesa, I found that installing Mesa AND then running the examples was confusing, because the examples were inside Mesa.

This repo exists and now it is up to date: https://github.com/projectmesa/mesa-examples

Question -- what is the purpose for us to have it in this repo, which couldn't be solved by having 2 repos on our machines / remote machine for testing? Why does examples NEED to be in Mesa? Discuss.

• Devil's advocate.

[jackiekazil]

• @projectmesa/maintainers

[njvrzm]

The only strong argument I can think of for keeping the examples repo inside the main mesa repo is versioning: we're still in a phase where we're making breaking API changes pretty regularly, and if the examples repo is separate, even if we maintain parallel versioning, it's possible users will end up with mismatched versions of the two projects. If we do go with separate repos, we should handle this case gracefully, detect the mismatch and issue a clear error so people know what to do. This should be pretty straightforward.

This gets me thinking about having mesa be a command, not just a library. The Click library has some cool ideas around this. If we did that, after someone installed mesa via pip, they could do something like mesa fetch_examples to checkout the examples repo, then mesa launch GameOfLife to run that example. mesa launch by itself could look for a server.py in the current directory or its immediate subdirectories. We could integrate batch running and benchmarks this way too. We could also follow the example of virtualenv and include a mesa init <project_name> command that would set up a skeletal module in the standard format. This could make it easier for people to get started and to contribute new examples.

(edit) For some reason I'm inclined to keep the examples with the repo, but I'm not sure yet why. I'll think about it.

[jackiekazil]

Here is the disconnect with the examples & version. User does pip install mesa==0.7.6. We are currently at 0.7.7. User says, oh look examples! User does git clone [email protected]:projectmesa/mesa.git. User says...

1. Why am I cloning the code when I installed it?
2. Why don't these examples work.

---

That being said, I think with Mesa commands, you could "fix" this. I agree. We talked about doing something like this for mesa init <project_name>, by using cookie cutter. See https://github.com/projectmesa/mesa/pull/242

I don't know why that got closed. I don't think it should have been. Uh oh... yes... I think we should do that. Maybe we should talk about this and how to execute at our next stand up?

[brianrusso]

Keeping examples within this repo will simplify automated testing.

[jackiekazil]

Yes, but moving it to another folder will help those that run pip install mesa to be able to run the examples without having to dig to some crazy place in their python path.

[jackiekazil]

Sprints -- propose a plan on how this might work.

[djmitche]

It sounds like co-versioning the examples is still important at this phase in the application's lifecycle, as @njvrzm said. I suspect changing that would lead to a lot of friction for users. It would also make #509 very difficult -- doing multi-repo stuff with Travis is hard as @brianrusso pointed out.

The new importlib.resources support that was discussed at PyCon US 2018 might be a good way to include the example files in the source package (skipping the need to write code to fetch from github) and write them out to a local directory for experimental playing-with.

So I think I'm suggesting:

• create a mesa command with Click
• continue including examples in the repo
• write a subcommand that works like mesa init --template=boid-flockers test and basically just copies a directory out of the package resources. The default template can be an empty Mesa project.

[jackiekazil]

Note -- importlib.resources makes for an easy way to get at files in a system. Barry Warsaw's talk about this @ Pycon -- https://www.youtube.com/watch?v=ZsGFU2qh73E

@djmitche -- this seems like the right approach to me.

[jackiekazil]

Next to do on this. Let's figure out that current patterns for library versus examples are by looking at ...

• Scikit learn
• Keras
• Networkx -- this has an examples folks at the top level Any others?

Maybe also check out if there is a place of experts that we can go ask online?

[rht]

Another alternative: split the examples into essential examples (making sure they cover all the built-in features) and non-essential examples (where they heavily use the functionality of other libraries). In this case, keras has keras-contrib, though keras-contrib also extends base keras.

[jackiekazil]

@rht ty for the suggestion and direct example. I will look into it.

[djfurman]

Suggestion: could each example be implemented within a pipenv that would create an isolated virtual environment for the model and include mesa (plus give deterministic dependencies) to be sure that the dependencies are using the known working versions.

[SebastianoF]

Keeping examples within this repo will simplify automated testing.

My 2 cents thought about this: having tests relying on examples may not be the best practice, as folders visible to tests should only be tests and mesa. (Let me know if I can help with this, the mailing list link is not working...)

About the @djfurman suggestion, I understand the issue! At the same time a virtualenv for each example would be, I guess, unpleasantly tedious to maintain. What do you think about using pip install mesa for the latest stable and pip install -e . from within the repo for your development version?

---

Edit:

Tests on examples and relying on examples looks to me more like integration testing rather than unit testing.

What do you think about storing the integration testing under mesa-examples/tests, and to use the mesa/tests folder for all the remaining unit tests?

(Yes, the problem can be in the mesa repo when an integration test fails in the mesa-examples, although as the fail is triggered running one of the examples, the test should be in the examples repo. Future development of unittesting should avoid the situation of having bugs revealed by the integration testing and not by the unittesting).

Below PR towards this direction.

[Corvince]

Thanks for joining the discussion @SebastianoF! While I totally agree with your opinion, I am not sure I totally understand the following:

My 2 cents thought about this: having tests relying on examples may not be the best practice, as folders visible to tests should only be tests and mesa. (Let me know if I can help with this, the mailing list link is not working...)

Yes, this is certainly not best-practice and you are completely right that we are abusing integration tests for unit testing (which is a bad idea). But whats the reasoning behind only making "tests" and "mesa" visible to tests? Or: What would be different under your proposed re-structuring?

[SebastianoF]

Thanks for the quick reply! I guess an example is worth 1000 words: please check PR 735. The percentage lost in the coverage should be recovered with unittesting in the next iterations. (Got nothing to do in the week end!)

[SebastianoF]

If the question was about why good practice, the main advantage of splitting examples and code is after the following question:

• If you change some feature of mesa, do you have to update it in every one of the examples using this feature, to have the new feature landed on master?

With 2 repos, in the mesa-examples repo there will be a requirements.txt pointing at a specific version of mesa. So that the development of the code repository can be independent from examples, and it will be never out of date respect to the mesa version it is requiring.

Same thing can be said about the documentation.

Also semantically the tests folder in the repository mesa is expected to be there to test mesa. if some of them are testing mesa examples, don't you think the should be below the examples folder, or better in the examples repository?

[Corvince]

Thanks for the quick reply! I guess an example is worth 1000 words: please check PR 735. The percentage lost in the coverage should be recovered with unittesting in the next iterations. (Got nothing to do in the week end!)

okay, then it is just that I would have expected it the other way around - first do the missing unit tests and then remove the example tests

[Corvince]

If the question was about why good practice, the main advantage of splitting examples and code is after the following question:

• If you change some feature of mesa, do you have to update it in every one of the examples using this feature, to have the new feature landed on master?

With 2 repos, in the mesa-examples repo there will be a requirements.txt pointing at a specific version of mesa. So that the development of the code repository can be independent from examples, and it will be never out of date respect to the mesa version it is requiring.

You know what? That may make me look a bit stupid, but I really never thought about that simple solution. I always assumed having two repos makes it very hard to sync them. But pinning the version and then somewhat regularly testing against the latest version sounds easy enough.

Also semantically the tests folder in the repository mesa is expected to be there to test mesa. if some of them are testing mesa examples, don't you think the should be below the examples folder, or better in the examples repository?

I think the original plan was not to test the examples but really to test Mesa with the examples (with all the pitfalls discussed above). But yes thinking long time you are completely right (with tests in the example repo folder)

[SebastianoF]

Hello!

first do the missing unit tests and then remove the example tests Yes, this was the logical thing to do :-/

I believed I could recover the tests lost in the integration testing quickly with unittesting, as the experiment-based tests removed where only two.

Of course I was optimistic... It seems the main bottleneck in passing from integration testing to unittesting is the unittest of the embedded visualization tools.

Happy to hear any suggestion about how to continue with: https://github.com/projectmesa/mesa/issues/134 I can close it and start again unittesting first as said above!

Also when testing other parts, some more questions, or possibilities have arised: https://github.com/projectmesa/mesa/issues/740 https://github.com/projectmesa/mesa/issues/741 https://github.com/projectmesa/mesa/issues/742 https://github.com/projectmesa/mesa/issues/743

Thanks!

[jackiekazil]

Examples were moved.