Configuration/Feature Bloat

[jekirl]

It seems like things are getting somewhat out of hand with all of these configuration options. Even I am not sure what features the bot has.

Perhaps we should have many different configurations rather than 1 giant one. Another idea is to have larger features be extensions that the user can enable or disable and configure independently.

Soliciting comments/input/ideas/whatever....

Thanks all :)

[mhofer117]

when i made pull request #110 (which got merged out again for now) and #125 i made sure to make the new config options optional,

so we could just provide a config.json.minimal and a config.json.advanced and recommand everyone to use the minimal one for easy setup.

then if they want to change something chances are it's already available within the advanced options, just have to document those

[Jay1]

@destiny117 this could work. I'm not so sure about the 2 separate files tho, I would put it in the same file, at the bottom in some optional section, you put a line like "past this line is optional features"

[Zyrixion]

I think that with proper documentation and exampling in the readme, as well as having more complicated settings defaulting to off, and maybe denoting them as more advanced/extra options, it should be fine to keep everything in one config

[beeedy]

I would be a fan of providing a single 'minimal' config template and simply documenting all the extra things that can be configured/enabled. I can see the appeal of the answer put forward by @Jay1 however I think that if we go down that route we should try and organize the config better in general and try and group similar configurations with very obvious separation between groups.

Another thing that I believe should be discussed is refactoring into smaller and more specific modules (ie evolution, transfer, etc.) It has become quite irritating to work on a change only to find that when you are ready to submit a pr someone has gone and changed a majority of the code that your change was based around. Just my 2c!

[sftwninja]

How about an interactive prompt that creates the initial config.json object? Give each option a sensible default and ask first-hand if the user wants to even consider experimental options. This way, the configuration options can be documented at the time of implementation, and the big manual that is the README doesn't need to be such an anchor for information.

The best example I can give of a good system of this is something like semantic-ui. When first installing it, it presents you with a menu that looks like:

Express installation (Give me sensible defaults) Advanced installation (I know my way around under the hood, give me a few extra options like accounts/pokemon to keep/pokemon to release/CP-LV options) Custom installation (I minmax everything, let me see all the numbers)

All new configurations must exist as a setup option, must give a sensibly short description to the user, allow reconfiguration (e.g. --ignore-config), be specified in one off flags (as most are now), and still result in a human-readable JSON config.

I personally believe this approach is, and has been a great way of doing configuration and exposing features. Linux package managers have been approaching one-time initial configurations like this for years, and I believe it's good for that just by its longevity.

[beeedy]

@sftwninja I like that idea. Even something like a simple python script that walks the user through the creation of the config based on their desires and needs. Also, one thing that I don't know if it has been addressed yet but it would be nice to change defaults without the need to get in the code. I run >6 bots and the configs between them are close to the same, with only minor differences for experimentation. It would be really nice to just define defaults in one place and specify specific details that may be needed per account.

[Jay1]

@sftwninja I thought about it, I think it would be the best way of doing things. But the updating is super bleeding edge right now and I don't know if everybody want to take a couple minute to update the prompt every 2-3 PRs... In a perfect world we keep the config file for develop branch but updating the prompt is mandatory to merge develop into master branch. That way people can work fast on develop and casual users get their fancy and easy to use prompt since they pull from master. We only need to update the prompt every ~10-15 PRs since we don't merge develop very often

[sftwninja]

@beeedy I haven't delved into it, but we could just overlay flags passed to the client on top of the config.json. Would being able to pass additional flags (e.g. --min-keep 1 --step-size 300) be a step in the right direction?

@Jay1 Yeah, there's a much better way of planning them out I'm sure, and if it's an idea that's decidedly the better way to go, we can totally figure that out. Thanks for the support.

[Zyrixion]

@sftwninja I think command line arguments like that go against the idea of clean, visible, readable configuration. You'd have to start writing batch files just to run the darn thing with all the correct parameters, which is what having a config file seeks to alleviate.

[sftwninja]

@Zyrixion Go have a read my suggestions again, that's not what I implied. The config.json would be evaluated first, and any command line flags would be overlayed on top. This could help those users who make good use of multiple bots at once, but again, ideas are welcome.

[mhofer117]

How about something like this:

pokecli.py:

from jsonmerge import merge
[..]
loaded = [merge(load['defaults'], load['accounts'][i]) for i in config.accounts]

not sure if it works, but this way we can set up all the 'complicated' options within the defaults of the same config file and the accounts array can be kept simple.

another advantage is that the GUI proposed by @sftwninja can be created seperately (if anyone is willing to) and updated periodically for the master-branch. but we are good without in the dev branch

not sure how to describe all the options, maybe with "_comment" values inside the json?

config.json.example would look something like this, i also grouped some of the options:

{
  "accounts": [
    {
      "_comment"; "set your account information here",
      "auth_service": "google",
      "username": "[email protected]",
      "password": "password",
      "location": "University of Southern California, Los Angeles, CA",
      "GMAPS_API_KEY": "some api key here",
      "_comment"; "below are optional values overriding the defaults",
      "behavior": {
        "STEP_SIZE": 50,
        "STAY_WITHIN_PROXIMITY": 500
      },
      "release_terms": {
        "MIN_SIMILAR_POKEMON": 2,
        "THROW_POKEMON_NAMES": ["PIDGEY"]
      },
      "MIN_ITEMS": {
        "ITEM_POTION": 10,
        "ITEM_SUPER_POTION": 10,
        "ITEM_HYPER_POTION": 10,
        "ITEM_MAX_POTION": 20,
        "ITEM_BLUK_BERRY": 10,
        "ITEM_NANAB_BERRY": 10,
        "ITEM_REVIVE": 10,
        "ITEM_MAX_REVIVE": 10,
        "ITEM_RAZZ_BERRY": 10
      },
      "POKEMON_EVOLUTION": {
        "PIDGEY": 12,
        "WEEDLE":12 
      },
    },
    {
      "auth_service": "ptc",
      "username": "someusername",
      "password": "password",
      "location": "University of Southern California, Los Angeles, CA",
      "USE_GOOGLE": false,
      "KEEP_CP_OVER": 1000
    }
  ],
  "defaults": {
      "_comment": "these are defaults for all extensive options",
      "USE_GOOGLE": true,
      "behavior": {
        "STEP_SIZE": 100,
        "EXPERIMENTAL": true,
        "SKIP_VISITED_FORT_DURATION": 600,
        "SPIN_ALL_FORTS": true,
        "STAY_WITHIN_PROXIMITY": 500
      },
      "release_terms": {
        "KEEP_CP_OVER": 500,
        "KEEP_IV_OVER": 50,
        "KEEP_IV_MIN_PERCENT_CP": 0,
        "MAX_POKEMON_HIGH_IV": 999,
        "MIN_SIMILAR_POKEMON": 1,
        "KEEP_POKEMON_NAMES": ["MEWTWO"],
        "THROW_POKEMON_NAMES": []
      },
      "POKEMON_EVOLUTION": { },
      "MIN_ITEMS": { },
      "MAX_CATCH_ATTEMPTS": 10,
      "console_output": {
        "LIST_POKEMON_BEFORE_CLEANUP": true,
        "LIST_INVENTORY_BEFORE_CLEANUP": true,
        "VERBOSE_PATHING": false,
        "PLAYER_DATA_FREQUENCY": 1
      }
  }
}

[huntergregal]

So it may cause some overhead but imo would be where this bot is heading anyway:

• Implement a database with a single table
• In the flask app create an "Add account" page that has a form for all the config options (blank = disabled or default)
• Use flask_sqlaclhemy to update/add the account and it's configs to the DB.
• when the bot is run with an account ID (-i id), query the configs from the DB

This is obviously a decent overhaul, and i could even get working on this if I get the time and if this gets +1's. Again the added database may scare some people off but even if a docker mysql container is thrown in with a quick startDB.sh init script it should fix alot of these config bloat/user friendly issues.

Edit: Could even have an "enabled" toggle under each account or in another "account manager" page that has a table populated with the accounts in the DB and a "enable/disable" toggle button to choose whether the account is active.

In this manager page the table with rows for each account could also have a "status" button which takes you to the stats page for that account.

Again...this is thinking with expandability. Perhaps not where this bot wants to go.

[jekirl]

as @Jay1 said right now, things are

moving fast and breaking things Complicates the entire situation.

[beeedy]

@huntergregal that sounds like where I personally would like to see this bot go. I don't want to speak for all the users/devs but +1 from me

[RikudouSage]

I think it would be best to use a web gui in which user can select features he/she wants. Fully configurable, saved in db and dynamic.

[sftwninja]

I like @huntergregal's idea. I'd like to hear a bit more on how the initial lifecycle would work when a user hits the project for the first time. For instance, would a user add an account, set all the options on the web gui and then click a button that said "GO"?

And, my two cents, SQLite (and its accompanying API) is more than enough to do DB work, and I'd be happy to offer my time to help do that. SQLite is integrated into Python, introduces no new dependencies, and is plenty small and fast to do this kind of work. We're not seeing heavy bandwidth/contentious data here. A database file per user/bot should be fine.

[RikudouSage]

If you don't mind I could write the whole web app in php and it would communicate with the worker via db only, best would probably ve SQLite as it does not need anything installed

[olife373]

This is an aging post but i think that the whole bot should start to turn to an SQL database driven structure for user data and config and maybe even logging and statistics.

1.) Its easy to implement through the web app @RikudouSage , 2.) All the config could be wrapped in the web app as well as eliminate user data dumps (because it can be written to the database instead). It could provide secure login or even remote control over the trainer functionality such as allowing the user to stop and start the bot loop or even adding additional accounts on the fly and controlling them all. 3.) If we write an API we can easily implement new features over it versus re writing web.py to include parsing new values. We would instead just update the template to pull the new data from the API.

The point is the possibilities are endless if we can manage to store most of the data in an SQL database of some sort and have a way of interacting with the data via a web app.

I would really like to see this come to life.