Requirements for a V3(beta)

[novadragonDOTspace]

First of, i am taking reference to this message by @naramsim here

Hi guys, if you like we can start working on a V3beta version! If you are volunteering we can start writing some code.

On my side, I can only help you with the deployment part, since I don't really have time to implement such complicated things now. We can start anew and use or drop Django, same goes for CSVs, we could switch to other formats. Reply if we can on you!

I'll copy this comment over at #966.

Originally posted by @Naramsim in https://github.com/PokeAPI/pokeapi/issues/1037#issuecomment-1973742468

As much as i personally would love to start anew, there are some informations missing before one could actually do such,

What are the requirements to an v3 (Technical or Modelwise)? What should be carried over from v2? What explicitely not? Or does nothing matter and someone should just start?

[Naramsim]

Hi, so far the APIv2 had a great success and the only big problem for the user is understanding the difference between pokemon, pokemon-species, pokemon-forms.

I say we cannot modify the current properties nor delete them but only add them for the current API due to the huge consumption, as we have done so far.

That said other small problems were raised in the past. I'll write down a small list of thoughts

1. https://github.com/PokeAPI/pokeapi/issues/332
2. We use changelog tables. In my mind, I thought we might have an API that's generation-centric, such as:

/api/v3/generation-i/pokemon/bulbasaur
/api/v3/generation-iii/pokemon/bulbasaur
/api/v3/generation-i/item/old-rod
/api/v3/generation-iii/item/old-rod

But maybe this adds difficulties in understanding. 3. We are serving static files so the new one should also be able to be parsable and servable with a simple function or we'll incur again in terrific costs. 4. We could start from a predefined standard as OpenAPI and/or use an API generator. 5. We might just work on the GraphQL current API and promote it to V3. Especially if we could strip the pokeapi_v2_ prefix from all the fields. And maybe getting rid of pokemon-forms and pokemon-species and condense them into a single PG table that then can be retrieved with GQL. 6. GQL now has no costs, but it crashes when someone submits a heavy query. We should at least upgrade to 2cpu8ram machine that's about 50$ per month, but our current ad revenue is 30$/m and the donations are basically non-existing. 7. A big problem for us is finding people to update the data. It would be cool to automate it and maybe build the data from external sources without having to maintain CSV files.

To be honest the point 5 is really cool!

This are just some thoughts.

[novadragonDOTspace]

I'll add my own personal thoughts, with the

0. Some of my thoughts are also based upon the fact that in semantic versioning, a major version change is accompanied by Breaking API Changes.
1. I personally dont like the pokemon, pokemon-species, pokemon-forms table format, cause it doesnt combine with a relatively more modern understanding of what a "Pokemon" is.
2. the current csv files are not that easy to understand how to edit, unless you already have Database Knowledge.
3. that there is big doubling with the ID and the Name fields in the NamedRessources. The ID could aswell actually just be the name, in the named-ressources.

Additional understanding question: How/Where are the static files served? Is it just json-formatted files outputted somewhere? 4. If its just json-formatted files somewhere on github, couldn't we actually just forego any actual programming of an csv -> json conversion, and directly edit the json files, and a CI-Run to verify that the edit works.

[omniuni]

One thing I would, as a user, specifically request is NOT to drop non-GQL support.

GraphQL is powerful, but it's a huge pain to interface with if you aren't using a pretty heavyweight library to do most of it for you.

So far, the biggest complaints I have that I would like to see addressed:

• The ID and name for a resource should be included with any data point. This is mostly for items with a URL. Whether it's another API endpoint or the graphic URL, there should be separate fields that give the id (say, 1) and key (say, bulbasaur).
• The weirdness with encounters data should be addressed. In addition to the changes noted in the PR linked above, the encounter-method endpoints are somewhat broken, and show what I'm fairly certain should be a description under the name field. Also, several games, such as shining-diamond just don't show up at all, even though I'm fairly certain that there is indeed data for them under the base version, such as diamond.

[novadragonDOTspace]

Not really actively writing code for this, and rather preparing schema notes for it privately, but I would have had the id for bulbasaur not be 1 but bulbasaur, so what is currently listed as name in this iteration

The ID gives one the weird flavor as if it were the National Dex Number, but with Mega Venusaur the earliest, we would need to already diverge there. Would the id be 4? 0003M? M0003? That would be a notation that would need to be invented first. (In the current db, its 10033, which barely makes sense to me). in my opinion it should be venusaur-mega. string based IDs are not unheard of, given that some like to employ UUIDs for that.

One big issue that pops up in my mind is, that localized strings are kinda handled terribly, IMO. For each little localized string, you'd have to search up the language again and again. (At least in PokeAPI.Net its an often repeat Pokemon.Names.Where(n => n.Language == "de").FirstOrDefault(); just to possibly get an name in german.) Inspired by like fandom wikis , one could have refer the url https://pokeapi.co/api/v3/pokemon/ditto to exclusively english data, and have https://pokeapi.co/api/v3/de/pokemon/ditto to the same data in, in this case, german. alternatively, one could also just use subdomains, like wikipedia for this alias https://de.pokeapi.co/api/v3/pokemon/ditto, or another alternative, https://dpokeapi.co/api/v3/pokemon/ditto/de How'd that be handled in the background? Still a big question, i guess the actual database schema would still refer to an languages table or something, a thing that already just looks annoying in an simplified ERM

There's also the big question, where does each other want what data. Loaded Question, i know. Example: Does one actually want an separate Berry endpoint, or have an optional Berry-Datapoint at the actual Item endpoint, that appears if something is a berry or not.

The Evolution Endpoint: The current Evolution Detail Type seems like a big mess. Gamefreak progressively adds evolution details that are only there for single pokemon, turn_upside_down is like the most infamous one when Inkay released, and the only one that is listed in the api. Actually Terrible in database design. Should that actually exist, or should just have what is like an trigger field that is like "level, item, trade, special" (simplified), and the evolution-detail field is a string that is filled depending on the trigger. (Thats maybe more dynamic than what someone wants)

An issue i have with the current DB: Do we actually need Pokemon Conquest Data? I dont even know why its here, why would someone want it there. What was veekun cooking there? from an biased viewpoint, i'd say Mystery Dungeon Data, and on an unbiased view Legends: Arceus Data is more asked for, especially since Legends: Arceus is mainline, but changed many a mechanic.

[novadragonDOTspace]

I get why FlavorTexts/Proses are localized, but do we actually want the same for Effect Texts? Related: Why does the ShortEffect Text even exist?

[Naramsim]

I agree with the ID being the name of the pokemon! And getting rid of the conquest data.

[EKO-GITHUB]

@GreatNovaDragon the long effect / short effect texts make sense for some categories such as moves / abilities. The long effect can get extremely long without any overview of what the move actually does, so a short effect can be useful. For example the substitute move I've rewritten, the long effect goes on for about 4 times the amount shown:

I didn't like the data provided by PokéAPI so I've rewritten all of them along with a custom parser.

[novadragonDOTspace]

I am currently writing some code for some reduced personalized usecases in a repo mirrored to GreatNovaDragon/pokeapi-nova i probably could later expand that into full API that displays most of the PokeAPI Data.

Will still take a while

[novadragonDOTspace]

Several things like Moves, Abilities, Pokemon have an Generation field, which should be replaced with VersionGroup. Like, platinum version-group already introduced an alternate Pokemon Form in Girantina Origin Forme, so i dont know why that already wasnt in veekun/pokedex, maybe they thought it was a oneoff?

[EKO-GITHUB]

Several things like Moves, Abilities, Pokemon have an Generation field, which should be replaced with VersionGroup. Like, platinum version-group already introduced an alternate Pokemon Form in Girantina Origin Forme, so i dont know why that already wasnt in veekun/pokedex, maybe they thought it was a oneoff?

As far as I'm aware Veekun stopped just before SWSH. Did they already include USUM data? USUM introduced a bunch of Gen 7 moves that were released after SM, so just listing Gen 7 for is not completely correct as SM does not include those moves. There are also moves that exist in the USUM data but not in the SM game data, so a Pokémon knowing those moves can not be imported into that version group. (for example Mind Blown) There are also Gen 6 moves introduced with ORAS that are not present in the XY game data so importing those will not work (e.g. Origin Pulse) The most correct way would be to list the specific version group that move was introduced in but that requires a bit of research

[novadragonDOTspace]

In GreatNovaDragon/PokeAPI-Nova (far from finished), I have set the "IntroducedIn" field as a Version group.

Given that for #1020 I am already doing my own research, finding out in which version group something released in is the smallest issue.

[phalt]

Just chimining in to say I'm very busy with some pretty awful things happening in my life at the moment (my youngest son just passed away). So I don't really have time to comment or contribute other than saying: I give my grace (whatever that is worth) for you all to build this. I trust ya 😄

[EKO-GITHUB]

I'm very sorry to hear that @phalt no parent should ever go through that.

[VentanYu]

7. A big problem for us is finding people to update the data. It would be cool to automate it and maybe build the data from external sources without having to maintain CSV files.

@Naramsim I have a strong feeling about this point. Personally, when I look for information, I always use wiki (such as bulbapedia, wiki.52poke). Although the data is not structured, wiki is still the most detailed and accurate source of information that I know of, as they provide a very good editing method for ordinary users, which is something that the PokeAPI currently lacks.

Therefore, my thoughts:

1. Is it possible to collaborate with these wikis to use their content and structure it?

2. Alternatively, provide a more user-friendly data editing method, reducing difficulty should be the best way to distribute maintenance costs.

[Naramsim]

Hmm, that's a great point as well for continuing supporting v2 as well!

So, for 1. I think the data on those websites is not copyrighted. See the footer of https://bulbapedia.bulbagarden.net/wiki/Ultra_Space for example. So we might scrape content from there to keep the data up-to-date. The best would be to use the content of decompiled ROMs/games, that's what @veekun used to do but couldn't afford to carry on anymore due to numerous factors I believe.

For 2. we build the API from the SQL database that it's build on top of CSV files, with many modifications on the way. So it might be hard to provide a simple UI able to reverse these modifications/build-process, in a way to have CSV modified

[EthanRDoesMC]

I’ve been building a Pokédex-type app in Swift in the last couple of weeks so I’ll throw in some unsolicited requests/ideas as someone who’s been kinda dealing with the API from a client perspective as of late:

1. Yes, absolutely, please have some kind of way to avoid having to do a lookup for each and every localized string. I don’t need all 9 at once, I’m only displaying one at a time. And to that end, if we could fallback on an English string (and maybe include some kind of flag for when that happens) that would be awesome.

better form support: 2. since Pokédex entries link to species, my current app structure is divided into dexes with lists of species. Indicating which form is default for a given Pokédex… at present that’s not something I can do, and I’m having to write that manually. Someone mentioned wanting better Legends Arceus support… here’s one area it reallllly needs help in. Perhaps return a collection of forms, rather than a species? Or both? 3. Flavor text does not exist for individual forms, which means that Mega varieties do not and currently cannot have their flavor text within the current design. Also again see the Legends Arceus support, right now I have a flavor text from PLA commenting on how different a very normal-looking default Decidueye looks 4. we have is-mega but not is-gigantamax, why? lol perhaps for future proofing as GF seems likely to continue with gimmicks, it might be better to have some kind of form-type node with its own info. 5. actually just in general the way species, forms, varieties (??????) and pokemon objects are organized is really confusing and often requires multiple fetches to get what i want to access, can that be reorganized thanks. why do i have to do two fetches to find out if a form is a mega

6. Also im very down for a more user-friendly way to contribute. having individual file objects in the repo for entries might be a good way to take advantage of GitHub for contribution, but i dont know how costly that is for performance from a db perspective, and non-developer people aren’t going to contribute that way.

good luck with v3 all :)

[Genschere]

Here some thoughts from me regarding the data model:

1. Text based IDs I think using text values as primary ID is not the best idea. If this is put into a database, the text IDs will lower the performance when joining tables together significantly and it can also cause index fragmentation and I'm pretty sure there will be an index on that column. I don't want to say you shouldn't do it. I just want you to understand that it can have bad side effects. When thinking about the oversized pokemon_moves table I get a bit of a headache though.

2. Merge Pokemon types Does it make sense to remove pokemon_types and integrate the data into pokemon as 2 columns? Doing the additional lookup doesn't make sense imo and having a lot of NULL values in the column for the second type slot wouldn't be that big of a deal. Except we see it as likely that they add pokemon with 3 types. That would be stupid but you can never know. It's nothing that will go out of hand like the conditions for evolutions.

3. Linking dex entries If pokemon_species doesn't exist anymore, the entries in pokemon that belong together should be linked together somehow. The species_id in pokemon_dex_numbers won't work anymore when removing the species part. This will have to work differently and it probably needs another table for linking the different varieties of a pokemon to the same dex entry. Adding a dex_order column to this would be a good idea as well.

4. Types of Forms/Varieties Putting the different kinds of varieties into a separate file makes perfect sense. This could also be used for better differentiation. Apart from mega and gigantamax, it could also be used for the current forms_switchable column in pokemon_species and split up between different ways of form change like ability (e.g. Castform, Arceus, Aegislash) or outside battle item use (e.g. Kyurem, Hoopa) and regional forms.

5. Evolution conditions I agree with the many columns for the different evolution methods being bad design. It would make more sense to separate them and link only this new evolution_method_id to the evolutions but this won't work for everything. Some pokemon require more than one condition and some need an extra value for things like the item ID, known move ID but also happiness and level. If this is separated, it will require 2 tables to make it work nicely and flexible enough to be future proof. One for the different kinds of conditions like level, held item, use item, daytime etc. The other links the evolution with the conditions including the ID of whatever is needed. If one evolution has more than one condition, all conditions need to be met. This would be 3 columns: The ID of the evolution from pokemon_evolution The new evolution_method_id The value for the condition. This can be the item ID, the level, happiness, the daytime or known move ID etc. With this it should be possible to cover everything. New ways to evolve a pokemon or new combinations of conditions for the evolution could be easily added. It will require some coding effort to make this look nice in the API though. When using the table name of what is needed (if there is a table for it) as identifier, it would probably be a bit easier.

6. Getting data from wikis One thing regarding scraping data from wikis: They don't like that. It produces a lot of traffic on their site in a quite short amount of time. A collaboration would make more sense since they could also profit from the result somehow. How? Not sure, I don't know how they work and update their sites.