[future] [comments] configuration migration

[electronicboy]

trafficstars

Papers configuration file has generally gotten pretty huge, 270 lines is what's currently sitting in my paper.yml, and then 168 lines in spigot.yml, paper.yml includes several options in which overrides settings in bukkit.yml (saving), or spigot.yml (e.g. item despawn), which really creates a clusterf*** for trying to configure servers

In the future, when we diverge from spigot, there will be no real valid reason to stick to having 3 separate configuration files, it creates a lot of confusion and in general is pretty messy, the existing layout of these files is also severely lacking in general organization, in part due to how hard it is to justify moving stuff around, every "move" inside of the yaml files would generally involve a few lines of code to facilitate that and maintain

In the future, it's worth looking into separating the configuration files into a more "friendly" format, multiple config files with consistent intents, I think the line count of these files is somewhat concerning if it comes in regards to throwing it all in a single file vs separating it out into a more "clean" modular layout, would also give us the time to modify the layout of these configurations to be more useful, we would however need to consider a migration plan, which doesn't sound all too pretty :/

the consideration of this is, do we consider also switching the format of these configurations, hocon, for example, offers some useful features such as the ability to retain comments, which would make it much easier to add documentation to stuff that could be used for automatic generation of the documentation on the site, something which is often overlooked when adding new options;

[LaurenceBarnes]

everyone liked that

[kashike]

👍 for cleaning up configurations 👎 for moving from YAML - YAML is what 95% of Bukkit plugins use, and is what people are most familiar with in this community.

[electronicboy]

Yea, hocon was just an idea I tossed around, not entirely too fond of it given the community, and apparently configurate may be able to help us here in the future if we wanted; being able to throw comments in the config classes, even if not persisted to the files, would maaaybe potentially be super nice, not 100% on how "nice" it would look however all in all, but would potentially provide means to auto-generate some config docs

[BillyGalbreath]

Idk how I feel about the modular approach, tbh. Like, it sounds like it would be a good thing but I can't help but imagine myself having to go to 15 different config files in 15 different locations to change 15 different things.

I'm definitely in favor of getting rid of the redundanct bukkit/spigot ymls, though. Simply squashing it all down to a single yml should fix those confusing redundant options. (maybe even getting rid of the vanilla server.properties, too, but not too concerned with that one)

I'm also with kashike about changing from yaml. Big thumbs down from me, too

[YouHaveTrouble]

Leave configs yaml , but squash all things into one file. There is enough confusion about which option is in which config file already. I'd say config sections over separate files.

[benkerllenevich]

If you do go the route of having multiple config files for different categories, consider moving the files into a config directory as to not clutter the root directory.

[electronicboy]

part of my thoughts are that at the very least, we wanna use a new separate file just to make invoking migration easier, if modern config does not exist, hit migration, else, just load

[Machine-Maker]

I like either consolidating stuff into one files, or a config directory with different files for different categories cause one file might get too big. Like I think the essentials config.yml is just too long for one file.

I don’t really care one way or another about what format is used for the config file. Hocon for comments would probably be nice. Or even if we wanted to stick to YAML, have it setup so the files are just copied over so comments stay in-tact, similar to how plugins can do that themselves (no writing to the file as that’s what removes comments).

[YouHaveTrouble]

I like either consolidating stuff into one files, or a config directory with different files for different categories cause one file might get too big. Like I think the essentials config.yml is just too long for one file.

I don’t really care one way or another about what format is used for the config file. Hocon for comments would probably be nice. Or even if we wanted to stick to YAML, have it setup so the files are just copied over so comments stay in-tact, similar to how plugins can do that themselves (no writing to the file as that’s what removes comments).

ctrl+f is faster with 1 file instead of having to search something inside multiple files.

[Machine-Maker]

ctrl+f is faster with 1 file instead of having to search something inside multiple files.

but if stuff is separated into multiple files based on category, if its done correctly, you will know what file to look in without opening each of them.

[YouHaveTrouble]

but if stuff is separated into multiple files based on category, if its done correctly, you will know what file to look in without opening each of them.

In most cases when you're looking for a one or two specific options to change (after initial configuration). Imo searching by the name of that option is more efficient than having to think which category it belongs to. It's not 100% clear in many cases.

[Malfrador]

Isn't YAML (probably with some effort) also able to retain inline comments? EssentialsX does it I think, and some other plugins as well. If its one single configuration file, inline comments would be very helpful for structuring.

I would be against switching to a different format, as YAML is not just very common, but also easy to use (and I personally think it looks way cleaner than other formats)

[Prof-Bloodstone]

I feel like 1 vs multiple files is a matter of preference. I think it'd be nice to split it in a separate dir, but I personally think it'll be very hard given relations of the configurations and going 1 big file is easier. Though I do worry about the size of it, especially if Paper were to add comments to options :hand_with_index_and_middle_fingers_crossed:

I personally don't mind searching for option with grep merge *.yml.

[Machine-Maker]

Yeah, YAML has no issues w/comments as long as you are only reading values. Once you try to save any changes made somewhere else (via a command or whatever) is where you run into problems.

[YouHaveTrouble]

By the way, while we're talking about configs and changes with them, did anyone think about doing someting more like /gamerule command for paper options, at least for the simpler ones? A lot of paper options are per-world already and there are quite a lot that would fit that format. I'm just throwing this idea out here, maybe it'll inspire something?

[Prof-Bloodstone]

I understand that YAML is very common in the community, but given how many issues it presents (even as simple as indentation, people constantly come to paper-help with config broken due to this), I'd love to see alternative, more "flattened" format. And I say this as a person writing YAML files for most of my day at work.

Yaml does support writing comments, iirc even snakeyaml which is included in bukkit can do that.

If Configurate were to be used, it does support multiple different types of files - maybe it'd be possible to prioritize reading paper.hocon file, if it doesn't exist - read paper.yml, fallback to creating paper.hocon if neither exist. It feels like very little logic is needed, and lets users migrate to yaml if they prefer it.

[Prof-Bloodstone]

@YouHaveTrouble I personally don't think it's a good idea. If I have config in file, it's easier to see and search all options as well as version them with git.

[YouHaveTrouble]

@YouHaveTrouble I personally don't think it's a good idea. If I have config in file, it's easier to see and search all options as well as version them with git.

I'm not saying replace the config with a command, I'm saying maybe add the command to enable changing setting from in-game

[Malfrador]

I really don't think that having to keep track of opening and closing brackets and other symbols makes the config easier to understand than YAML for the not-so-experienced user. I'm saying that based on my experience with some of our server team members, explaining them how to create files for a plugin that uses YAML was always way easier than for plugins that use other formats.

I would have nothing against offering both options of course, but it seems somewhat unnecessary to me.

[granny]

These are all the ideas I've seen so far along with the reasons/(arguments/counter-arguments) brought up for each idea.

Config type to use

• hocon
  • allows us to add comments in the config
• yaml
  • what's already being used
  • easier for end-users?
  • will be able to use comments programmatically (Configurate is working on this functionality)

Config Format to use

• everything in one file
  • can easily search through the file for an option using Ctrl+F
• multiple files in a modular approach (config directory or world-specific options in the same directories as the worlds they apply to)
  • can search multiple files easily using the grep command (grep <option> *.yml)

Documentation

• inline/header comments for options
  • categorization using comments
• no comments, use web documentation (what's done already)
  • categorization done in the docs

Let me know if I missed/misinterpreted anything

[prototype464]

Moving my suggestion over from a duplicate report (oops)

Is your feature request related to a problem?

Currently, the paper.yml config is quite large compared to many configs I, as a server owner, go through on the daily. spigot.yml is quite large, but paper.yml has been ever-expanding.

It becomes difficult to go through this config to find what I'm trying to look for, and sometimes without knowing an exact word for what I'm looking for, I have to scroll up and down several times.

This problem is caused by everything being put into the config in a seemingly random order of listing each time it's generated, and any attempts I've personally made for an organized one either leads to it becoming outdated with new options added, or it just not working at all.

Describe the solution you'd like.

A perfect solution to this would be to add text separators to mark sections of the config - if an option is to be added to an already existing config, just have a New section at the bottom!

This would be incredibly useful and much more user-friendly to go through, as a Gameplay section could be for things such as arrow despawning, shield cooldown, anything that affects gameplay (including villager mechanics, etc.)

Another good section would be Exploits, such as the unsupported settings part for headless pistons and the such. Zombie villager discounts and the like.

Generation for anything that affects terrain or mechanics related to it (such as snowing), Networking or Connections for things such as throttling and packet settings, the possibilities for something like this go on and on.

Upon taking more of a closer look at the config, I realize that the aforementioned is already in a settings and world-settings part so to say, but I do think the config could use better overall formatting / sectioning to make it easier to go through the variety of settings available as there are still many options despite those being there.

Edit: This could also be done for Tuinity / Airplane features, and be in a section of their own. Eitherway, sections would be much nicer than the config being purely auto-generated!

[electronicboy]

cleaning up and reorganising the configs is a key part of the goal here already

[ryantheleach]

Spigot has managed to change something in yaml files so comments are now persisted. So that's one less reason to switch away from yaml.

With Paper becoming the default by usage stats, and Spigot close second the minecraft community has lost something valuable, which is supported compatibility with vanilla, whether it's server owners valuing performance over compatibility, or the default effect raising it's ugly head with how difficult it is to restore vanilla behavior.

This has caused rifts in large sections of the Minecraft community, to the point where Fabric servers are becoming increasingly popular among technical players / people influenced by technical play style influencers, and was the large appeal of SpongeVanilla support.

This isn't necessarily a problem, but with the large amounts of bukkit/spigot/paper plugins, it results in a LOT of duplication of effort, as people ask for ports of popular plugins to fabric... When most of this could be solved by papermc shipping a configuration preset that contains vanilla behavior, with justification on why each change is 'bad', e.g. performance, detailing exploits, etc.

Most of this is due to the lack of 'support' for CraftBukkit without Spigot, and the fact you lose the extra API, so if server admins are going to pick between 2 performance and API forks, they would rather pick the downstream one.

Arranging the configurations by whether they exist to fix exploits, performance issues, or customization of hard coded mechanics would go a long way to making this particular pain point easier to address, as well as shipping (disabled by default?) configuration files which can drop-in revert to vanilla behavior, as much as is possible by configuration anyway.

[electronicboy]

craftbukkit hasn't supported vanilla on a technical level for many years, that's hardly changed at all, nor is anybody willing to step up to provide a "vanilla" set of changes for a config file in the first place, we have 0 inclination to spend hours upon hours tryna determine this stuff when after years of asking for help with this nobody has bitten.

There are a handful of things that are behaviour changes in the config files that are relevant to those using paper, EAR and the dupe settings being the only ones I can really recall off the top of my head (almost bed time) which are by default enabled and cause actual behavioural differences.

Most of the breaks between vanilla code changes going back almost a decade, this is something we want to work on, but, once again, after years of asking for support in achieving this I've given up.

the reorganisation of the configuration will be attempted to be done more organically rather than 200 config nodes in a single tree, but, separating stuff towards exploit/perf/customisation seems to be too far wide of a scope of stuff for any sane config organisation given all of the stuff in there

[ryantheleach]

we have 0 inclination to spend hours upon hours tryna determine this stuff when after years of asking for help with this nobody has bitten.

I think the issue is trying to tackle it all at once, rather then just fixing it as things are found 1 by 1. And that it would be damn near impossible before the hard fork anyway, as having little control over upstream means that it could go out of date on any merge.

If a public commitment was made, and a place to sort and organize those concerns, I believe the situation would improve, but it may never be perfect.

[MiniDigger]

I think the issue is trying to tackle it all at once, rather then just fixing it as things are found 1 by 1.

issue is, nobody really comes forward with stuff like that. they just cry that stuff is different and ask for a kill all switch, so I feel like the first step would be to collect and document the differences.

[kashike]

Cleaning up the configuration is something that is on our radar again. The new configuration will likely be split into sections, somewhat similar to what was described by @prototype464 above.

[Machine-Maker]

For some feedback on what I've got so far, you can look here. I wrote some basic tests to make sure the whole "fallback to world defaults" worked here.

[jaen]

Is there some way to import other config files considered?

My use case is that I need to template some dynamic values into configuration (such as PostgreSQL credentials) and it would be more ergonomic to be able to template just the dynamic part separately and import it from the static configuration.

For example I could have the following in LuckyPerms' entirely static config.yml:

!include ./postgresql.yml

# loads of other permission config

and next to it, a postgresql.yml templated from some dynamic source:

data:
  address: {{ secrets.postgresql.address }}
  database: {{ secrets.postgresql.database }}
  username: {{ secrets.postgresql.username }}
  password: {{ secrets.postgresql.password }}

Here's an example of something like this implemented for Python's YAML parser https://github.com/tanbro/pyyaml-include.