Cannot parse files that contain UTF8 BOM

[espidev]

Hey,

So some of the users of my project insist on using Windows notepad to edit their configs, but end up with errors since it seems that nightconfig issues errors when it attempts to load files that are saved with it (Invalid bare key: ?#). It works fine when I use other proper text editors. Is there a fix for this, or am I doing something wrong?

[TheElectronWill]

Hello,

It seems like a BOM problem: if the file's encoding is misconfigured, the editor will insert a few extra bytes at the beginning of the file (named "BOM" for Byte-Order Mark), creating an invalid config key.

Solution: The users should choose the encoding UTF8 without BOM or something like that.

Edit: utf8 without BOM isn't available in older versions of the windows' notepad. In that case:

1. try saving the file in ANSI
2. if it doesn't work, you should use another editor. Old versions of notepad are really bad.

[TheElectronWill]

On a second thought, NightConfig could detect the BOM and skip it. Adding this to my todo list!

[espidev]

Any update on this?

[TheElectronWill]

Still on my TODO-list, I'll implement the feature during my next holidays :smiley:

[juliand665]

I'll implement the feature during my next holidays

lies! just tracked a user's bug report down to this lol

[TheElectronWill]

whoops! :eyes: In my defence, a BOM (Byte Order Mark) is useless for UTF-8 because the standard imposes the order, on all platforms. But I haven't forgotten this issue ^^

[ProfessorFartsalot]

Same issue using the built in editor on my pterodactyl panel and forge 40.2.0. Pterodactyl has syntax highlighting too, so having this fixed would be really nice. I shouldn't have to download the config file, edit it in notepad++ and then reupload it to load a config imo.

Edit: This is why I'd rather use pterodactyl's built in editor than have to download and edit a config:

[TheElectronWill]

I take note of the high demand of this feature ;)

The "pterodactyl" editor looks good, indeed, but does it actually add a BOM? I have no idea why it would do that, instead of just editing the file as it finds it on the disk :thinking: If it's not pterodactly, I shall ask: where does the file comes from?

[ProfessorFartsalot]

I'm thinking this may actually already be supported. I have tried editing a toml file just recently with Notepad++ and regardless of UTF-8 or ANSI, it still gets reset. I'll attach a zip containing Storm-5.toml (the reset toml file) and Storm.toml (the one edited by the panel) for analysis.

Pterodactyl is a web panel for server hosts. They have an embedded file editor that loads the files from a Docker filesystem (which is why it's named /home/container). I had no issues editing a config file from extreme reactors, which now makes me wonder if there's an issue with Weather2's config system in particular.

Weather2.zip

[ProfessorFartsalot]

Here is another sample. This one is more of a nuisance to me as every server restart resets it no matter what I do. Misc.zip

Misc-1.toml.bak is the one I had saved and thought it would work, as I only edited misc-1.toml in the panel and saved it as Misc.toml. However, it regenerated it with defaults (Misc.toml). I really don't understand why one mod can be configured while another does not seem to like it.

edit: clarification

[TheElectronWill]

Thanks for the files! There is no BOM in them. Hmm, so if it only happens with some mods it might be a bug in those mods.

Do the logs say something? Is there an error?

[ProfessorFartsalot]

The only thing I see is

[09:18:06] [Thread-0/WARN] [ne.mi.fm.co.ConfigFileTypeHandler/CONFIG]: Configuration file /home/container/config/Weather2/Misc.toml is not correct. Correcting
[09:18:06] [Thread-0/WARN] [ne.mi.fm.co.ConfigFileTypeHandler/CONFIG]: Configuration file /home/container/config/Weather2/Sand.toml is not correct. Correcting

[ProfessorFartsalot]

It is worth noting that I never even edited this "sand.toml" file that is erroring. The server generated it and it's still an error, so maybe it is indeed a bug with that particular mod?

[TheElectronWill]

Hmm, so the config is loaded and then "corrected", probably with some ConfigSpec managed by the mod. Looks like a bug somewhere. In any case I think it would be worth it to report that to the mod :)

[ProfessorFartsalot]

I shall do so, thank you!

[TheElectronWill]

Note that UTF-16 BOM is supported out-of-the box thanks to Java's StandardCharsets.UTF_16. You can pass the charset as a parameter of ConfigParser#parse, and the charset will handle the BOM for you. Only the UTF-8 BOM causes problems.