Suggestion: Improved JSON format for mod info

[rbuckton]

Introduction

After having spent quite a bit of time working with NodeJS and NPM, I've become quite impressed with the power and simplicity of the package.json file format. I've noticed that FML internally uses a somewhat similar format in the form of the mcmod.info file, but I've found that there are a few drawbacks to the format:

• While most mods embed mcmod.info in the JAR, there doesn't seem to be a public repository of these files. As a result, each mod needs to write its own update checking logic (or rely on a shared library).
• The "version" field of mcmod.info is arbitrary. As a result, update checking logic must be unique to the version format used by the mod author.
• The "dependencies" and "requiredMods" fields of mcmod.info do not specify version information.

The package.json format used by nodejs shares similar characteristics with mcmod.info, but provides some additional capabilities:

• The "version" field requires a canonical format, via semantic versioning.
• The "dependencies" field is a map of required dependencies to their minimal version, using a specialized format for specifying version ranges.
• The "repository" field contains information about where to clone or contribute to the package if it is open source.
• The "bugs" field contains information about where to file bugs against the package.
• The "directories" and "files" fields contain information about the contents of the package.

Proposal

I propose that a new JSON format be introduced for FML mods. This format would use the following structure:

NOTE: The following listing uses TypeScript interfaces to describe the structure of the JSON file.

• The ? token following a field name marks the field as optional.
• The | token between types means the field must be either of the types.
• The [] tokens following a type or parenthesized type indicate an array.

NOTE: Not all of the existing fields from mcmod.info are listed below, and some have been renamed or moved. This does not mean they should be lost, but rather merged into this format as appropriate.

// The primary structure for the JSON file
interface Mod {
  // The name of the mod, used as a key for dependencies (previously "modid")
  name: string;

  // A descriptive name for the mod (previously "name")
  title?: string;

  // A long description for the mod
  description?: string;

  // Additional keywords or tags for the mod
  keywords?: string[];

  // The semantic version of the mod: http://semver.org
  version: string;

  // The primary author of the mod (previously "credits"/"authorList"). 
  author?: string | Person;

  // Additional contributors to the mod (previously "authorList")
  contributors?: (string | Person)[];

  // A url to the homepage for the mod
  homepage?: string;

  // A url to download the binary version of the mod
  url?: string | FileResource;

  // Urls to download additional resources for the mod
  urls?: (string | FileResource)[];

  // The source code repository for the mod, if it is open source
  repository?: string | Resource;

  // The means with which users can file bugs
  bugs?: BugTracker

  // Required dependencies for the mod (previously "dependencies", "requiredMods", "mcversion")
  dependencies?: DependencyMap;

  // Optional dependencies for the mod
  optionalDependencies?: DependencyMap

  // Developer dependencies for the mod
  devDependencies?: DependencyMap;

  // License information for the mod. See https://spdx.org/licenses/
  license?: string;

  // Directories in the mod (under the mods/[mod name] folder)
  directories?: string[];

  // Files in the mod (under the mods/[mod name] folder)
  files?: string;

  // Extensibility for future updates
  [key: string]: any;
}

// Extended information about a Person
interface Person {
  // The display name for the person
  name: string;

  // The email for the person 
  email?: string;

  // A url for the person (homepage, blog, profile, etc.)
  url?: string;
}

// Extended information about a url resource
interface Resource {
  // What type of resource is this? A file? A feed? A git repository?
  type: string;

  // The url to the resource
  url: string;
}

// Extended information about a file url resource
interface FileResource extends Resource {
  // The MD5 hash of the file resource (in hexadecimal)
  md5?: string;
  // The SHA1 hash of the file resource (in hexadecimal)
  sha1?: string;
}

// Extended information about a bug tracker
interface BugTracker {
  // A url to a public bug tracker
  url?: string;

  // An email for bug reports
  email?: string;
}

// A key-value map for dependencies to the required version spec:
interface DependencyMap {
  [dependency: string]: string;
}

Dependencies

A dependency can specify the version range required (useful both for version testing when loading the mod, and to allow loaders or mod management tools to check for updates and version mismatches. For an example of version ranges, see https://docs.npmjs.com/misc/semver.

Example

Here is an example from converting the mcmod.info for Thermal Foundation:

{
  "name": "ThermalFoundation",
  "title": "Thermal Foundation",
  "description": "The foundation of a Thermally enhanced world!",
  "version": "1.0.0-RC4+58",
  "author": "TeamCoFH",
  "contributors": [
    "KingLemming",
    "skyboy026",
    "ZeldoKavira"
  ],
  "homepage": "http://www.curse.com/mc-mods/minecraft/222880-thermal-foundation",
  "dependencies": {
    "Minecraft": "1.7.10",
    "MinecraftForge": "10.13.2",
    "CoFHCore": "3.0.0"
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/CoFH/ThermalFoundation"
  },
  "bugs": {
    "url": "https://github.com/CoFH/Feedback/issues"
  },
  "files": [
    "ThermalFoundation-[1.7.10]1.0.0RC4-58.jar"
  ]
}

And here is an example for Forge:

{
  "name": "MinecraftForage",
  "title": "Forge",
  "version": "10.13.2+1291",
  "dependencies": {
    "Minecraft": "1.7.10"
  },  
  "homepage": "http://files.minecraftforge.net/",
  "author": "LexManos",
  "urls": [
    {
      "type": "Universal",
      "url": "http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.7.10-10.13.2.1291/forge-1.7.10-10.13.2.1291-universal.jar",
      "md5": "e71e88c744588fdad48d3b3beb4935fc"
    },
    {
      "type": "Installer-win",
      "url": "http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.7.10-10.13.2.1291/forge-1.7.10-10.13.2.1291-installer-win.exe",
      "md5": "0b8598a75955dedf99a56e39c5c721ce"
    },
    {
      "type": "Changelog",
      "url": "http://files.minecraftforge.net/maven/net/minecraftforge/forge/1.7.10-10.13.2.1291/forge-1.7.10-10.13.2.1291-changelog.txt"
    }
  ],
  "repository": {
    "type": "git",
    "url": "https://github.com/MinecraftForge/MinecraftForge"
  },
  "bugs": {
    "url": "https://github.com/MinecraftForge/MinecraftForge/issues"
  }
}

Benefits

By adopting this format, and assuming a central repository to which mod authors could publish these files, in addition to embedding them in the JAR, there would be a number of benefits:

• Mod authors can easily specify version ranges for dependencies.
• Mod authors have control over how their mods are downloaded (automatic updates, or require the user to navigate to a home page or an adfly link, etc).
• A central repository (like http://modlist.mcf.li, Curse, or something else similar to http://npmjs.org) could manage these dependencies.
• A tool similar to npm could be used to install and update mods on a server or client
• Server operators could easily update their servers and ensure compatible versions of dependent mods are updated all at once.
• Clients could easily download mods by following links in the JSON files stored either in a central repository or from a Minecraft server.

[AbrarSyed]

By all means, I totally agree and think this is a great specification.

Big Problems

• Specifying a canonical version notation: Modders will cry bloody murder if they are forced to conform to any standard whatsoever.
• A central repository: Much easier said than done. Curse is definitely trying hard to get that status, but only time will tell if it will actually gain the adoption necessary.
• Security: NPM packages are distributed as the source code (or minfied editions) that can be checked by the users very easily. Minecraft mods are compiled editions, and arnt nearly as easily audit-able.
• Addoption: the mcmod.info format came out at arround MC 1.3.2 if I recall correctly.. and there are still mods that dont have one.

Suggested Improvements

• Add a potential keybase account link to the Person interface. Could potentially allow for a sort of WOT system where people have signed each others mods and stuff.

Edit added 1 more big problem.

[matthewprenger]

• I think enforcing a versioning standard would actually be a great idea. People would bitch, but they'll get over it. It would eliminate many of the troubles that are around today dependency wise.
• I agree with Abrar that a central repository isn't feasable by Forge. CurseForge seems to be the best option so far.
• There are few things in both the @Mod and the ModMetadata that could probably be removed/changed for 1.8.3

Overall :+1: to the idea

[rbuckton]

@AbrarSyed

• Semantic Versioning - Yes, switching to semver if you're not already using it can be difficult, but that would only be needed for the package format. It works for a very large population of NodeJS package authors, however.
• A central repository - I think a central repository could evolve over time. Rather, what would be more interesting is just having the ability to point at a url to read the JSON file. At that point, a central repository is just a place that has the community consensus as being the central repository. With a well defined REST approach to reading the mod info, private servers could provide their own endpoint for a compatible client to use to define which mods are needed to connect to the server.
• Security - This would come from community consensus around a trusted central repository. The proposed syntax provides some simple mechanisms for validating the signature of a file, but in general its no less secure of an approach than what we already have today.

[cpw]

Semver is definitely a long term goal for versioning..

[immibis]

Why is semantic versioning a requirement for automatic update checking? Surely as long as versions are dotted-numeric, it's easy to compare them.

[cpw]

semver is an easy way to specify dotted numeric, and it also provides a relationship between the various numbers separated by the dots, allowing for lessthan and greaterthan relationships to be asserted..

[immibis]

I'm just confused about that, because you can already compare dotted-numeric version numbers, even if they don't follow semver.

What would Forge do with enforced semver anyway? I guess you could have an option for "only show major updates" or something like that - but "major" means something different to players than to developers, and it doesn't exactly correspond to the first part of a semver number.

[Kubuxu]

Problem is when people start include reviews or alphabetic characters. Semver just says how to compare them but more importantly when increase which version number.

[LexManos]

We will NEVER get people to agree on specifically semver. But hopefully getting people to agree on psudo-semver is the idea. Being able to do 1.0.1 > 1.0 > 0.1 is UNGODLY USEFUL. The thought that 1.2.0 is not compatible with 1.1.0 we dont care about. {would be useful as we could prevent/handle api changes, However MC is special as APIs SHOULD be compatible within a MC version} The important part is the 1.0.1>1.0>0.1 But untill we can force that... we can't really do anything.

[JBYoshi]

Here's an idea for the version arguments: How about we have two version fields? One would be an integer for machines, which would be incremented every version, and one would be a string for users, which would not have to follow any format whatsoever.

[OvermindDL1]

Give it up, I tried this year's ago, most modders, programmers, etc are too stupid to follow decent versioning, proper api handling, or anything that make a modicum of sense at all.

[ST-DDT]

Any progress here? I offer my help if it is needed.

[AbrarSyed]

Write me a regex (or two) that detects semver compatible version strings, and also supports x.x.x.x.x.x.x with infinite separated numbers, and also accounts for a preceding MC version.

Then I can put that in ForgeGradle as a warning and hopefully start nudging everyone over.

[Cazzar]

@AbrarSyed https://regex101.com/r/tT1dX7/3 though, if you also want it to be a minimum of x.y.z as the version string, I would use (?:(\d+\.\d+(?:\.\d+){0,1})-){0,1}(\d+(?:\.\d+){2,}) A nice thing with this one, you have the 2 capture groups, 1 being Minecraft Version and 2 being the "SemVer"

[Cazzar]

Though if you want a more... Advanced one that is lenient to (x.y.z-)a.b.c...(-ANYTHING) kashike and I ended up with this which has named capture groups for Minecraft and full version (which contains version and extra).

The examples in the link should help explain it.

[ST-DDT]

@Cazzar This is a really good regex that we should use to verify the version format. Maybe allow MC as MC version prefix and x as wildcard? But I'm not sure. Ex: MC1.8.x-1.0.1.20150832

[Cazzar]

@ST-DDT that's as simple as prefixing the regex with: (?i:MC)?

[Kubuxu]

Remember that semver allows for prerelease - and metadata +. Fullest Semver I can think of is 9.9.9-abc.rc2.5+md5.abcd.git.abc123 Order of prerelease and metadata can be swapped but AFAIK they can't exist more than once.

[ST-DDT]

@Cazzar We need one regex to verify the format. One that unites them/us all. I'm not sure whether we should add full semver format support. The (prerelease) state is ok but the rest should be discardable meta stuff.

[Cazzar]

@Kubuxu The laziest way I could do that is this

And whilst the thought of having the standard of SemVer fully enforced would be nice, though it also has previously brought up some issues IIRC

[maxov]

@Kubuxu @Cazzar https://github.com/sindresorhus/semver-regex/blob/master/index.js#L3 might be useful

[AbrarSyed]

https://github.com/MinecraftForge/ForgeGradle/commit/65f4c7f0b5409cb38b6ca0d4a4f8050ea38a160d