Collision logging, Transformers for JSON and Standard Files

[Master-Code-Programmer]

Last edit: 2023-11-01

Hi, I fitted the collsion_logging merge request from chapmajs to your current source code (https://github.com/johnrengelman/shadow/pull/126). Also I followed your wish to use typed parameters (which was really needed, because after adding them, some problems became obvious).

After that I used the modified plugin and with the warnings I could fix my problems instantly. Great!

The only problem was, that I was flooded with warnings about colliding META-INF/NOTICE, META-INF/license.txt, readme.txt and similar files. So I added a transformer, StandardFilesMergeTransformer, which takes these standard files, which exist in all big projects in big numbers, and merges them. Since these are all pretty primitive files regarding their structure, no special YAML or JSON files, I just concatenate their contents the following way:

• Standard files are "META-INF/license", "META-INF/notice", "META-INF/readme" or "readme". Which may have the following extensions: none, "txt", "md", "htm" or "html".
• If a standard file is found (case insensitive), its content is read, stripped of leading and trailing newlines, and then searched in the list of already existing contents of that file. The search is case insensitive.
  • If no existing entry was found a new entry is created and its content and origin is recorded.
  • If an existing entry is found, its list of origins is extended with the origin of the content.
• Later, the transformed entries are written like that:

    Origins: attoparser-2.0.5.RELEASE.jar, commons-codec-1.15.jar

                                 Apache License
                           Version 2.0, January 2004
    ...
    ================================================================================
    Origins: flyway-core-8.5.4.jar, flyway-mysql-8.5.4.jar

    Copyright (C) Red Gate Software Ltd 2010-2022

    Licensed under the Apache License, Version 2.0 (the "License");
    ...

• The StandardFilesMergeTransformer transformer is added as default. It can be removed with "removeDefaultTransformers". I added it as default because otherwise:
  • In projects with many dependencies the user gets flooded with information about duplicate entries like "META-INF/notice.txt", "META-INF/license.txt"...
  • Important licensing information written in META-INF/license.txt and other files may be lost.
  • Helpful information written in readme files may be lost.
  • The merging of this plain text files is safe, there is no important logic to follow. Merged HTML may not look that good, but it works.

Edit:

• Added JsonTransformer for JSON files, based on JsonTransformer from @LogicFan. Their code is also under Apache License 2. Extended it, so it can be applied to multiple files.
• Lowered the logging level for colliding META-INF/MANIFEST.MF files, from warning to debug, so the user doesn't get flooded with unimportant messages.

With that changes, any user should be capable to build a fat JAR with more than one Spring (Boot) dependency, right out of the box. Otherwise it's a real pain to find out which files collide, and how to merge colliding JSON files, since that is the only colliding file type, which misses a merging transformer from this plugin. And JSON is very common!

@johnrengelman Please re-visit this, when browsing through pull request.

[Master-Code-Programmer]

@johnrengelman This tests will always fail, since this merge into master from: https://github.com/johnrengelman/shadow/commit/38566d50345de45d25b1404669f9e1d6856e5e99 "Merge pull request https://github.com/johnrengelman/shadow/pull/769 from desiderantes/master" broke them.

Also visible in the history of CircleCI: https://app.circleci.com/pipelines/github/johnrengelman/shadow

This pull-request itself didn't break any tests.

[Master-Code-Programmer]

Just for clarity: With the latest merge from master, all tests run fine.

[REPLicated]

Shameless plug: If you need such functionality before it is implemented in the shadow plugin, you may consider my plugin https://github.com/REPLicated/classpath-collision-detector (it only detects collisions and does not attempt to create fat jars).

[Master-Code-Programmer]

@johnrengelman Hi, can I do anything for you to merge this pull request, does this PR have any problems?

[Goooler]

@Master-Code-Programmer Thanks for the PR! Would you mind rebasing?

[Master-Code-Programmer]

@Master-Code-Programmer Thanks for the PR! Would you mind rebasing?

Sounds great. I will rebase it.

[Master-Code-Programmer]

@Master-Code-Programmer Thanks for the PR! Would you mind rebasing?

Did the rebase on another branch and all tests are passing, but I want to check it manually, to make sure everything went actually fine.

[Master-Code-Programmer]

I reopened this on another branch in PR #962.