Cleanup maven structure - use semantic versioning - deploy on Maven central?

[axkr]

How can we move releases to Maven Central?

• https://maven.apache.org/repository/guide-central-repository-upload.html

Do we have to include all dependencies which are not on Maven Central? External Java sources are already refactored into maven artifact: matheclipse-external with this commit a230537 But what about (Snpshot) JARs from other repositories (currently: jas, apfloat,...)? Are there Maven tools which can help?

Do we have to rename these dependencies to don't get in conflict with other JARs?

Are GitHub Packages a better choice?

• https://github.com/features/packages

How to reduce dependencies? For example hipparchus is used but dependencies use commons-math. antlr is used in logicngbut we always use our own parser. commons-csv vs univocity.

How can we keep this structure more "stable" in future releases (semantic versioning):

• https://semver.org/

The current coredependency tree:

org.matheclipse:matheclipse-core:jar:1.0.0-SNAPSHOT
+- org.matheclipse:matheclipse-external:jar:1.0.0-SNAPSHOT:compile
+- org.matheclipse:matheclipse-frontend:jar:1.0.0-SNAPSHOT:compile
+- com.atlassian.commonmark:commonmark:jar:0.15.0:compile
+- com.atlassian.commonmark:commonmark-ext-gfm-tables:jar:0.15.0:compile
+- de.uni-mannheim.rz.krum:jas:jar:2.6.6017:compile
+- org.apfloat:apfloat:jar:1.10.0-SNAPSHOT:compile
+- com.google.guava:failureaccess:jar:1.0.1:compile
+- com.google.guava:guava:jar:29.0-jre:compile
|  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
|  +- org.checkerframework:checker-qual:jar:2.11.1:compile
|  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
|  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
+- org.apache.logging.log4j:log4j-core:jar:2.13.2:compile
+- org.apache.logging.log4j:log4j-api:jar:2.13.2:compile
+- org.slf4j:slf4j-simple:jar:1.7.25:test
|  \- org.slf4j:slf4j-api:jar:1.7.25:compile
+- org.slf4j:log4j-over-slf4j:jar:1.7.25:compile
+- org.hipparchus:hipparchus-core:jar:1.7:compile
+- org.hipparchus:hipparchus-clustering:jar:1.7:compile
+- org.hipparchus:hipparchus-fft:jar:1.7:compile
+- org.hipparchus:hipparchus-fitting:jar:1.7:compile
+- org.hipparchus:hipparchus-ode:jar:1.7:compile
+- org.hipparchus:hipparchus-optim:jar:1.7:compile
+- org.hipparchus:hipparchus-stat:jar:1.7:compile
+- commons-codec:commons-codec:jar:1.14:compile
+- org.apache.commons:commons-csv:jar:1.4:compile
+- com.google.code.findbugs:jsr305:jar:3.0.0:compile
+- junit:junit:jar:4.12:test
|  \- org.hamcrest:hamcrest-core:jar:1.3:test
+- com.tngtech.archunit:archunit:jar:0.14.1:test
+- com.tngtech.archunit:archunit-junit4:jar:0.14.1:test
+- org.logicng:logicng:jar:1.6.2:compile
|  \- org.antlr:antlr4-runtime:jar:4.7.1:compile
+- com.itranswarp:compiler:jar:1.0:compile
+- org.jgrapht:jgrapht-core:jar:1.4.0:compile
|  \- org.jheaps:jheaps:jar:0.11:compile
+- org.jgrapht:jgrapht-io:jar:1.4.0:compile
|  \- org.apache.commons:commons-text:jar:1.6:compile
|     \- org.apache.commons:commons-lang3:jar:3.8.1:compile
+- org.gavaghan:geodesy:jar:1.1.3:compile
+- tech.tablesaw:tablesaw-core:jar:0.38.1:compile
|  +- org.apache.commons:commons-math3:jar:3.6.1:compile
|  +- it.unimi.dsi:fastutil:jar:8.3.0:compile
|  +- org.roaringbitmap:RoaringBitmap:jar:0.8.12:compile
|  |  \- org.roaringbitmap:shims:jar:0.8.12:runtime
|  +- com.univocity:univocity-parsers:jar:2.8.4:compile
|  +- com.ibm.icu:icu4j:jar:65.1:compile
|  \- io.github.classgraph:classgraph:jar:4.8.60:compile
+- tech.tablesaw:tablesaw-html:jar:0.38.1:compile
|  \- org.jsoup:jsoup:jar:1.12.1:compile
+- tech.tablesaw:tablesaw-jsplot:jar:0.38.1:compile
|  +- com.fasterxml.jackson.core:jackson-databind:jar:2.10.3:compile
|  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.10.3:compile
|  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.10.3:compile
|  \- io.pebbletemplates:pebble:jar:3.1.2:compile
|     \- org.unbescape:unbescape:jar:1.1.6.RELEASE:compile
+- tech.tablesaw:tablesaw-json:jar:0.38.1:compile
|  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.10.3:compile
|  \- com.github.wnameless.json:json-flattener:jar:0.8.1:compile
|     +- com.eclipsesource.minimal-json:minimal-json:jar:0.9.5:compile
|     \- com.github.wnameless.json:json-base:jar:1.0.0:compile
+- tech.tablesaw:tablesaw-excel:jar:0.38.1:compile
|  \- org.apache.poi:poi-ooxml:jar:4.1.1:compile
|     +- org.apache.poi:poi:jar:4.1.1:compile
|     |  \- org.apache.commons:commons-collections4:jar:4.4:compile
|     +- org.apache.poi:poi-ooxml-schemas:jar:4.1.1:compile
|     |  \- org.apache.xmlbeans:xmlbeans:jar:3.1.0:compile
|     +- org.apache.commons:commons-compress:jar:1.19:compile
|     \- com.github.virtuald:curvesapi:jar:1.06:compile
\- org.owasp.encoder:encoder:jar:1.2.2:compile

[axkr]

@rjolly moved script-engine from Maven module matheclipse-core to matheclipse-io.

matheclipse-core should contain only packages which can be used in android development in the future.

[rjolly]

Noted. Looking forward to being able to draw everything from maven central!

[axkr]

Using google-java-format to reformat source codes:

• https://github.com/google/google-java-format

The Eclipse plugin adds a google-java-format formatter implementation that can be configured in Window > Preferences > Java > Code Style > Formatter > Formatter Implementation.

[HannesWell]

I'm also interested in Symja being published on Maven-Central and therefore offer my help in this issue.

But before going for the actual deployment I suggest to improve the following points in the maven structure plus the points you already mentioned in your initial code.

I would like to start with improving and simplifying the structure of the pom.xml files.

• Define the versions of all dependencies in a dependencyManagement section in the root pom (they can then be updated from one central location)
• provide test dependencies in the root pom for all modules (avoids duplication and if they are not needed it is not problematic)
• build java-doc and sources for all modules in the regular build (java doc generation can be made less verbose) not only during releases. This way you get early feed back about java-doc errors
• generally try to avoid duplication and reduce the number of properties
• Beautify the pom files by complying to Maven's element order recommendation/convention. This way the pom files have a appears that is similar to what most of the developers are used to.

In the second step I would improve the dependencies in the following ways.

The jars in the symja_android_library/lib seem to be not necessary for me? The dependencies are defined in Maven's pom.xml. This brings me to the question: Do you use the Eclipse Maven integration m2e? From the many Eclipse files I assume you use Eclipse.

Regarding your questions:

Do we have to include all dependencies which are not on Maven Central? External Java sources are already refactored into maven artifact: matheclipse-external with this commit a230537

You can include any available Maven-style repository in the <repositories> of a pom. So the answer is: no, you don't have to include dependencies not on maven-central, as long as they are available in any Maven-style repository. Since you can specify a version for a dependency you can also stick with an old version without any problem. EDIT: While it is possible to refer to dependencies not on Maven-Central it is not recommended.

But what about (Snpshot) JARs from other repositories (currently: jas, apfloat,...)? Are there Maven tools which can help?

For Snapshots it's basically the same, they are also not a problem. You just have to specify the desired snapshot version (usually with suffix -SNAPSHOT) and have to make sure that the corresponding snapshot repositories are enabled as well. EDIT: I was wrong here. Artifacts that are deployed to Maven Central must not contain SNAPSHOT dependencies. The Maven-Enforcer-Plugin offers a rule to enforce this.

How to reduce dependencies? For example hipparchus is used but dependencies use commons-math. antlr is used in logicngbut we always use our own parser. commons-csv vs univocity.

If certain transitive dependencies are not used by Symja, respectively the code paths that require them, they it can be excluded trough the pom. If dependencies use a similar library (like with hipparchus and commons-math) I fear that it is not easily possible to replace one by the other without porting the code, since the namespaces are different. The only way I see would be to ask the dependency to update from commons-math to hipparchus as well.

As third step I would work towards publishing symja to maven-central.

So if you agree I would start with the points mentioned first and open

[axkr]

Yes please go ahead.

I'm using the JARs in the symja_android_library/lib for faster testing different versions to see the differences and to use a similar structure as in the releases.

If possible the releases should also be generated:

• https://github.com/axkr/symja_android_library/releases
• https://github.com/axkr/symja_android_library/wiki/Installation

[HannesWell]

@axkr thanks for merging the PR. I've added a brief section to the developer wiki, on how to setup the IDE and how to import the Symja sources from git. https://github.com/axkr/symja_android_library/wiki/Developer-FAQ#how-can-i-prepare-an-ide-to-work-on-symja-source-code

[HannesWell]

As @rjolly pointed out in the discord-chat and one can deduce from this Maven-FAQ (https://maven.apache.org/repository/guide-central-repository-upload.html#faq-and-common-mistakes) patched dependencies can be (re-)published by using your groupid. So for symja we don't have to concern to much about embedded dependencies, they can be simply published using the <groupid>org.matheclipse</groupid>. Furthermore I think we should move embedded dependencies into their own Maven module. I'am about to open a separate issue, to discuss this suggestion.

[HannesWell]

After I have read through the guide at https://central.sonatype.org/publish/ and created the following list of tasks from it in order to prepare the publication of Symja to Maven Central:

• [x] Claim the org.matheclipse namespace at Sonatype and request access to the repository
  • @axkr since you already deploy snapshots manually from time to time to https://oss.sonatype.org/content/repositories/snapshots/org/matheclipse/ I assume you have done that already?
• [x] Fulfil the publishing requirements for Maven-Central (as listed in https://central.sonatype.org/publish/requirements/)
  • [x] Supply Javadoc and Sources
  • [x] Sign Files with GPG/PGP
    • [x] Publish the Public Key as described here: https://central.sonatype.org/publish/requirements/gpg/#distributing-your-public-key
      • @axkr have you done that already? I found md5 and sha1 signatures in the snapshot repositories but I'm not sure if they are created with the required key
  • [x] Sufficient Metadata and dependencies so that the deployed artifacts are self-contained with respect to Maven central
    • This is a simple task and most, if not everything, is already present (but I have to check it)
  • [x] Set up deployment plug-ins and targets
    • The exact configuration depends on if you want to inspect the staged artifacts or not. At least for the first release I think it would not be bad if this is done.

Furthermore I think we should automate the deployment process using GitHub actions. So as a first step I can

• [x] set up the workflow to sign the artifacts and deploy them as snapshots with every change on the master branch.

If this is operational the release deployment should be simple. Furthermore I think we should

• [x] set up GitHub actions to create a release in GitHub with a git-tag, the jars on the release page and maybe a change log automatically upon a release. Most of the steps necessary are described here: https://github.com/jonashackt/github-actions-release-maven

Before we perform the actual release I'm in favour to resolve the following issues:

• #259
• #272

@axkr I can take core of most of the listed tasks. If not yet done you should take care of calming the org.matheclipse namespace and requesting access to the Sonatype OSSRH repo and of generating and distributing the signing keys. Furthermore the access credentials and signing keys have to be uploaded into this GitHub repos secret store in order to automate the deployment via GitHub actions. I can write you the details as soon as I have prepared the workflow (I will use my clone of Symja in order to test it) or maybe we have a video call where we can also review/discuss details if necessary.

And just that you don't wonder: I'm on a one week vacation next week so I cannot continue then.

[axkr]

Claim the org.matheclipse namespace at Sonatype and request access to the repository

* @axkr since you already deploy snapshots manually from time to time to https://oss.sonatype.org/content/repositories/snapshots/org/matheclipse/ I assume you have done that already?

I commented on the old ticket:

• https://issues.sonatype.org/browse/OSSRH-25060?focusedCommentId=1093903&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1093903

[HannesWell]

From now on snapshot builds of Symja are published to the OSSRH on each push to the master branch. The deployed artefacts have sources, javadoc and are gpg signed.

One can get the latest Symja master snapshots with the following coordinates (use the corresponding child-module you need):

<dependency>
  <groupId>org.matheclipse</groupId>
  <artifactId>matheclipse-core</artifactId>
  <version>2.0.0-SNAPSHOT</version>
</dependency>