Enable release train repository aggregation for Eclipse

[nikhilnanivadekar]

Cc: @guw

[guw]

FWIW, I'm not sure if there is better documentation but this one is a good intro: https://wiki.eclipse.org/Simrel/Contributing_to_Simrel_Aggregation_Build https://wiki.eclipse.org/SimRel/Simultaneous_Release_FAQ

Basically, you need to make the p2 repo url available to the B3 aggregation process. This is the one that is aggregating from all participating project.

Certain requirements must be met: https://wiki.eclipse.org/SimRel/Simultaneous_Release_Requirements

[ujhelyiz]

Hi,

we at the VIATRA project are really interested in Eclipse Collections. However, given we are part of the Photon release train, for that we have to ensure that Eclipse Collections is available from the aggregator. Do you have any plans how and when would you like to join?

If you need any help, I also volunteer to help; I have followed the required process for the VIATRA project last year.

Thank you for your help!

[nikhilnanivadekar]

@ujhelyiz we are definitely interested in adding Eclipse Collections to the release train. We wanted to do that for Oxygen but we were too late to get to the process. We would like to join as soon as it is possible and feasible. Before we get on the aggregator we will need our p2 artifacts to be signed. I tried a lot to get that going as a part of 9.0.0 but I was not successful. If you could help us with that as a first step it will get one deliverable out of the way. We can do service releases for the signing purposes. Is there something in Eclipse Collections which would need to be fixed or will need approvals from the Photon release group then we can get that going as well? What are the timelines we want to target the integration?

Thanks again for your interest in Eclipse Collections!

[ujhelyiz]

The most important target dates are available in the following two sites:

• https://wiki.eclipse.org/SimRel/Simultaneous_Release_Requirements
• https://wiki.eclipse.org/Photon/Simultaneous_Release_Plan

As far as I can see, most things you are doing are fine for inclusion into the simultaneous release, no specific approval is required. There are some steps to be done (mostly related to stating your intent):

1. Create a release record at PMI for the build that corresponds to the version that will be included in Photon. It can be reversioned later, if your plans are not finalized yet, e.g. https://projects.eclipse.org/projects/modeling.viatra/releases/1.8.0 is still a stub because we have an interim release this december.
2. At least one committer from the collections project must register to the cross-projects issues list and send an email with the PMI record (something like this: https://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg14670.html)
3. Have your update site content aggregated into Photon site.

Steps (1) and (2) have to be done before Photon M4 (Friday, December 15, 2017), but it is recommended to do as soon as possible. For including the code, the common policy is again as soon as possible; also depending on the projects you use. It is also common to include milestone builds for the various.

For the VIATRA project, it would be nice if some version (unsigned milestone is also ok for now) of Eclipse Collections would be included into the M2 build, but I know that is quite tought thing to ask because the short deadline (M2 builds need to be available next week).

---

About the signing, I will have a look whether I can find out something and will report back soon.

[guw]

Just to clarify: Eclipse Collection does not need to participate in the common repository in order to be consumable by others. For years we've been using EclipseLink and Jetty and added those versions to the common repo without Jetty or EclipseLink participating in the aggregator themselves.

The VIATRA project can simply consume the bits from the Eclipse Collections update site as a dependency and make it available from their update site. The statement "we have to ensure that Eclipse Collections is available from the aggregator" is wrong. You can simply include it in your repo and contribute it like any other 3rd party library to the release train repo.

Having said that, it would still be nice to have one version contributed by Eclipse Collection in order to avoid multiple parallel versions. However, it's not mandatory for any other project depending on Eclipse Collections.

[ujhelyiz]

@guw Thanks for the clarification; indeed I know we could consume it from their release sites. However, given the preexistence of this issue, I think the project wanted to be included in the release train by their own right. For this reason, we believe it is better for us to help the project to be included in the repository and then consume them accordingly. The steps I have described are related for this kind of inclusion.

To summarize, if Eclipse Collections wants to join the release train, I try to help every way I can; if not, VIATRA will consume them as a dependency.

[ujhelyiz]

About the signing issue, I have resumed the thread at #294 (that seemed related) - I hope I have found the issue you have.

[ujhelyiz]

Hi again,

This week (in preparation for the M2 build of Photon) I would like to add a build of VIATRA to the simultaneous release repository that has a dependency on Eclipse Collections. For this reason, I have one of the following options:

1. I can add VIATRA with a dependency to the latest released version of Eclipse Collections; and when you provide your own build, we will coordinate to ensure that only a single version of Eclipse Collections is included into the repository. In this case, I will stick to a release version for now to keep dependencies clean.
2. If you are interested, I can initialize an entry for Eclipse Collections in the release train repository (and provide you a Gerrit change to comment on). In this case, we could reasonably include any build from download.eclipse.org; my first guess would be 9.0.0.M4; but I am open to alternatives.

@nikhilnanivadekar Which approach do you prefer?

[nikhilnanivadekar]

@ujhelyiz let us go with option 1 for now. We can initialize an entry for Eclipse Collections day after. I am going to release 9.0.0 tomorrow. What do you think?

[ujhelyiz]

Option 1 is fine by me; however, tomorrow might too late for Photon M2 (generally, contribution deadlines are around 17:00 CET). Given that, I will add our latest build with Eclipse Collection 8.2.0 as the latest release, and will update the contribution after 9.0.0 is available (if it is done before the deadline, it will be added to the M2 builds as well).

[nikhilnanivadekar]

@ujhelyiz we have 9.0.0 released. Can you please guide me for the next steps to contribute to release train?

[ujhelyiz]

Basically, everything is described in https://wiki.eclipse.org/Simrel/Contributing_to_Simrel_Aggregation_Build

But to provide a short overview, the following steps are required:

• You should ask webmaster for write access to the simrel git repository for your committers who want to be able to push releases.
• You have to install the Aggregation Editor in an Eclipse version; a step-by-step description is available in the aforementioned wiki page; if you are familiar with the Eclipse Installer, it also have a configuration for the simultaneous release.
• Then you have to initialize a contribution as described in https://wiki.eclipse.org/Simrel/Contributing_to_Simrel_Aggregation_Build#For_new_project_contributions
  • When it is done, you have created an eclipse collections-specific descriptor file that you should later keep up-to-date with your current releases.
  • You have to add some contact information who can be connected in case there is some issue in the aggregation. If you think, feel free to add myself also as a contact; I can almost certainly help with build-related issues.
  • Given VIATRA already provides Eclipse Collections 8.2, it should be removed during the initialization to avoid having two different versions aggregated.
  • If you'd like to, my previous offer still stands to create this initial contribution; I only ask you who you want to add as contact person.
• All changes should go through the Gerrit code review tool at the foundation; the most specific reason for this is that it allows verifying the build before merging.

I hope this is helpful; if not, feel free to ask for clarifications.

[nikhilnanivadekar]

@ujhelyiz apologies for not responding sooner. We were out at JavaOne. I will like to take you up on the offer to create the initial contribution. You can add me as the contact person. Thanks!

[ujhelyiz]

I have created an initial version here: https://git.eclipse.org/r/#/c/110114/ (and also added you as a reviewer).

What I have just noticed right now that the p2 repository does not contain any Eclipse Feature that is used to provide a user-visible version of your plug-in or plug-ins, e.g. features are included in the category list of the simultanous release repository.

If you are interested, I will have a look at extending your site with a feature as well; in the meantime Eclipse Collections is not added explicitly to any category in the simrel contribution.

[guw]

@ujhelyiz That is correct. There is no feature but only a category. It might be possible to create a feature but it could become tricky. I'm not sure if a different reactor is necessary, which might result in a second Jenkins job.

[nikhilnanivadekar]

@ujhelyiz I got the gerrit review. I am not an expert in this domain. So, I will go with your and @guw 's guidance. I will be interested in adding a feature to help make it discoverable. Thanks a ton for helping out with this.

[ujhelyiz]

Ok; I will have a look at creating a feature in your build process shortly and respond when its available (it might take a few days).

In the meantime, what do you think, should we wait add collections 9.0 to the repository as suggested in the gerrit review, or wait for the feature (and a corresponding build later)?

[ujhelyiz]

I have provided PR #383 that provides a p2 repository with an eclipse feature. Sadly, as @guw suspected, it is really necessary to provide an extra maven reactor, otherwise the feature project did not build as it did not found the p2 bundle provided by EBR.

[nikhilnanivadekar]

@ujhelyiz I have created 9.1.0 release and also have pushed first milestone release successfully 9.1.0.M1. What are the next steps?

[ujhelyiz]

Now the Gerrit change https://git.eclipse.org/r/#/c/110114/ should be updated with this release and merged into master. We are right in time to do that before the M4 milestone of Photon.

I tried to update the change myself, however, if your milestone build is at http://download.eclipse.org/collections/9.1.0.M1/repository, there is some issue as it does not contain any feature (e.g. there is no features folder there). Was the build published by https://hudson.eclipse.org/collections/job/publish-p2-repo/11/? Because I don't see in the logs any sign that the second maven reactor executed.

[nikhilnanivadekar]

@ujhelyiz I did not change the build config. I guess I need to do that. I can create a M2 build tomorrow. Can you please help me with what needs to change in the build config in publish-p2-repo? Do you have access to view the config? https://hudson.eclipse.org/collections/job/publish-p2-repo/configure If not I can copy paste the details here.

Thank you very much for your help!

[ujhelyiz]

Yes, I can see the job configuration, and it should be updated as described in https://github.com/eclipse/eclipse-collections/pull/383#issuecomment-341016338:

Publishing the repository happens basically the same way as you do in https://hudson.eclipse.org/collections/job/publish-p2-repo : you copy the directory from p2-feature/org.eclipse.collections.repository/target/repository (and p2-feature/org.eclipse.collections.repository/target/p2-repository-«version».zip if you also want to publish the zipped version as well) into the directory /home/data/httpd/download.eclipse.org/collections (in any structure you seem fit).

The easiest way I see to add this is to modify the publish-p2-repo job:

• Keep the existing maven build step as is.
• Add a new maven build step between the existing maven build step and the shell script execution.
  • The most important thing to set up is POM file attribute (hidden behind an advanced button) - it should be set to p2-feature/pom.xml
  • Otherwise a default clean install should work fine.
• Then update the shell scripts to use the repository created by the second maven step.

[nikhilnanivadekar]

I updated it. Can you please verify if it looks good? I can trigger a publish build for SNAPSHOT version. If it works fine will cut another milestone release later today.

[ujhelyiz]

The maven build looks fine, but I have some feedback on the shell script:

• I would not copy $WORKSPACE/p2-repository/org.eclipse.collections/target/repository - it is entirely succeded by the repository inside the feature-repository folder - you can simply leave out the corresponding cp call
• I am not entirely sure about the cp -r $WORKSPACE/p2-feature/org.eclipse.collections.repository/target/p2-repository-$RELEASE_TAG.zip . call; it might work, but we should check the results.

Finally, before you run this build again, I'd remove the previous 9.1.0.M1 build from download.eclipse.org - you can do that either by running a shell script in a hudson job that deletes the corresponding folder; or use sftp to log in the dev.eclipse.org, and navigate to the download folder there. If you don't want to delete the previous build, use a different RELEASE_TAG - merging the files from the two repositories is a dangerous idea.

[nikhilnanivadekar]

I removed the p2-repository copy command. The build should be removed automatically as rm -rf $RELEASE_TAG is executed before copying over the artifacts.

[nikhilnanivadekar]

We forgot to add p2-feature module in the parent pom: https://github.com/eclipse/eclipse-collections/blob/master/pom.xml#L118 That is why the version never got upgraded. Let me know if you would like to contribute. If not I will raise a PR later tonight. I will create M2 tomorrow and we can test it out.

[ujhelyiz]

We cannot add p2-feature to the parent pom - it has to be a separate build, otherwise it will not find its dependencies in a p2 repository.

If you want to ensure that this new build has the version specified as $RELEASE_TAG, you can add a third maven step (between the two maven steps), selecting the p2-feature as the pom.xml to build and add org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=$RELEASE_TAG as goals. Don't use here the usual Maven Versions plug-in as that does not know about eclipse artifacts such as features and repositories.

[nikhilnanivadekar]

I updated the config and ran the build. The build is failing due to some license issue:https://hudson.eclipse.org/collections/job/publish-p2-repo/13/console

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.eclipse.collections:p2-feature:[unknown-version]: Could not transfer artifact org.eclipse.collections:eclipse-collections-parent:pom:9.1.0-SNAPSHOT from/to eclipse.license (http://download.eclipse.org/cbi/updates/license/): Cannot access http://download.eclipse.org/cbi/updates/license/ with type p2 using the available connector factories: BasicRepositoryConnectorFactory and 'parent.relativePath' points at wrong local POM @ line 14, column 13

Any ideas what is going wrong?

[ujhelyiz]

I have tried to rerun this build on my computer using the command mvn org.eclipse.tycho:tycho-versions-plugin:set-version -DnewVersion=9.1.0.M1 -V -B -e -DRELEASE_TAG=9.1.0.M1 and it worked as expected.

Looking at the detailed log it might be a connection error towards download.eclipse.org - can you try to rerun the build as is?

[nikhilnanivadekar]

tried again, no luck. From the exception seems like it is related to something in the pom?