Support compile code for each environment separately

[sratz]

I am not sure if this is really a Tycho issue or a more general problem or even works as designed, but here's the situation.

Assume you have a multi-platform target definition:

      <plugin>
        <groupId>org.eclipse.tycho</groupId>
        <artifactId>target-platform-configuration</artifactId>
        <version>${tycho-version}</version>
        <configuration>
          <environments>
            <environment>
              <os>win32</os>
              <ws>win32</ws>
              <arch>x86_64</arch>
            </environment>
            <environment>
              <os>linux</os>
              <ws>gtk</ws>
              <arch>x86_64</arch>
            </environment>
            <environment>
              <os>macosx</os>
              <ws>cocoa</ws>
              <arch>x86_64</arch>
            </environment>
          </environments>
        </configuration>
      </plugin>

Now, in a platform-independent bundle com.foo.platformindependentbundle, accidentally reference an API that is platform-specific, e.g.

import org.eclipse.swt.internal.win32.OS;

This dependency is provided by a platform-dependent fragment org.eclipse.swt.win32.win32.x86_64 and can only be satisfied on the win32 environment.

Yet, a Tycho build of such a setup will not yield any compile / dependency errors. Those would only occur at runtime, when trying to launch this in a linux or macosx environment.

Looking at the dependency:tree of such a plugin, I can see that during compilation time all platform-specific fragments are considered in the classpath:

[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ com.foo.platformindependentbundle ---
[INFO] com.foo:com.foo.platformindependentbundle:eclipse-plugin:3.25.200-SNAPSHOT
[INFO]  +- p2.eclipse-plugin:org.eclipse.swt.win32.win32.x86_64:jar:3.118.0.v20211123-0851:system
[INFO]  +- p2.eclipse-plugin:org.eclipse.e4.ui.workbench.renderers.swt.cocoa:jar:0.12.600.v20200416-0656:system
[INFO]  +- p2.eclipse-plugin:org.eclipse.swt.gtk.linux.x86_64:jar:3.118.0.v20211123-0851:system
...

This makes it hard to catch such errors during compile time.

Is there a way this could be done to catch such mistakes early-on?

[laeubi]

This makes it hard to catch such errors during compile time.

You can't catch this at compile-time as it is a runtime-error.

[laeubi]

Is there a way this could be done to catch such mistakes early-on?

The only think I can though of would be to compile each bundle for each platform but what should then be bundled into the final jar?

[sratz]

Hmm, I think this is one more situation where Import-Package is preferable over Require-Bundle:

In this example, com.foo:com.foo.platformindependentbundle was using Require-Bundle: org.eclipse.swt.

If I use Import-Package: org.eclipse.swt.internal.win32 I get the expected resolution error:

[INFO] {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86_64}
[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: com.foo.platformindependentbundle 3.25.200.qualifier
[ERROR]   Missing requirement: com.foo.platformindependentbundle 3.25.200.qualifier requires 'java.package; org.eclipse.swt.internal.win32 0.0.0' but it could not be found

Of course, this would still not help if split-packackages were involved, but given that those have all sorts of other issues, this one would probably be not the worst one.

I guess in the end it comes down to making sure that the build pipeline executes unit tests on all platforms to catch such problems.

[laeubi]

@akurtakov maybe this might be also an option for SWT, it seems SWT currently uses some hard to understand ant scripts and manual setup for different OSes.

What I can think of is that we have a multiEnvironmentCompile flag in the Tycho compiler that compiles code for each matching environment skipping those that do not match a platform filter. Currently the Tycho compiler simply don't care and put every environment together.

If that is useful, I might be able to take a look at it in Tycho 3.0, but not sure if it would be worth the effort.

[akurtakov]

I'm not sure how much such thing would help for SWT as it's not only different environments (filters) but also entirely different source trees for each platform. Additionally SWT is not shipping any binary content in the bundle but rather everything comes from the os specific fragment. Frankly, it's quite ugly but I can't think how to improve it substantially without major reorganization for which I simply can't afford the time but would be very thankful to anyone that spends the time to improve it.

[laeubi]

it's not only different environments (filters) but also entirely different source trees for each platform.

Actually they are different source folder where only specific ones are enabled per platform as far as I can see.

What would be needed is one could attach platform-filters to a source folder (like the test attribute), this then won't require one to reorganize the code.

• JDT then ignores each source folder not matching the target environment (that is what one needs to do manually at the moment).
• Tycho on the other side, would compile the same source multiple times but also filtering the source folders for the environment as we already do when building a product most of that is available at Tycho already.

I'm not sure how much such thing would help for SWT

That's why I'd like to ask first if there is essential no real need from the project to change something its not worth the effort, but if I think it could simplify the development and build.

[akurtakov]

• JDT then ignores each source folder not matching the target environment (that is what one needs to do manually at the moment).

Let me query JDT devs for that. That would really be huge for SWT developers setup even if doesn't improve releng immediately. JDT bits are prereqs for Tycho work so no need to start on Tycho before JDT is at least "sold" to the idea.

[laeubi]

Sure that's just an idea that came into my mind remembering my last time development for SWT and thinking about the reported issue here :-)

In the end, this would then also allow consumers of SWT to easier develop bundles that contain some OS specific code. At the moment its a bit messy, for example I once tried to add OLE support for an RCP that is only for windows and have a polyfill for Linux ... If I rember correctly I ended up with two bundles and had to close the one not matching the current dev or something similar ....

[laeubi]

Let me query JDT devs for that.

As I struggled around with SWT build lately I wonder if there is any progress on this?

[tsmaeder]

• JDT then ignores each source folder not matching the target environment (that is what one needs to do manually at the moment).

Let me query JDT devs for that. That would really be huge for SWT developers setup even if doesn't improve releng immediately. JDT bits are prereqs for Tycho work so no need to start on Tycho before JDT is at least "sold" to the idea.

A couple of thoughts on this:

1. One of the problems here is that JDT does not understand OSGI semantics: org.eclipse.swt.internal.win32.OS is not an exported package, so if JDT understood that semantics, one problem would go away.
2. The assumption that the test attribute just takes stuff off the class path is not correct, it hides the test package fragment roots only from non-test code, and only in select circumstances. JDT understands the "test" vs. "non-test" semantics.
3. Even if we did have a "filter" attribute on package fragment roots, tycho would still have to set that attribute on the JDT project, right? If we're special casing this, why not just special case the rename of .classpath.win32 to .classpath?

I think in the SWT case, the problem is that we're really looking at three different bundles, one for each platform. They happen to provide the same packages, but otherwise are independent (both in Java code and the native code). I would think that the setup is the way it is now (as a single project with a classpath for each platform) for historical reasons. My recommendation would be to make three different JDT projects in the repo (SWT_win32, SWT_gtk, SWT_cocoa). Yes, there is some common code, but I believe we could find a solution for that somehow (fragments?)

I think the "just filter pkg fragment roots from the classpath" would work, but it seems a bit overkill if it's just to fix SWT setup issues. In an ideal solution, the windows code would see the windows code and the linux code would see the linux code. Platform-independent code would see what is available at runtime (there are mechanisms for this, for example OSGI package export)

On the other hand, answering the question: "what can I see in this compilation unit" is an interesting question that would deserve a more general answer than what we have now. When JDT started, the "compile time class path" was the only visibility model that existed. Since then, visibility has become ever more restricted: OSGI and module package visibility rules, Maven's test vs main folder, but you could also lump OS dependencies with that. What about usage of non-API? Same problem, in the end.

All of these visibility filters are (except the Java modules) are specific to build or runtime systems. So IMO, they should not be built into the system, but implemented as extensions. In the end, such an extension needs to answer the question:

isVisible(String name, JavaElement elementFoundOnClasspath, CompilationUnit where)

When JDT is looking up a name in the context of a given compilation unit. This does not have to be any less efficient than what we do now. We are already doing those checks, and there is no reason to assume code that is specific to Maven, for example, could not answer this question as efficiently as the JDT code does now.

@mickaelistria for tycho/m2e insights.

[laeubi]

1. One of the problems here is that JDT does not understand OSGI semantics

It don't need to to understand QSGi just what is called "Platform Filters" but I don't know if these are supplied by PDE? That is more a part provided by P2, e.g. the JUnit Classpathcontainer already uses P2 to resolve jars from the running eclipse platform.

2. The assumption that the test attribute just takes stuff off the class path is not correct

This is just an example for that class-path entries already can carry additional information.

3. Even if we did have a "filter" attribute on package fragment roots, tycho would still have to set that attribute on the JDT project, right? If we're special casing this, why not just special case the rename of .classpath.win32 to .classpath?

The filter should not be applied to any packages, but to source folders and the renaming is what is currently used, but that's very inconvenient to use and hard to implement with tools (that's why currently ant is used for that) and not portable at all.

I think in the SWT case, the problem is that we're really looking at three different bundles, one for each platform. They happen to provide the same packages, but otherwise are independent (both in Java code and the native code).

Nope, that is the result of the build but in the IDE there is one project and different source-folders, there is a "common" and there is a "os specific part" that is switched on/of by copy prepared .classpath files around.

I think the "just filter pkg fragment roots from the classpath" would work, but it seems a bit overkill if it's just to fix SWT setup issues.

It would be useful for any one using swt as well (e.g. if I want to write code that uses the OLE API)

All of these visibility filters are (except the Java modules) are specific to build or runtime systems. So IMO, they should not be built into the system, but implemented as extensions. In the end, such an extension needs to answer the question:

I think the visibility of a package is already provided by the acces-rules, and it is more like "consider this sourcefolder: yes/no) anyways if there are other options thats fine, but one must make sure that this is accessible to build-tools (tycho) as well so it should be stored somewhere (e.g in .classpath) and then I won't mind if really JDT does handle this or PDE or m2e ...

[tsmaeder]

Nope, that is the result of the build but in the IDE there is one project and different source-folders

That's why i qualified the sentence with "really". It is not now, but it should be.

[laeubi]

As far as i know there where some attempts to do so but never come to a conclusion, so I'm not very optimistic here. But if you have an idea, maybe you can open a PR at swt and propose an alternative solution?

[akurtakov]

Changing SWT in that way would be very tough as it will not only require to handle the Java side but also the C headers/object files/etc. @tsmaeder So having "conditional source folders" supported by JDT is not feasible?

[tsmaeder]

@laeubi I think you took my analysis of the problem for an action plan there ;-) My analysis is: visiblity of Java names is a complicated problem and it depends on build system, etc. My action plan is: let's take those build-system(etc.)-specific parts out of JDT and make the build-system specific.

[tsmaeder]

Changing SWT in that way would be very tough as it will not only require to handle the Java side but also the C headers/object files/etc. @tsmaeder So having "conditional source folders" supported by JDT is not feasible?

Of course it's feasible (it's software after all), but not trivial: for example, if platform filters are a PDE concept, we's have to introduce that concept into JDT (becuase of dependency order). However, in my opinion, it's a step in the wrong direction: we're adding more special case handling into JDT instead of reducing it.

[laeubi]

Maybe @merks can tell more, but the "platform filtering" is actually a thing from the P2 world not from the PDE world I think as these filters are also applied when computing the provisioning plan.

[merks]

Arguably this problem should be caught in the IDE. I suppose it's more something PDE would/should know to check. No imported package should come from a fragment where that package is not also exported by the fragment's host (except when it's used by another fragment with the same (or less restrictive) filter, I suppose.

But this this package is marked internal so that might be the first clue to double check that this import is a good idea. And this it's flagged with win32 which should be the second clue that this is not a good idea.

I'm not sure that fixing something like this is worth the investment relative to all the other bugs that could be fixed and features that could be implemented.

[laeubi]

Ed, this is not about packages or imports, swt has currently the following structure:

https://github.com/eclipse-platform/eclipse.platform.swt/tree/master/bundles/org.eclipse.swt/Eclipse%20SWT

you will see there are folders

1. common
2. emulated
3. gtk
4. win32
5. ...

these are all source folders and depending on the platform you develop SWT (that is your os) one needs for example copy https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/.classpath_gtk_win32 -> .classpath to enable a certain subset of these source folders.

The idea now is, that there is only one .classpath with all variants, but I can attach a filter like(ws='win32')or (ws='gtk') so when i start my eclipse and set the target then only those are "active" that match my target environment, that's for the IDE part...

At buildtime, currently we have a maven build that calls ant, that calls a compile for each platform and then in the end we have three bundles as a result. So given that such a filter would be supported in the IDE tycho could support this without having to call ant and do some classpath tricks.

[merks]

@laeubi It's a long, long thread where mostly it seemed to be asking about catching mistakes. And yes, I know about SWT's structure. That's why there is this in SWT's setup task:

But I'm not exactly sure how the question about catching errors early turned into a discussion about how to restructure SWT's build...

Will that really solve Sebatian's issue? (I'm doubtful; nothing will prevent importing a package that is visible only on one specific platform, or? And keep in mind I'm asking whether will this prevent a problem in the IDE not whether will it be noticed by Tycho later on where of course you will argue that it builds multiple times and hence will notice.)

How much work will be invested in JDT, PDE, and Tycho to make this work? (It seems like a lot.)

How much better will that make the Eclipse world? (Marginally better it would seem to me.)

Is that worth the investment? (I wouldn't invest my time in it, but no one is asking me to do the work.)

[laeubi]

But I'm not exactly sure how the question about catching errors early turned into a discussion about how to restructure SWT's build...

To give some technical background here:

The problem @sratz is facing here, is that Tycho currently resolves each enabled platform into s set of bundles, so when you build products for three platforms you get the corresponding "natives" for this platform and end up with three "packages" (win, linux, mac), but at compile time Tycho currently just squash all platforms into one build-path because there is only one compile step.

So this turned into the idea, if we could do what we do for products, might be adapted to the compile, so compiling will the succeed for win, but fail for linux+mac, or if successful result in three artifacts.

Of course, as you mentioned, this wil require some effort, and even if most pices are there in Tycho would be more beneficial if we could not just do it for rare cases (like described by @sratz ) we have not a direct usecase, we can do it instead to finally fully "mavenize" the SWT build, where we are facing a similar problem.

How much better will that make the Eclipse world?

Don't know, given that SWT is a crucial technology for the Eclipse eco system and that today we need these ant-scripts

https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/build.xml (70 LOC) https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/buildFragment.xml (340 LOC https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/buildInternal.xml (50 LOC) https://github.com/eclipse-platform/eclipse.platform.swt/blob/master/bundles/org.eclipse.swt/buildSWT.xml (1090 LOC)

having a total of 1500+ LOC and last time I'm asked for help how to adjust the build of SWT and there was close to nothing as response it seems valuable to at least think if we can replace this with a simple <enablePlatformSpecificCompile>true</enablePlatformSpecificCompile>.

How much work will be invested in JDT, PDE, and Tycho to make this work? (It seems like a lot.)

I don't know yet, at least it sound it could be done e.g as of today if I delete something in the .classpath the source folder gets magically removed, if I put it back its get magically added again, so lets assume I do not delete the entry but well have just an opportunity to filter it (whether its done by PDE or JDT). And from the Tycho side, we also has the opportunity already to distinguisch platforms, just no way to tell tycho "pleas compile for A, B and C and if you do so use folder X,Y for A and folder Z,Y for B ...)

And finally I can only quote a famous book by Michael Ende here

Sometimes, when you've a very long street ahead of you, you think how terribly long it is and feel sure you'll never get it swept. And then you start to hurry. You work faster and faster and every time you look up there seems to be just as much left to sweep as before, and you try even harder, and you panic, and in the end you're out of breath and have to stop--and still the street stretches away in front of you. That's not the way to do it.

You must never think of the whole street at once, understand? You must only concentrate on the next step, the next breath, the next stroke of the broom, and the next, and the next. Nothing else.

That way you enjoy your work, which is important, because then you make a good job of it. And that's how it ought to be.

And all at once, before you know it, you find you've swept the whole street clean, bit by bit. what's more, you aren't out of breath. That's important, too...

So if we always look at the long street, for sure it seems we can never reach the end, and probably better give up right now, but if we just look at the next step, we might be surprised that if finally happens in the end. All that stuff about using maven-artifacts, wrapping them and finally having support in PDE has started about three years ago and there where many concerns it would be possible at the end, so it was a really long road, nerveless in the end I can today use it and it helped me a lot in my daily work.

[merks]

I'm at Shinjuku Station and I see many long roads. I must pick and choose. Others of course do the same, and choose differently. :-)

[laeubi]

So maybe if your train is late and you don't want to choose another, do you remember where in P2 it evaluates if a given "platform filter" matches the current Eclipse install? :-)

[merks]

I think finding the call hierarchy of org.eclipse.equinox.p2.metadata.expression.IMatchExpression.isMatch(T) will find such places. E.g., Oomph has method like this for determine if a task's filter matches the current OS:

  @SuppressWarnings("restriction")
  public static boolean matchesFilterContext(String filter, OS os)
  {
    if (StringUtil.isEmpty(filter))
    {
      return true;
    }

    Map<String, String> filterContext = new LinkedHashMap<>();
    filterContext.put("osgi.ws", os.getOsgiWS()); //$NON-NLS-1$
    filterContext.put("osgi.os", os.getOsgiOS()); //$NON-NLS-1$
    filterContext.put("osgi.arch", os.getOsgiArch()); //$NON-NLS-1$

    org.eclipse.equinox.internal.p2.metadata.InstallableUnit filterContextIU = (org.eclipse.equinox.internal.p2.metadata.InstallableUnit)org.eclipse.equinox.internal.p2.metadata.InstallableUnit
        .contextIU(filterContext);

    try
    {
      IMatchExpression<IInstallableUnit> matchExpression = org.eclipse.equinox.internal.p2.metadata.InstallableUnit.parseFilter(filter);
      return matchExpression.isMatch(filterContextIU);
    }
    catch (RuntimeException ex)
    {
      // If the filter can't be parsed, assume it matches nothing.
      return false;
    }
  }

[laeubi]

Thanks Ed I think something like that would be suitable her already!

[laeubi]

org.eclipse.equinox.executable.feature also uses ant to compile a feature for different platforms.