OSGI bundle

[ebocher]

Hi,

Firstly, thanks a lot for this excellent job. We use your library in open source project OrbisGIS. Our system architecture is based on OSGI. We'd like to know if OSGI will be supported in the future.

Best regards.

[akbertram]

I don't think it should be a problem to add -- but unfortunately I don't really understand OSGI very well - maybe you could help us set this up?

[ebocher]

You must add in the pom something like that

<plugin>
+            <groupId>org.apache.felix</groupId>
+            <artifactId>maven-bundle-plugin</artifactId>
+            <version>2.3.7</version>
+            <extensions>true</extensions>
+            <executions>
+              <execution>
+                <id>bundle-manifest</id>
+                <phase>process-classes</phase>
+                <goals>
+                  <goal>manifest</goal>
+                </goals>
+              </execution>
+            </executions>
+            <configuration>
+              <instructions>
+                <Bundle-Vendor>Renjin authors</Bundle-Vendor>
+              </instructions>
+            </configuration>
+          </plugin>
         </plugins>

[akbertram]

Is there an easy way to test the correctness of the OSGI bundle produced? Do you expect one jar/bundle produced that includes all of Renjin's dependencies, or just add the manifest to all of Renjin's existing jars?

[nicolas-f]

Hi, I'm working with @ebocher .

Is there an easy way to test the correctness of the OSGI bundle produced?

Easy, no. But it is possible to write a maven project to run integration test, as this has been done here https://github.com/orbisgis/h2gis/tree/master/h2spatialtest

Do you expect one jar/bundle produced that includes all of Renjin's dependencies, or just add the manifest to all of Renjin's existing jars

The second option (as advised by OSGi groups), add the manifest to all of Renjin's existing jars.

Thank you best regards,

-- Nicolas Fortin Lab-STICC – CNRS UMR 6285 et Ecole Centrale de Nantes GIS http://orbisgis.org Spatial DB http://h2gis.org Noise http://noisemap.orbisgis.org

[akbertram]

Ok, thanks! What about the third-party jars that Renjin depends on, like guava, or asm?

[nicolas-f]

Guava is already OSGi compatible and asm also

[nicolas-f]

If you are making big java application (with many modules), or web server application, it is very advised to use OSGi architecture for many reasons:

• With OSGi you could run multiple version of the same library without any problems. Without it will not work under monolithic java (single classloader)
• You could install - update modules of your webservers without having to restart it.
• A very interesting feature is declarative services, the instantiation of services objects are done automatically when all service dependency are available. No more headache on the creation order of services.

[akbertram]

@nicolas-f How do we deal with third party dependencies that are not OSGI modules? For example, we depend on jtransforms, which doesn't include OSGI entries in the MANIFEST.MF file.

I get the following error when I try to run BundleTest:

ERROR: Bundle org.renjin.core [17] Error starting file:/home/alex/dev/renjin/osgi-integration-test/target/bundle/renjin-core.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle org.renjin.core [17]: Unable to resolve 17.0: missing requirement [17.0] osgi.wiring.package; (osgi.wiring.package=edu.emory.mathcs.jtransforms.fft))
org.osgi.framework.BundleException: Unresolved constraint in bundle org.renjin.core [17]: Unable to resolve 17.0: missing requirement [17.0] osgi.wiring.package; (osgi.wiring.package=edu.emory.mathcs.jtransforms.fft)
    at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3826)
    at org.apache.felix.framework.Felix.startBundle(Felix.java:1868)
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1191)
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:295)
    at java.lang.Thread.run(Thread.java:745)
ERROR: Bundle org.renjin.script-engine [19] Error starting file:/home/alex/dev/renjin/osgi-integration-test/target/bundle/renjin-script-engine.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle org.renjin.script-engine [19]: Unable to resolve 19.0: missing requirement [19.0] osgi.wiring.package; (&(osgi.wiring.package=org.renjin)(version>=0.7.0)(!(version>=1.0.0))) [caused by: Unable to resolve 17.0: missing requirement [17.0] osgi.wiring.package; (osgi.wiring.package=edu.emory.mathcs.jtransforms.fft)])
org.osgi.framework.BundleException: Unresolved constraint in bundle org.renjin.script-engine [19]: Unable to resolve 19.0: missing requirement [19.0] osgi.wiring.package; (&(osgi.wiring.package=org.renjin)(version>=0.7.0)(!(version>=1.0.0))) [caused by: Unable to resolve 17.0: missing requirement [17.0] osgi.wiring.package; (osgi.wiring.package=edu.emory.mathcs.jtransforms.fft)]
    at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:3826)
    at org.apache.felix.framework.Felix.startBundle(Felix.java:1868)
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1191)
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:295)
    at java.lang.Thread.run(Thread.java:745)
693 [main] ERROR org.ops4j.pax.exam.nat.internal.NativeTestContainer - Bundle [org.renjin.core [17]] is not resolved
693 [main] ERROR org.ops4j.pax.exam.nat.internal.NativeTestContainer - Bundle [org.renjin.script-engine [19]] is not resolved
``

[ebocher]

Dear @akbertram,

Thanks for all the efforts. JTransforms seems actively developed and available on github.
What do you think if we try to contact the developer ?

[akbertram]

@ebocher seems worth trying. However, we have a few more third-party dependencies which are not OSGI dependencies, is it worth embedding these dependencies into a specific osgi dependency? @nicolas-f , how do you deal with the dependency on JTS in h2spatial?

[ebocher]

Yes you are right. ;-) For JTS we have a specific project https://github.com/orbisgis/h2gis/blob/master/jts-osgi/pom.xml @nicolas-f any idea ?

[nicolas-f]

@akbertram OSGi modules should never embed third-party dependencies. You don't have to provide a solution for external libraries. However for your integration test, you can quickly add a pom that embed and publish an external module as we done for jts. However you should not publish that on maven central (use a maven profile for integration test and non osgi external libs builds.)

[akbertram]

@nicolas-f sorry, still not quite sure I understand: how is the h2spatial module loaded into OrbisGIS or another OSGI container if the container is unable to load a transitive dependency like JTS?

[nicolas-f]

OrbisGIS platform defined its own set of OSGi form of external dependency here: https://github.com/orbisgis/orbisgis/tree/master/external

But as H2GIS is a library and not a platform, it does not worry about non-osgi external dependency.

[akbertram]

So how would someone use Renjin as an OSGI bundle? Can you tell me a bit more about your use case for a Renjin OSGI bundle? Is the goal to use Renjin as a dependency for an OrbisGIS plugin, for example? Or for Renjin to a be a plugin itself that provides one implementation of a ScriptEngine?

In either of those cases, how would it be possible to load the Renjin bundle without bundles for JTransforms, commons-math, netlib-java, etc?

[nicolas-f]

Hi,

Yes the goal to use Renjin as a dependency for an OrbisGIS gui console plugin. This plugin would not use ScriptEngine API because some custom configuration and variables need to be injected.

In order to fix OSGi wiring, all non-osgi dependency will be provided as new projects in https://github.com/orbisgis/orbisgis/tree/master/external

[SPalominos]

Hi, I'm also working with @ebocher and @nicolas-f.

We'd like to know if you have news about the OSGI support of Renjin. As said by @nicolas-f, we would like to use Renjin as a dependency to build a gui console in OrbisGIS (which uses OSGI).

Best regards.

[akbertram]

I started adding the configuration to the pom.xml files in Renjin on this branch: https://github.com/bedatadriven/renjin/tree/osgi

But I don't know enough about OSGI to finish it. If you guys can submit a pull request with a proper integration test and we can get it passing, then I'd be happy to get it merged and released.

[SPalominos]

Thank you for your work. I will take time to finish that work and submit a pull request.

[akbertram]

Great! Look forward to it!

[SPalominos]

Hi, I've worked about the OSGI-ification and how to resolve the dependencies problem.

---

The better way will be to make all the dependencies be OSGI compatible. But those changes would probably be done for the very last version of the dependencies. So the versions used by Renjin will need to be updated (kind a lot of work).

To avoid that, there is two solution :

• Use <Embed-Dependency> in the Renjin pom in the OSGI configuration to include the non-OSGI dependencies. Doing that, the dependencies will be included inside the Renjin jar.
• Create a Renjin module with all the external libraries like we have done in OrbisGIS see here with a pom doing the OSGI-ification of the dependency. The pom will contains the instructions to include the dependencies inside its jar and how to expose them in the OSGI way. So Renjin will uses the dependencies inside the external module.

---

[to sum up] Three solutions :

• OSGI-ification of the latest version dependencies and updates Renjin.
• Use <Embed-Dependency> in Renjin pom to include the dependencies.
• Create Renjin module like the external one in orbisgis to make OSGI the dependencies.

[akbertram]

@SPalominos I think we'd like to avoid maintaining our own versions of dependencies. I also don't want to by default include all dependencies in the renjin-script-engine jar as there are many other contexts where you want transitive dependency conflicts resolved.

I think the best option sounds like have a separate renjin-osgi jar (ideally in the dist/ subdirectory of the Renjin project) that embeds the script engine and all non-osgi dependencies.

The other consideration is how you want to handle R packages... for example if someone wants to use the randomForest package in OrbisGIS...

In the end, the most important to me is that we have a solid integration test as part of the Renjin project so we can be sure whatever solution we choose works and stays working as dependencies are added/removed.

[SPalominos]

Hi, we worked about the OSGIfication of Renjin. On our repository, we created several OSGI external dependencies in order to integrate Renjin inside a R console in OrbisGIS. Stills several errors, but it works great! Once everything will be fixed, we will do a pull request to Renjin, taking into account your requirement.

One of the errors get might be linked to your comment :

The other consideration is how you want to handle R packages... for example if someone wants to use the randomForest package in OrbisGIS...

The R command library(...) does not work. It seems to be working only with interactive Read-Eval-Print-Loop (REPL) but not if Renjin is integrated in an application. I am right? Should we manage the R package by ourselves or did we miss something?

[akbertram]

library() and require() delegate to the PackageLoader implementation associated with the Renjin Session. The interactive REPL uses the AetherPackageLoader implementation, but by default the ScriptEngine uses the simple ClasspathPackageLoader implementation, as loading and running arbitrary binaries over the network is generally not desired behavior for a webapp, for example.

I'm not sure what behavior is appropriate in a context like OrbisGIS. Do you want packages loaded on demand over the network? Do you want to ship specific packages with OrbisGIS? Should those packages be loaded through OSGI? Or should the user explicitly manage the list of packages available to the ScriptEngine?

You can specify the PackageLoader implementation to use via the SessionBuilder API, for example: https://github.com/bedatadriven/renjin/blob/master/cli/src/main/java/org/renjin/cli/Main.java#L155

[SPalominos]

Hi, some news about the issue. We have implemented a R console working with Renjin for the R code execution. For that we have transformed Renjin and its dependencies into OSGI bundles.(For the package loading problem, we have used the AetherPackageLoader)

But it stills some very little problems. The core packages (stats, graphics ...) doesn't seem to be correctly loaded and we need to run a command library(stats) during the launch of the R console in orde rto make them available. Its seems to be link to OSGI and how it manages the ClassPath.

We still working on it and once everything is working we will contribute to Renjin.

[akbertram]

If you are building your own Session, than be sure to call SessionBuilder.withDefaultPackages() if you want the default packages like stats, graphics etc to be on the search path on startup.