Find a proper way of dealing with sharelib jars conflicting with user jars

[marekhorst]

Since I was unable to find a decent solution to this problem in #987, where the proposed fix was just a workaround to make affiliation matching working again, we should get deeper into spark/oozie/hadoop internals to find the best way of dealing with sharelib jars conflicting with the jars referenced directly by IIS (defined in pom.xml file and bundled with IIS oozie app).

#987 comments could be quite useful in exploration.

[marekhorst]

#1122 is one another related issue: in order to give user libs precedence over sharelibs/parcels in plain M/R jobs one should set this property in the workflow.xml definition:

<property>
    <name>oozie.launcher.mapreduce.user.classpath.first</name>
    <value>true</value>
</property>

[marekhorst]

In spark, under current CDH 5.16.2 OCEAN cluster setup, it seems user libs have already precedence over sharelibs/parcels.

As already mentioned in https://github.com/openaire/iis/issues/987#issuecomment-534184328 it can be also enforced using spark parameters.