iis
iis copied to clipboard
Run IIS experiments by relying on spark 3.4 version
This task is related to running a subset of IIS modules currently written in spark 2.4 on the newly available spark 3.4 version.
This may require:
- altering oozie workflow definition by specifying a new sharelib name and defining other spark related configuration parameters such as app/query listeners, history server address, event log directory etc
- changing dependencies
- new version of spark-utils: 2.11 will have to be released, compatible with scala 2.12: https://github.com/CeON/spark-utils/issues/16
- altering java code
- dropping
iis-commonscala sources which are difficult to maintain (and results in false positive errors indicated by the Eclipse IDE) and replacing those simple utility classes with java equivalents