iis icon indicating copy to clipboard operation
iis copied to clipboard

Run IIS experiments by relying on spark 3.4 version

Open marekhorst opened this issue 2 years ago • 0 comments

This task is related to running a subset of IIS modules currently written in spark 2.4 on the newly available spark 3.4 version.

This may require:

  • altering oozie workflow definition by specifying a new sharelib name and defining other spark related configuration parameters such as app/query listeners, history server address, event log directory etc
  • changing dependencies
  • new version of spark-utils: 2.11 will have to be released, compatible with scala 2.12: https://github.com/CeON/spark-utils/issues/16
  • altering java code
  • dropping iis-common scala sources which are difficult to maintain (and results in false positive errors indicated by the Eclipse IDE) and replacing those simple utility classes with java equivalents

marekhorst avatar Aug 10 '23 09:08 marekhorst