iis icon indicating copy to clipboard operation
iis copied to clipboard

Simplify spark jobs configuration after having them all aligned with Spark 2.4

Open marekhorst opened this issue 1 year ago • 0 comments

This could be considered as #1475 follow-up because citation matching was the last module written in Spark 1.6.

The following properties defined in workflow.xml files:

        <property>
            <name>spark2ExtraListeners</name>
            <value>com.cloudera.spark.lineage.NavigatorAppListener</value>
            <description>spark 2.* extra listeners classname</description>
        </property>
        <property>
            <name>spark2SqlQueryExecutionListeners</name>
            <value>com.cloudera.spark.lineage.NavigatorQueryListener</value>
            <description>spark 2.* sql query execution listeners classname</description>
        </property>
        <property>
            <name>spark2YarnHistoryServerAddress</name>
            <description>spark 2.* yarn history server address</description>
        </property>
        <property>
            <name>spark2EventLogDir</name>
            <description>spark 2.* event log dir location</description>
        </property>

and their spark1 equivalent will be redundant and should be removed.

marekhorst avatar Oct 10 '24 14:10 marekhorst