mareksimunek

Did you do step 3 from https://github.com/linkedin/dr-elephant/wiki/Quick-Setup-Instructions-(Must-Read) ? But anyway, myself I needed to delete some test to be able to compile with 2.3 version and after successfully running Elephant...

@varunsaxena > If you want Dr.Elephant with open source Spark 2.3, you may have to contribute a fetcher. We can help review it. 1. Do I need to just change...

Try to compile it without specifying hadoop and spark versions (just with default) In @ShubhamGupta29 said: > Dr.Elephant currently doesn't support Hadoop3 with ATS v2. But you can use Dr.E...

Also can't see any Spark metrics in Dr. Elephant UI. SHS version `2.3.0` I got `com.linkedin.drelephant.spark.SparkMetricsAggregator : applicationDurationMillis is negative. Skipping Metrics Aggregation:-1548370957424`

@ShubhamGupta29 Just FYI: I managed to run https://github.com/songgane/dr-elephant/tree/feature/support_spark_2.x and it worked with spark 2.3+. There is couple of tests which needs to be fixed (I skipped them). I have question:...

1. I used hadoop 2.3.0 and spark 2.1.2 https://github.com/songgane/dr-elephant/blob/feature/support_spark_2.x/compile.conf I think, I tried rebase to current master and with higher versions there were more failing tests so I stick with...

@ShubhamGupta29 every spark job, I suspect it's beacuse of this. https://github.com/songgane/dr-elephant/blame/feature/support_spark_2.x/app/org/apache/spark/deploy/history/SparkDataCollection.scala#L178 That `info.memUsed` is only available when the job is running, but I am not sure if Dr.Elephant is fetching...

@ShubhamGupta29 yep its zero. Checked `http://someHost:18081/api/v1/applications/application_1587409317223_1104/1/executors` ``` [ { "id" : "driver", "hostPort" : "someHost:37121", "isActive" : true, "rddBlocks" : 0, "memoryUsed" : 0, "diskUsed" : 0, "totalCores" : 0,...

@xglv1985 if you are using current master, you can't see any metrics from spark 2.3+ More in: https://github.com/linkedin/dr-elephant/issues/389 Check your logs there will be some parsing error. That's why i...

@ShubhamGupta29 nice, the `ShubhamGupta29/test23` works like a charm. It now even shows GC stats. `executor memory used` still not showing, but I suppose if it's not available in SHS it...