streaming-benchmarks
streaming-benchmarks copied to clipboard
yahoo
Updated version of some tools in main script, since some mirrors have removed old versions
Thank you for submitting this pull request, however I do not see a valid CLA on file for you. Before we can merge this request please visit https://yahoocla.herokuapp.com/ and agree to the terms. Thanks! :smile:
Changing SCALA_BIN_VERSION to 2.11 from 2.10 causes the code to break and consequently the benchmark will not execute propoerly even on single node. The error is presumably because of of Kafka, my guess is that bumping its version as well might fix the issue.
Attaching stack trace for SPARK benchmark executed on a single machine for reference, similar for STROM and FLINK as well.
16/11/17 22:27:37 INFO SparkDeploySchedulerBackend: Granted executor ID app-20161117222737-0000/0 on hostPort 192.168.140.52:59604 with 8 cores, 1024.0 MB RAM
16/11/17 22:27:37 INFO AppClient$ClientEndpoint: Executor updated: app-20161117222737-0000/0 is now RUNNING
16/11/17 22:27:37 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at spark.benchmark.KafkaRedisAdvertisingStream$.main(AdvertisingSpark.scala:64)
at spark.benchmark.KafkaRedisAdvertisingStream.main(AdvertisingSpark.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/11/17 22:27:37 INFO SparkContext: Invoking stop() from shutdown hook
16/11/17 22:27:37 INFO SparkUI: Stopped Spark web UI at http://192.168.140.52:4040
16/11/17 22:27:37 INFO SparkDeploySchedulerBackend: Shutting down all executors
16/11/17 22:27:37 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
16/11/17 22:27:37 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/11/17 22:27:37 INFO MemoryStore: MemoryStore cleared
16/11/17 22:27:37 INFO BlockManager: BlockManager stopped
16/11/17 22:27:37 INFO BlockManagerMaster: BlockManagerMaster stopped
16/11/17 22:27:37 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/11/17 22:27:37 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/11/17 22:27:37 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/11/17 22:27:37 INFO SparkContext: Successfully stopped SparkContext
16/11/17 22:27:37 INFO ShutdownHookManager: Shutdown hook called
16/11/17 22:27:37 INFO ShutdownHookManager: Deleting directory /tmp/spark-569e7add-4144-4589-83a9-b1eb130bfbc9
16/11/17 22:27:37 INFO ShutdownHookManager: Deleting directory /tmp/spark-569e7add-4144-4589-83a9-b1eb130bfbc9/httpd-c323c571-2188-430b-9ae7-6365e1dba32e
When I run the tests with this patch I still see an empty seen.txt on spark. @ivanliu what platform are you running on that has this all work?
@revans2 I'm running the script on Mac OS (10.11.2).
Without this patch, some components cannot get installed with stream-bench.sh SETUP. I tried STORM_TEST, it seemed to work, but SPARK_TEST gave me empty seen.txt.
@ivanliu that is the real problem we can run the benchmark but it does not really do anything and we are still trying to debug why that is.