spark-perf
spark-perf copied to clipboard
Use Long instead of Int for numeric options
We should use Long
instead of Int
for numeric options in order to support larger values (e.g. huge numbers of records), and verify that the data generators work for huge datasets.
The current code fails in the options parsing phase if it attempts to parse numbers bigger than IntMax:
Exception in thread "main" joptsimple.OptionArgumentConversionException: Cannot convert argument '4000000000' of option ['num-records'] to class java.lang.Integer
at joptsimple.AbstractOptionSpec.convertWith(AbstractOptionSpec.java:94)
at joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:276)
at joptsimple.OptionSet.valuesOf(OptionSet.java:222)
at joptsimple.OptionSet.valueOf(OptionSet.java:171)
at joptsimple.OptionSet.valueOf(OptionSet.java:152)
at spark.perf.KVDataTest.createInputData(KVDataTest.scala:62)
at spark.perf.TestRunner$.main(TestRunner.scala:29)
at spark.perf.TestRunner.main(TestRunner.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:314)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: joptsimple.internal.ReflectionException: java.lang.NumberFormatException: For input string: "4000000000"
at joptsimple.internal.Reflection.reflectionException(Reflection.java:140)
at joptsimple.internal.Reflection.invoke(Reflection.java:122)
at joptsimple.internal.MethodInvokingValueConverter.convert(MethodInvokingValueConverter.java:48)
at joptsimple.internal.Reflection.convertWith(Reflection.java:128)
at joptsimple.AbstractOptionSpec.convertWith(AbstractOptionSpec.java:91)
... 14 more
Partially resolved by #38.