spark-perf icon indicating copy to clipboard operation
spark-perf copied to clipboard

Use Long instead of Int for numeric options

Open JoshRosen opened this issue 10 years ago • 1 comments

We should use Long instead of Int for numeric options in order to support larger values (e.g. huge numbers of records), and verify that the data generators work for huge datasets.

The current code fails in the options parsing phase if it attempts to parse numbers bigger than IntMax:

Exception in thread "main" joptsimple.OptionArgumentConversionException: Cannot convert argument '4000000000' of option ['num-records'] to class java.lang.Integer
  at joptsimple.AbstractOptionSpec.convertWith(AbstractOptionSpec.java:94)
  at joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:276)
  at joptsimple.OptionSet.valuesOf(OptionSet.java:222)
  at joptsimple.OptionSet.valueOf(OptionSet.java:171)
  at joptsimple.OptionSet.valueOf(OptionSet.java:152)
  at spark.perf.KVDataTest.createInputData(KVDataTest.scala:62)
  at spark.perf.TestRunner$.main(TestRunner.scala:29)
  at spark.perf.TestRunner.main(TestRunner.scala)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:606)
  at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:314)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:73)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: joptsimple.internal.ReflectionException: java.lang.NumberFormatException: For input string: "4000000000"
  at joptsimple.internal.Reflection.reflectionException(Reflection.java:140)
  at joptsimple.internal.Reflection.invoke(Reflection.java:122)
  at joptsimple.internal.MethodInvokingValueConverter.convert(MethodInvokingValueConverter.java:48)
  at joptsimple.internal.Reflection.convertWith(Reflection.java:128)
  at joptsimple.AbstractOptionSpec.convertWith(AbstractOptionSpec.java:91)
  ... 14 more

JoshRosen avatar Aug 13 '14 22:08 JoshRosen

Partially resolved by #38.

nchammas avatar Feb 17 '15 18:02 nchammas