NetworkAnomalyDetection icon indicating copy to clipboard operation
NetworkAnomalyDetection copied to clipboard

runtime issue : Field "label" does not exist. --> kmeansOneHotEncoder

Open pgibert opened this issue 5 years ago • 0 comments

hello Thanks nice job done ! but i got a runtime issue with the schema kmeansSimple works well but kmeansOneHotEncoder exits :

Exception in thread "main" java.lang.IllegalArgumentException: Field "label" does not exist. Available fields: duration, protocol_type, service, flag, src_bytes, dst_bytes, land, wrong_fragment, urgent, hot, num_failed_logins, logged_in, num_compromised, root_shell, su_attempted, num_root, num_file_creations, num_shells, num_access_files, num_outbound_cmds, is_host_login, is_guest_login, count, srv_count, serror_rate, srv_serror_rate, rerror_rate, srv_rerror_rate, same_srv_rate, diff_srv_rate, srv_diff_host_rate, dst_host_count, dst_host_srv_count, dst_host_same_srv_rate, dst_host_diff_srv_rate, dst_host_same_src_port_rate, dst_host_srv_diff_host_rate, dst_host_serror_rate, dst_host_srv_serror_rate, dst_host_rerror_rate, dst_host_srv_rerror_rate at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:267) at org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:267) at scala.collection.MapLike$class.getOrElse(MapLike.scala:128) at scala.collection.AbstractMap.getOrElse(Map.scala:59) at org.apache.spark.sql.types.StructType.apply(StructType.scala:266) at org.apache.spark.ml.feature.StringIndexerBase$class.validateAndTransformSchema(StringIndexer.scala:85) at org.apache.spark.ml.feature.StringIndexer.validateAndTransformSchema(StringIndexer.scala:109) at org.apache.spark.ml.feature.StringIndexer.transformSchema(StringIndexer.scala:152) at org.apache.spark.ml.Pipeline$$anonfun$transformSchema$4.apply(Pipeline.scala:184) at org.apache.spark.ml.Pipeline$$anonfun$transformSchema$4.apply(Pipeline.scala:184) at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57) at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66) at scala.collection.mutable.ArrayOps$ofRef.foldLeft(ArrayOps.scala:186) at org.apache.spark.ml.Pipeline.transformSchema(Pipeline.scala:184) at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74) at org.apache.spark.ml.Pipeline.fit(Pipeline.scala:136) at com.spark_batch_sample.examples.NetworkAnomalyDetection$RunClustering.kmeansOneHotEncoder(NetworkAnomalyDetection.scala:407)

pgibert avatar Nov 20 '19 07:11 pgibert