scala-spark-tutorial icon indicating copy to clipboard operation
scala-spark-tutorial copied to clipboard
verified publisher icon jleetutorial

Spark job is failing in writing output to local file system(saveAsTextFile())

Open raja550raja opened this issue 7 years ago • 3 comments

HI, any one help me in this my Spark job is failing in writing output to local file system(saveAsTextFile())

please find the below error and help in resolving thsi.

ExitCodeException exitCode=-1073741515: at org.apache.hadoop.util.Shell.runCommand(Shell.java:575) at org.apache.hadoop.util.Shell.run(Shell.java:478) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:766) at org.apache.hadoop.util.Shell.execCommand(Shell.java:859) at org.apache.hadoop.util.Shell.execCommand(Shell.java:842) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:661) at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:501) at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:482) at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:498) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:467) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:433) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:801) at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123) at org.apache.spark.internal.io.SparkHadoopWriter.open(SparkHadoopWriter.scala:89) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18/04/27 07:06:59 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3) ExitCodeException exitCode=-1073741515: at org.apache.hadoop.util.Shell.runCommand(Shell.java:575) at org.apache.hadoop.util.Shell.run(Shell.java:478) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:766) at org.apache.hadoop.util.Shell.execCommand(Shell.java:859) at org.apache.hadoop.util.Shell.execCommand(Shell.java:842) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:661) at org.apache.hadoop.fs.ChecksumFileSystem$1.apply(ChecksumFileSystem.java:501) at org.apache.hadoop.fs.ChecksumFileSystem$FsOperation.run(ChecksumFileSystem.java:482) at org.apache.hadoop.fs.ChecksumFileSystem.setPermission(ChecksumFileSystem.java:498) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:467) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:433) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:801) at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123) at org.apache.spark.internal.io.SparkHadoopWriter.open(SparkHadoopWriter.scala:89) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1125) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18/04/27 07:06:59 WARN TaskSetManager: Lost task 1.0 in stage 2.0 (TID 4, localhost, executor driver): ExitCodeException exitCode=-1073741515:

raja550raja avatar Apr 29 '18 15:04 raja550raja

Did you find any solution?

anikakelhanka avatar Aug 08 '18 07:08 anikakelhanka

How to resolve it

xiyuan-lpy avatar Nov 23 '18 06:11 xiyuan-lpy

i had an spark issue with same code. try this: https://answers.microsoft.com/en-us/insider/forum/insider_wintp-insider_repair/how-do-i-fix-this-error-msvcp100dll-is-missing/c167d686-044e-44ab-8e8f-968fac9525c5?auth=1

Here's the topic where i found it in https://stackoverflow.com/questions/45947375/why-does-starting-a-streaming-query-lead-to-exitcodeexception-exitcode-1073741

Dan-The-Lion avatar Oct 25 '19 11:10 Dan-The-Lion