reading csv from s3 throwing this error using (sample-spark-script-csv-to-csv.py) file

Open saiteja2420 opened this issue 1 year ago • 3 comments

df=spark.read.option('header','true').csv(input_path) java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found please help how to resolve this issue .

Apr 10 '24 08:04 saiteja2420

@saiteja2420 What is the input_path you are providing? Is it directly a S3 URL/URI?

Aug 06 '24 07:08 ashishgupta-in

@saiteja2420 we are using S3a://

Aug 08 '24 19:08 JohnChe88

@saiteja2420 if it csv to csv , then hadoop-aws.jar file version might be the problem, have you changed any version in the Dockerfile?

Aug 13 '24 13:08 JohnChe88

No activity

Nov 05 '24 20:11 JohnChe88