spark-on-aws-lambda
spark-on-aws-lambda copied to clipboard
aws-samples
Spark runtime on AWS Lambda
Hi Team, I have been trying to run Hudi job on Lambda, the function is getting properly invoked and the files are read fine as well. But the issue seems...
Please add the region and configuration .config("spark.hadoop.fs.s3a.endpoint", "s3.us-gov- region-1.amazonaws.com") \ Based on Gov vs Commerical.
*Issue #, if available:* *Description of changes:* Skipping ssl verification for yum repositories on companies that enforces proxys, which can cause SSL certificate problem By submitting this pull request, I...
df=spark.read.option('header','true').csv(input_path) java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found please help how to resolve this issue .
Amazon Linux 2023 version works has dependency issues Pyspark=3.3.0 is not installing properly but works with Amazon Linux 2
AWS lambda has a limit of 1024 open File descriptors that leads to task result loss failure if you are merging data in the target tables for building a idempotent...
Identify key configuration for Spark running local on a container. Adjust the JVM spin up cost ,maximize the memory capacity in AWS Lambda and reduce the container size. - Best...
Identify the reason why Lambda is not able to write the HUDI marker and timeline serve
Hellooo @JohnChe88 I'm a beginner with ECS, and then I noticed in the previous issue that you mentioned that you're testing this docker image on ECS, Is there any tutorial...
…upport - Add comprehensive Iceberg functions library (libs/glue_functions/iceberg_glue_functions.py) - Implement production-ready Lambda handlers for Iceberg table operations - Add time travel queries and metadata access capabilities - Include advanced features:...