spark-on-lambda
spark-on-lambda copied to clipboard
qubole
Apache Spark on AWS Lambda
Bumps [jetty-servlets](https://github.com/eclipse/jetty.project) from 9.2.16.v20160414 to 9.4.41.v20210516. Release notes Sourced from jetty-servlets's releases. 9.4.41.v20210516 Changelog This release resolves CVE-2021-28169 #6099 Cipher preference may break SNI if certificates have different key types...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
Bumps httpclient from 4.5.2 to 4.5.13. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a...
Bumps commons-io from 2.4 to 2.7. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a...
Bumps `fasterxml.jackson.version` from 2.6.5 to 2.12.1. Updates `jackson-databind` from 2.6.5 to 2.12.1 Commits See full diff in compare view Updates `jackson-annotations` from 2.6.5 to 2.12.1 Commits See full diff in...
Bumps [junit](https://github.com/junit-team/junit4) from 4.12 to 4.13.1. Release notes Sourced from junit's releases. JUnit 4.13.1 Please refer to the release notes for details. JUnit 4.13 Please refer to the release notes...
s3a error
in the example I have attached the following problem appears, which seems to be related to the management of shuffle in spark in the s3 context. Did I confirm that...
Compiling
Hello, I'm trying to install spark on lambda. When I run ./dev/make-distribution.sh --name spark-lambda-2.1.0 --tgz -Phive -Phadoop-2.7 -Dhadoop.version=2.6.0-qds-0.4.13 -DskipTests The Project Launcher fails and I get the following error. [ERROR]...
I have been struggling with setting up this framework on my EC2 server. I tried the best to follow the instruction of both this repo and also faromero's forked repo,...
I am using the Python example [python/ml/kmeans_example.py](https://github.com/qubole/spark-on-lambda/blob/lambda-2.1.0/examples/src/main/python/ml/kmeans_example.py). This file has a hard-coded path 'data/mllib/sample_kmeans_data.txt'. Now when I run `./bin/spark-submit --master lambda://test examples/src/main/python/ml/kmeans_example.py` under the driver folder, Spark's log shows `java.io.FileNotFoundException:...
Idea here is creating a docker image for Spark on Lambda so that Spark Driver can be spun up on either AWS ECS or on Fargate. It would be nicer...
