stocator
stocator copied to clipboard
Stocator fails with Spark Streaming
I tried reading and writing objects to COS using Stocator with Spark streaming but it fails with the below error
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.ibm.stocator.fs.ObjectStoreFileSystem not found
Here's what I tried to overcome the error and failed
- Built Stocator jar file and moved the jar to
/opt/spark/jarswith COS access key and secret key. Fails with jackson.databinding not found exception - Built SDK jar file and used the IAM api key Fails with jackson.databinding not found exception
This is what worked as an alternative to Stocator
I have to fall back to S3A by moving the required jar files to /opt/spark/jars folder and use s3a:// instead of cos://
Have you tried using the ready jar from https://mvnrepository.com/artifact/com.ibm.stocator/stocator ?