stocator icon indicating copy to clipboard operation
stocator copied to clipboard

Stocator fails with Spark Streaming

Open VidyasagarMSC opened this issue 4 years ago • 1 comments

I tried reading and writing objects to COS using Stocator with Spark streaming but it fails with the below error

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.ibm.stocator.fs.ObjectStoreFileSystem not found

Here's what I tried to overcome the error and failed

  1. Built Stocator jar file and moved the jar to /opt/spark/jars with COS access key and secret key. Fails with jackson.databinding not found exception
  2. Built SDK jar file and used the IAM api key Fails with jackson.databinding not found exception

This is what worked as an alternative to Stocator I have to fall back to S3A by moving the required jar files to /opt/spark/jars folder and use s3a:// instead of cos://

VidyasagarMSC avatar Mar 02 '21 04:03 VidyasagarMSC

Have you tried using the ready jar from https://mvnrepository.com/artifact/com.ibm.stocator/stocator ?

mrmadira avatar Apr 25 '21 14:04 mrmadira