Unable to use oci hdfs connector in yarn cluster mode

[KanaSukita]

trafficstars

Hello,

I am trying to to access oci data in a spark application in yarn cluster mode. I was able to access the data in a spark localhost mode, but when I switch to the cluster mode, there is an issue accessing the private pem file, and the log is as below. I double check that the file exist and the permission for the file is 777, otherwise I would not be able to access the data in localhost mode.

Please help me solve this problem.

Caused by: java.lang.IllegalArgumentException: Could not find private key: /root/.oci/private.pem
        at com.oracle.bmc.auth.SimplePrivateKeySupplier.get(SimplePrivateKeySupplier.java:30)
        at com.oracle.bmc.auth.SimplePrivateKeySupplier.get(SimplePrivateKeySupplier.java:16)
        at com.oracle.bmc.auth.SimpleAuthenticationDetailsProvider.getPrivateKey(SimpleAuthenticationDetailsProvider.java:68)
        at com.oracle.bmc.http.signing.internal.DefaultRequestSignerFactory.createKeySupplier(DefaultRequestSignerFactory.java:109)
        at com.oracle.bmc.http.signing.internal.DefaultRequestSignerFactory.createRequestSigner(DefaultRequestSignerFactory.java:53)
        at com.oracle.bmc.http.internal.BaseClient.<init>(BaseClient.java:96)
        at com.oracle.bmc.http.internal.BaseSyncClient.<init>(BaseSyncClient.java:36)
        at com.oracle.bmc.objectstorage.ObjectStorageClient.<init>(ObjectStorageClient.java:49)
        at com.oracle.bmc.objectstorage.ObjectStorageClient$Builder.build(ObjectStorageClient.java:143)
        at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.buildClient(BmcDataStoreFactory.java:632)
        at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createClient(BmcDataStoreFactory.java:338)
        at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createDataStore(BmcDataStoreFactory.java:147)
        at com.oracle.bmc.hdfs.BmcFilesystemImpl.initialize(BmcFilesystem.java:435)
        at com.oracle.bmc.hdfs.BmcFilesystem$1.load(BmcFilesystem.java:116)
        at com.oracle.bmc.hdfs.BmcFilesystem$1.load(BmcFilesystem.java:111)
        at shaded.oracle.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3570)
        at shaded.oracle.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2312)
        at shaded.oracle.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2189)
        at shaded.oracle.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2079)
        ... 47 more
Caused by: java.io.FileNotFoundException: /root/.oci/private.pem (Permission denied)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        at com.oracle.bmc.auth.SimplePrivateKeySupplier.get(SimplePrivateKeySupplier.java:28)
        ... 65 more