pg2bq

pg2bq copied to clipboard

Hello, I am trying your application pg2bq, however I am getting the following error

2018-02-12 19:19:40,050 [main] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-02-12 19:19:41,551 [main] INFO  com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase - GHFS version: 1.6.1-hadoop2
Exception in thread "main" com.google.cloud.bigquery.BigQueryException: Insufficient Permission
       at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:84)
       at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getDataset(HttpBigQueryRpc.java:95)
       at com.google.cloud.bigquery.BigQueryImpl$4.call(BigQueryImpl.java:230)
       at com.google.cloud.bigquery.BigQueryImpl$4.call(BigQueryImpl.java:227)
       at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
       at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
       at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
       at com.google.cloud.bigquery.BigQueryImpl.getDataset(BigQueryImpl.java:227)
       at com.google.cloud.bigquery.BigQueryImpl.getDataset(BigQueryImpl.java:218)
       at com.powerspace.pg2bq.BigQueryImporter.getOrCreateDataset(BigQueryImporter.scala:42)
       at com.powerspace.pg2bq.BigQueryImporter.<init>(BigQueryImporter.scala:13)
       at com.powerspace.pg2bq.Worker$.start(Worker.scala:17)
       at com.powerspace.pg2bq.Main$.delayedEndpoint$com$powerspace$pg2bq$Main$1(Main.scala:15)
       at com.powerspace.pg2bq.Main$delayedInit$body.apply(Main.scala:9)
       at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
       at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
       at scala.App$$anonfun$main$1.apply(App.scala:76)
       at scala.App$$anonfun$main$1.apply(App.scala:76)
       at scala.collection.immutable.List.foreach(List.scala:392)
       at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
       at scala.App$class.main(App.scala:76)
       at com.powerspace.pg2bq.Main$.main(Main.scala:9)
       at com.powerspace.pg2bq.Main.main(Main.scala)

The service account has admin permitions on BigQuery and Storage, so I don't know what it is causing the error If I put a .json file that does not exist on configuration.conf I get the same error Could you please help? Thank you :)

Hey @jast1000 ! Thanks for using pg2bq.

Did you set the GAC ? I just ran it using:

GOOGLE_APPLICATION_CREDENTIALS=token.json bin/pg2bq -Dconfig.file=pg2bq.conf

GOOGLE_APPLICATION_CREDENTIALS is used to use the proper service account from the key (instead of relying on the default gcloud environment). It contains the same value as gcloud.service-account-key specified in your config file.

Does it work for you with this?

Hello @chtefi That did the trick!, thank you very much.