bigquery4s icon indicating copy to clipboard operation
bigquery4s copied to clipboard

Published 20 hours ago • verified publisher icon seratch

add fromServiceAccountJsonInputStream

Open vijaykramesh opened this issue 7 years ago • 2 comments

similar to the BigQuery.fromServiceAccountJson (that expects a file path), this new method allows you to pass in an InputStream. We have a use-case where all configuration is happening via ENV vars, and I'd prefer to not have to write the config to the local (container) filesystem.

I also added a test for both of the service account json methods I have added. Same as the existing test, it requires you change yourOwnProjectId and to have a file existing at $HOME/.bigquery/service_account.json:

> testOnly *ServiceAccountUsageExamplesSpec
[info] Formatting 1 Scala source {file:/Users/vramesh/oss/bigquery4s/}root(test) ...
[info] Formatting 1 Scala source {file:/Users/vramesh/oss/bigquery4s/}root(compile) ...
[info] Compiling 1 Scala source to /Users/vramesh/oss/bigquery4s/target/scala-2.12/classes...
[info] Compiling 1 Scala source to /Users/vramesh/oss/bigquery4s/target/scala-2.12/test-classes...
[info] ServiceAccountUsageExamplesSpec:
[info] Simple query example with publicdata using service account json file
Oct 23, 2017 10:48:10 AM com.google.api.client.googleapis.services.AbstractGoogleClient <init>
WARNING: Application name is not set. Call Builder#setApplicationName.
2017-10-23 10:48:11,608 INFO bigquery4s.UsageExamplesSpec WrappedDatasets({"datasetReference":{"datasetId":"samples","projectId":"publicdata"},"id":"publicdata:samples","kind":"bigquery#dataset"})
2017-10-23 10:48:12,719 INFO bigquery4s.UsageExamplesSpec JobId: JobId(ProjectId(REDACTED),job_-s7T-GAUeKvcYxyH9tHNAxp24KWf)
2017-10-23 10:48:12,991 INFO bigquery4s.UsageExamplesSpec Job: WrappedCompletedJob({"configuration":{"query":{"createDisposition":"CREATE_IF_NEEDED","destinationTable":{"datasetId":"_05fb90c360272d1beef3822267407d6db9668147","projectId":"REDACTED","tableId":"anon3fc6a91182188a91987aa4f1347569a2e9c8631b"},"query":"\n        SELECT weight_pounds,state,year,gestation_weeks FROM publicdata:samples.natality\n        ORDER BY weight_pounds DESC LIMIT 100;\n      ","writeDisposition":"WRITE_TRUNCATE"}},"etag":"\"SdUekaVqp4U9n5pPfq5AV4uIdyk/YuWvQ10O8vfZNFDTfivjFFZsmco\"","id":"REDACTED:job_-s7T-GAUeKvcYxyH9tHNAxp24KWf","jobReference":{"jobId":"job_-s7T-GAUeKvcYxyH9tHNAxp24KWf","projectId":"REDACTED"},"kind":"bigquery#job","selfLink":"https://www.googleapis.com/bigquery/v2/projects/REDACTED/jobs/job_-s7T-GAUeKvcYxyH9tHNAxp24KWf","statistics":{"creationTime":"1508780891716","endTime":"1508780892405","query":{"cacheHit":true,"totalBytesBilled":"0","totalBytesProcessed":"0"},"startTime":"1508780892210","totalBytesProcessed":"0"},"status":{"state":"DONE"},"user_email":"[email protected]"})
2017-10-23 10:48:13,279 INFO bigquery4s.UsageExamplesSpec
*** QueryResult ***

weight_pounds,state,year,gestation_weeks
18.0007436923,null,2008,null
18.0007436923,null,2008,32
18.0007436923,null,2007,38
18.0007436923,null,2006,null
18.0007436923,KY,2004,38
18.0007436923,null,2005,null
18.0007436923,null,2007,null
18.0007436923,null,2005,40
18.0007436923,null,2006,37
18.0007436923,null,2005,42
18.0007436923,KY,2004,47
18.0007436923,KY,2004,41
18.0007436923,TX,1969,null
18.0007436923,KY,2004,39
18.0007436923,KY,2004,47
18.0007436923,null,2006,39
18.0007436923,null,2005,42
18.0007436923,KY,2004,47
18.0007436923,null,2005,35
18.0007436923,KY,2004,47

[info] - runs
[info] Simple query example with publicdata using service account json string
Oct 23, 2017 10:48:13 AM com.google.api.client.googleapis.services.AbstractGoogleClient <init>
WARNING: Application name is not set. Call Builder#setApplicationName.
2017-10-23 10:48:13,582 INFO bigquery4s.UsageExamplesSpec WrappedDatasets({"datasetReference":{"datasetId":"samples","projectId":"publicdata"},"id":"publicdata:samples","kind":"bigquery#dataset"})
2017-10-23 10:48:13,939 INFO bigquery4s.UsageExamplesSpec JobId: JobId(ProjectId(REDACTED),job_eELt2uPdtORwxOIBApXFsa7mVyEn)
2017-10-23 10:48:14,123 INFO bigquery4s.UsageExamplesSpec Job: WrappedCompletedJob({"configuration":{"query":{"createDisposition":"CREATE_IF_NEEDED","destinationTable":{"datasetId":"_05fb90c360272d1beef3822267407d6db9668147","projectId":"REDACTED","tableId":"anon3fc6a91182188a91987aa4f1347569a2e9c8631b"},"query":"\n          SELECT weight_pounds,state,year,gestation_weeks FROM publicdata:samples.natality\n          ORDER BY weight_pounds DESC LIMIT 100;\n        ","writeDisposition":"WRITE_TRUNCATE"}},"etag":"\"SdUekaVqp4U9n5pPfq5AV4uIdyk/ePu_0fQijyMGuQmEJ3dUSmRHgao\"","id":"REDACTED:job_eELt2uPdtORwxOIBApXFsa7mVyEn","jobReference":{"jobId":"job_eELt2uPdtORwxOIBApXFsa7mVyEn","projectId":"REDACTED"},"kind":"bigquery#job","selfLink":"https://www.googleapis.com/bigquery/v2/projects/REDACTED/jobs/job_eELt2uPdtORwxOIBApXFsa7mVyEn","statistics":{"creationTime":"1508780893661","endTime":"1508780893800","query":{"cacheHit":true,"totalBytesBilled":"0","totalBytesProcessed":"0"},"startTime":"1508780893772","totalBytesProcessed":"0"},"status":{"state":"DONE"},"user_email":"[email protected]"})
2017-10-23 10:48:14,397 INFO bigquery4s.UsageExamplesSpec
*** QueryResult ***

weight_pounds,state,year,gestation_weeks
18.0007436923,null,2008,null
18.0007436923,null,2008,32
18.0007436923,null,2007,38
18.0007436923,null,2006,null
18.0007436923,KY,2004,38
18.0007436923,null,2005,null
18.0007436923,null,2007,null
18.0007436923,null,2005,40
18.0007436923,null,2006,37
18.0007436923,null,2005,42
18.0007436923,KY,2004,47
18.0007436923,KY,2004,41
18.0007436923,TX,1969,null
18.0007436923,KY,2004,39
18.0007436923,KY,2004,47
18.0007436923,null,2006,39
18.0007436923,null,2005,42
18.0007436923,KY,2004,47
18.0007436923,null,2005,35
18.0007436923,KY,2004,47

[info] - runs
[info] Run completed in 4 seconds, 148 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

vijaykramesh avatar Oct 23 '17 17:10 vijaykramesh

@seratch any thoughts?

vijaykramesh avatar Oct 27 '17 22:10 vijaykramesh

hey @seratch I was on holiday and missed your comments. I switched things back a bit so the existing fromServiceAccountJson function opens and closes its own input stream (versus that function calling to the new fromServiceAccountJsonInputStream). It makes the code slightly less dry but I agree it makes the opening and closing of the resource much clearer.

vijaykramesh avatar Nov 21 '17 01:11 vijaykramesh