SNOW-594882: Getting 403 while downloading chunk from Azure Blob Storage and Google Storage

[sagar-celonaut]

We are encountering 403 errors while extracting data from snowflake using JDBC driver. The errors are happening for both Azure Blob storage as well as Google Storage. Could someone please assist in resolving this? Currently, we are using the JDBC driver version: 3.13.5. Could a driver update help here and which version?

Following are some error logs:

Errors from Azure Blob Storage

`downloader encountered error: Max retry reached for the download of #chunk257 (Total chunks: 880) retry=10, error=net.snowflake.client.jdbc.SnowflakeSQLException: JDBC driver encountered communication error. Message: Error encountered when downloading a result chunk: HTTP status=403. at net.snowflake.client.jdbc.DefaultResultStreamProvider.getInputStream(DefaultResultStreamProvider.java:65) at net.snowflake.client.jdbc.SnowflakeChunkDownloader$2.call(SnowflakeChunkDownloader.java:867) at net.snowflake.client.jdbc.SnowflakeChunkDownloader$2.call(SnowflakeChunkDownloader.java:781) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source)

Error response: HTTP Response code: 403, request: GET https://xxx.blob.core.windows.net/results/01a42f59-0b03-061b-0000-62113985e2aa_0%2Fmain%2Fdata_2_2_10?sv=2020-08-04&spr=https&se=2022-05-11T08%3A02%3A23Z&sr=b&sp=r&sig=****&rsce=gzip HTTP/1.1

Error fetching chunk from: https://xxx.blob.core.windows.net/results/01a42f59-0b03-061b-0000-62113985e2aa_0%2Fmain%2Fdata_2_2_10?sv=2020-08-04&spr=https&se=2022-05-11T08%3A02%3A23Z&sr=b&sp=r&sig=****&rsce=gzip

Response status line reason: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Response content: <Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. RequestId:0d6b8d1f-101e-000a-1c0d-658b73000000 Time:2022-05-11T08:06:12.2883178Z</Message><AuthenticationErrorDetail>Signed expiry time [Wed, 11 May 2022 08:02:23 GMT] must be after signed start time [Wed, 11 May 2022 08:06:12 GMT]</AuthenticationErrorDetail></Error>

net.snowflake.client.jdbc.SnowflakeSQLLoggedException: JDBC driver internal error: Max retry reached for the download of #chunk257 (Total chunks: 880) retry=10, error=net.snowflake.client.jdbc.SnowflakeSQLException: JDBC driver encountered communication error. Message: Error encountered when downloading a result chunk: HTTP status=403. at net.snowflake.client.jdbc.DefaultResultStreamProvider.getInputStream(DefaultResultStreamProvider.java:65) at net.snowflake.client.jdbc.SnowflakeChunkDownloader$2.call(SnowflakeChunkDownloader.java:867) at net.snowflake.client.jdbc.SnowflakeChunkDownloader$2.call(SnowflakeChunkDownloader.java:781) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.base/java.lang.Thread.run(Unknown Source) at net.snowflake.client.jdbc.SnowflakeChunkDownloader.getNextChunkToConsume(SnowflakeChunkDownloader.java:519) at net.snowflake.client.core.SFArrowResultSet.fetchNextRowUnsorted(SFArrowResultSet.java:233) at net.snowflake.client.core.SFArrowResultSet.fetchNextRow(SFArrowResultSet.java:210) at net.snowflake.client.core.SFArrowResultSet.next(SFArrowResultSet.java:346) at net.snowflake.client.jdbc.SnowflakeResultSetV1.next(SnowflakeResultSetV1.java:99) at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:1160) at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:1160)

`

######################################

Errors from Google Storage

`Response content: <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>GOOG4-RSA-SHA256 20220524T170758Z 20220524/auto/storage/goog4_request 5fdb689a58c057967dd78a3d81eed3560a87854cbf0f97bbb7c2aa916698c591</StringToSign><UnsignedSensitiveHeaders>x-goog-encryption-key-sha256</UnsignedSensitiveHeaders><CanonicalRequest>GET /results/01a47c03-0000-8ddb-0000-0399592d52f2_0/main/data_0_0_2 X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx-stage%40us-central1-stage1-6e7d.iam.gserviceaccount.com%2F20220524%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220524T170758Z&X-Goog-Expires=21600&X-Goog-SignedHeaders=host&retryCount=3 host:xxx-stage.storage.googleapis.com

host UNSIGNED-PAYLOAD</CanonicalRequest></Error>

Response status line reason: Forbidden

Error fetching chunk from: https://xxx-stage.storage.googleapis.com/results/01a47c03-0000-8ddb-0000-0399592d52f2_0/main/data_0_0_2?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx-stage%40us-central1-stage1-6e7d.iam.gserviceaccount.com%2F20220524%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220524T170758Z&X-Goog-Expires=21600&X-Goog-SignedHeaders=host&X-Goog-Signature=****

Response content: <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>GOOG4-RSA-SHA256 20220524T170758Z 20220524/auto/storage/goog4_request 5fdb689a58c057967dd78a3d81eed3560a87854cbf0f97bbb7c2aa916698c591</StringToSign><UnsignedSensitiveHeaders>x-goog-encryption-key-sha256</UnsignedSensitiveHeaders><CanonicalRequest>GET /results/01a47c03-0000-8ddb-0000-0399592d52f2_0/main/data_0_0_2 X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=xxx-stage%40us-central1-stage1-6e7d.iam.gserviceaccount.com%2F20220524%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20220524T170758Z&X-Goog-Expires=21600&X-Goog-SignedHeaders=host&retryCount=3 host:xxx-stage.storage.googleapis.com

host UNSIGNED-PAYLOAD</CanonicalRequest></Error>`

[sfc-gh-wshangguan]

Hi @sagar-celonaut , could you please try the latest JDBC version, e.g. 3.13.19 at the moment, and see is the problem is still there? We've fixed a scoped URL bug recently which looks similar to what you met with Google Storage. For Azure issue, it's not that clear whether related, but using the latest driver can also help to clear the path to some extent. Thanks!

[sagar-celonaut]

Hi @sfc-gh-wshangguan, Thank you for the reply! I will try with the latest driver and let you know here if it helps.

For Azure issue, I found the driver hangs at different times like:

1. For a query that completes execution on Snowflake instance in 1min 37secs, the driver took ca. 3 hours to respond with the result set.
2. Also, during chunk download it randomly hangs where it takes 8mins to 140 mins to respond whereas sometimes it completes in a few seconds.

[sagar-celonaut]

Driver version: 3.13.20 Max retry still occurs but the error object is null. ` net.snowflake.client.jdbc.SnowflakeSQLLoggedException: JDBC driver internal error: Max retry reached for the download of #chunk187 (Total chunks: 472) retry=10, error=null.

at net.snowflake.client.jdbc.SnowflakeChunkDownloader.getNextChunkToConsume(SnowflakeChunkDownloader.java:580)

at net.snowflake.client.core.SFArrowResultSet.fetchNextRowUnsorted(SFArrowResultSet.java:233)

at net.snowflake.client.core.SFArrowResultSet.fetchNextRow(SFArrowResultSet.java:210)

at net.snowflake.client.core.SFArrowResultSet.next(SFArrowResultSet.java:346)

at net.snowflake.client.jdbc.SnowflakeResultSetV1.next(SnowflakeResultSetV1.java:99)

at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:1160)

at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:1160)`

[sfc-gh-wshangguan]

Hi @sagar-celonaut , can you please to provide JDBC logs? That would be very helpful for us to debug the problem.

[sfc-gh-igarish]

To clean up and re-prioritize more pressing bugs and feature requests we are closing all issues older than 6 months as of March 1, 2023. If there are any issues or feature requests that you would like us to address, please create them according to the new templates we have created. For urgent issues, opening a support case with this link Snowflake Community is the fastest way to get a response.