hadoop-connectors icon indicating copy to clipboard operation
hadoop-connectors copied to clipboard

Handle partial write update from GCS

Open mayanks opened this issue 2 years ago • 2 comments

Connector supports resuming upload only from MAX_BYTES_PER_MESSAGE boundary. If GCS reports a committed offset which is not a multiple of this number, then we cannot resume.

mayanks avatar May 12 '22 06:05 mayanks

/gcbrun

mayanks avatar May 12 '22 06:05 mayanks

Codecov Report

Merging #781 (71ecf70) into master (d465885) will decrease coverage by 18.33%. The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master     #781       +/-   ##
=============================================
- Coverage     81.23%   62.89%   -18.34%     
+ Complexity     1978     1446      -532     
=============================================
  Files           133      133               
  Lines          8818     8822        +4     
  Branches       1025     1026        +1     
=============================================
- Hits           7163     5549     -1614     
- Misses         1240     2735     +1495     
- Partials        415      538      +123     
Flag Coverage Δ
integrationtest 62.89% <0.00%> (-0.15%) :arrow_down:
unittest ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...doop/gcsio/GoogleCloudStorageGrpcWriteChannel.java 64.46% <0.00%> (-15.09%) :arrow_down:
...a/com/google/cloud/hadoop/util/AccessBoundary.java 0.00% <0.00%> (-100.00%) :arrow_down:
...om/google/cloud/hadoop/util/IoExceptionHelper.java 0.00% <0.00%> (-100.00%) :arrow_down:
...le/cloud/hadoop/io/bigquery/ShardedInputSplit.java 0.00% <0.00%> (-100.00%) :arrow_down:
...d/hadoop/gcsio/testing/GcsItemInfoTestBuilder.java 0.00% <0.00%> (-100.00%) :arrow_down:
...bigquery/output/FederatedBigQueryOutputFormat.java 0.00% <0.00%> (-100.00%) :arrow_down:
...query/output/FederatedBigQueryOutputCommitter.java 0.00% <0.00%> (-100.00%) :arrow_down:
...gcsio/StorageRequestToAccessBoundaryConverter.java 0.00% <0.00%> (-98.77%) :arrow_down:
...d/hadoop/util/testing/MockHttpTransportHelper.java 0.00% <0.00%> (-96.67%) :arrow_down:
...adoop/io/bigquery/DynamicFileListRecordReader.java 0.00% <0.00%> (-93.60%) :arrow_down:
... and 70 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d465885...71ecf70. Read the comment docs.

codecov[bot] avatar May 12 '22 07:05 codecov[bot]