spark-bigquery-connector icon indicating copy to clipboard operation
spark-bigquery-connector copied to clipboard

Published 20 hours ago • verified publisher icon GoogleCloudDataproc

parallelism setting in connector

Open cwang9208 opened this issue 2 years ago • 0 comments

Hi @davidrabinowitz , I'm wondering why the maxStreamCount is set to such a big number (DEFAULT_MAX_PARALLELISM = 20_000)? It seems BigQuery Storage API can only support a maximum of 1,000 parallel read streams. How this maxStreamCount should be exactly set :(

Thanks for your time in advance.

cwang9208 avatar Sep 13 '22 15:09 cwang9208