Error 400 while loading data to Google Bigquery by SingleBatchWriter

[tungntt]

Dear wepay Team,

I am trying to test kafka-connect-bigquery with confluent in loading data to GG Bigquery. However, after starting and creating table automatically, it meet an issue

Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "The destination table's partition demo_comment_topic$2018427 is outside the allowed bounds. You can only stream to partitions within 0 days in the past and 0 days in the future relative to the current date.",
    "reason" : "invalid"
  } ],
  "message" : "The destination table's partition demo_comment_topic$2018427 is outside the allowed bounds. You can only stream to partitions within 0 days in the past and 0 days in the future relative to the current date.",
  "status" : "INVALID_ARGUMENT"
}
        at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
        at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:321)
        at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1065)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:419)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:352)
        at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:469)
        at com.google.cloud.bigquery.spi.DefaultBigQueryRpc.insertAll(DefaultBigQueryRpc.java:283)
        ... 10 more

As far as I know, kafka-connect-bigquery is using the google-cloud-api out-of updated from 2016 (gcloud-java:0.2.7) rather than current libraries (google-cloud:0.45.0-alpha). May be root core of issue? Please help me to review this issue!

Thanks, Tung Nguyen

[criccomini]

Hmm. We have not observed this, and have created tables successfully a week or two ago. Have you tried upgrading the library locally to see if it fixes the problem? If so, we can definitely merge a PR to fix it.

[tungntt]

Hi @criccomini ,

I have figure out this issue. The root cause is not relevant with version of google API. The main account is getPartitionedTableName method in TopicToTableResolver.class. Due to declaring wrong partitioned table name. It must be [PREFIX]_YYYYMMDD rather than [PREFIX]_YYYYMD as you can see in google message. I have some fixes like that:

    String baseTableName = baseTableId.table();
    sb.append(baseTableName);
    sb.append("$");
    
    DateTimeFormatter formatter = DateTimeFormatter.ofPattern("YYYYMMdd");
//    int year = localDate.getYear();
//    int month = localDate.getMonthValue();
//    int day = localDate.getDayOfMonth();
//    sb.append(year);
//    sb.append(month);
//    sb.append(day);
    sb.append(localDate.now().format(formatter));
    String partitionedTableName = sb.toString();

Please review may it be suitable for PR?

[criccomini]

Yes, please submit a PR!

cc @mtagle

[mtagle]

This has been fixed already, maybe we just haven't put out a new version since the fix was implemented?: https://github.com/wepay/kafka-connect-bigquery/pull/57

[criccomini]

@tungntt we just released 1.1.0. I suggest you try that version.