seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

The maxcompute source does not support the conversion of date type

Open lucas666666 opened this issue 2 years ago • 1 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

The maxcompute source does not support the conversion of date type ,when batch job running 。

SeaTunnel Version

v2.3.0

SeaTunnel Config

env {
  spark.app.name = "seatunnel"
  spark.executor.instances = 2
  spark.executor.cores = 1
  spark.executor.memory = "1g"
  // 这个配置必需填写
   #spark.sql.catalogImplementation = "hive"
}
source {
  Maxcompute {
    accessId="accessId"
    accesskey="accesskey"
    endpoint="http://service.cn-shanghai.maxcompute.aliyun.com/api"
    project="noah_ods"
    table_name="bak_go_goal_new_t_index_info"
    #partition_spec="<your partition spec>"
    #split_row = 10000
  }
}


sink {
  Console {}
}

Running Command

./bin/start-seatunnel-spark-connector-v2.sh --config config/maxcompute2rds1.conf -e client -m 'local[2]'

Error Exception

"ERROR ParallelBatchPartitionReader: BatchPartitionReader execute failed.
org.apache.seatunnel.connectors.seatunnel.maxcompute.exception.MaxcomputeConnectorException: ErrorCode:[COMMON-12], ErrorDescription:[Source reader operation failed, such as (open, close) etc...] - java.lang.ClassCastException: java.util.Date cannot be cast to java.lang.Long
  at org.apache.seatunnel.connectors.seatunnel.maxcompute.util.MaxcomputeTypeMapper.resolveObject(MaxcomputeTypeMapper.java:285)"

Flink or Spark Version

spark-2.4.8

Java or Scala Version

scala-2.13.10

Screenshots

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

lucas666666 avatar Jan 05 '23 04:01 lucas666666

maxcompute 2.0 date type , You can view this connection . https://help.aliyun.com/document_detail/159541.html

lucas666666 avatar Jan 05 '23 05:01 lucas666666

I want to try this

msftgitsklsd avatar Jan 21 '23 07:01 msftgitsklsd

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] avatar Feb 21 '23 00:02 github-actions[bot]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.

github-actions[bot] avatar Mar 01 '23 00:03 github-actions[bot]