Error: org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `map' for nil:NilClass Did you mean? tap

[GitSumito]

trafficstars

When I executed embulk. (mysql to BigQuery) I encountered below error.

Error: org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `map' for nil:NilClass Did you mean?  tap

I'm not sure what was wrong. If you have any idea to resolve this issue, please kindly let me know.

here is a detail of error.

2020-02-18 11:07:27.090 +0000 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.10.1)
2020-02-18 11:07:31.068 +0000 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.6.4)
2020-02-18 11:07:31.117 +0000 [INFO] (0001:transaction): JDBC Driver = /root/.embulk/jruby/2.3.0/gems/embulk-input-mysql-0.10.1/default_jdbc_driver/mysql-connector-java-5.1.44.jar
2020-02-18 11:07:31.126 +0000 [INFO] (0001:transaction): Fetch size is 10000. Using server-side prepared statement.
2020-02-18 11:07:31.128 +0000 [INFO] (0001:transaction): Connecting to jdbc:mysql://mask.amazonaws.com:3306/account_management options {useCompression=true, socketTimeout=1800000, useSSL=false, user=mask, useLegacyDatetimeCode=false, tcpKeepAlive=true, useCursorFetch=true, connectTimeout=300000, password=***, zeroDateTimeBehavior=convertToNull}
2020-02-18 11:07:31.504 +0000 [INFO] (0001:transaction): Using JDBC Driver mysql-connector-java-5.1.44 ( Revision: b3cda4f864902ffdde495b9df93937c3e20009be )
2020-02-18 11:07:31.504 +0000 [WARN] (0001:transaction): embulk-input-mysql 0.9.0 upgraded the bundled MySQL Connector/J version from 5.1.34 to 5.1.44 .
2020-02-18 11:07:31.505 +0000 [WARN] (0001:transaction): And set useLegacyDatetimeCode=false by default in order to get correct datetime value when the server timezone and the client timezone are different.
2020-02-18 11:07:31.505 +0000 [WARN] (0001:transaction): Set useLegacyDatetimeCode=true if you need to get datetime value same as older embulk-input-mysql.
2020-02-18 11:07:31.567 +0000 [INFO] (0001:transaction): Using local thread executor with max_threads=4 / output tasks 2 = input tasks 1 * 2
2020-02-18 11:07:31.621 +0000 [INFO] (0001:transaction): embulk-output-bigquery: Get dataset... mask-data-strategy-dev:embulk
2020-02-18 11:07:34.070 +0000 [INFO] (0001:transaction): embulk-output-bigquery: Create table... mask-data-strategy-dev:embulk.LOAD_TEMP_0be4f76a_98ca_4b2b_8da2_8380d4fa7ab0_embulk_hoge
2020-02-18 11:07:34.071 +0000 [INFO] (0001:transaction): embulk-output-bigquery: Get table... mask-data-strategy-dev:embulk.hoge
org.embulk.exec.PartialExecutionException: org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `map' for nil:NilClass
Did you mean?  tap
	at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:375)
	at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:607)
	at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:391)
	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:387)
	at org.embulk.spi.Exec.doWith(Exec.java:25)
	at org.embulk.exec.BulkLoader.run(BulkLoader.java:387)
	at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:180)
	at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:331)
	at org.embulk.EmbulkRunner.run(EmbulkRunner.java:174)
	at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:468)
	at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:100)
	at org.embulk.cli.Main.main(Main.java:28)
	Suppressed: java.lang.NullPointerException
		at org.embulk.exec.BulkLoader.doCleanup(BulkLoader.java:496)
		at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:427)
		at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:423)
		at org.embulk.spi.Exec.doWith(Exec.java:25)
		at org.embulk.exec.BulkLoader.cleanup(BulkLoader.java:423)
		at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:184)
		... 5 more
Caused by: org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `map' for nil:NilClass
Did you mean?  tap
	at RUBY.fields_from_table(/root/.embulk/jruby/2.3.0/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery/bigquery_client.rb:47)
	at RUBY.fields(/root/.embulk/jruby/2.3.0/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery/bigquery_client.rb:39)
	at RUBY.create_table_if_not_exists(/root/.embulk/jruby/2.3.0/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery/bigquery_client.rb:423)
	at RUBY.auto_create(/root/.embulk/jruby/2.3.0/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:309)
	at RUBY.transaction(/root/.embulk/jruby/2.3.0/gems/embulk-output-bigquery-0.6.4/lib/embulk/output/bigquery.rb:332)
	at RUBY.transaction(/opt/embulk.jar!/embulk/output_plugin.rb:64)

Error: org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `map' for nil:NilClass
Did you mean?  tap

Dockerfile is below.

FROM java:8
ENV EMBULK_VERSION 0.8.39
 
RUN curl -L https://bintray.com/artifact/download/embulk/maven/embulk-${EMBULK_VERSION}.jar -o /opt/embulk.jar
RUN java -jar /opt/embulk.jar gem install embulk-input-mysql
RUN java -jar /opt/embulk.jar gem install embulk-output-bigquery
 
WORKDIR /work 
COPY config.yml.liquid /work/
COPY cert/dev.json /work/dev.json
 
ENTRYPOINT ["java", "-jar", "/opt/embulk.jar", "run", "config.yml.liquid"]

config.yml.liquid is below.

in:
  type: mysql
  user: {{env.DB_USER}}
  password: {{env.DB_PASS}}
  database: {{env.DB_NAME}}
  host: {{env.DB_HOST}}
  query: {{env.DB_QUERY}}
  default_timezone: "Asia/Tokyo"
out:
  type: bigquery
  auth_method: json_key
  json_keyfile: dev.json
  project: hoge-data-strategy-dev
  dataset: embulk
  auto_create_table: true
  table: embulk_hoge
  template_table: hoge
  encoders:
  - {type: gzip}

Execute

docker run -e DB_USER=user -e DB_PASS=pass -e DB_NAME=database -e DB_HOST=mask.ap-northeast-1.rds.amazonaws.com  -e DB_QUERY="select user.id from user" --rm -it dmp/extract-by-embulk

[hiroyuki-sato]

Hello, @GitSumito

I'm not familiar with this plugin much. It seems that the plugin failed to get table info from template_table. https://github.com/embulk/embulk-output-bigquery/blob/master/lib/embulk/output/bigquery/bigquery_client.rb#L47 Does this table exist? Have you ever tried to use schema_file instead of template_table?

[hiroyuki-sato]

I close this issue but feel free to reopen if you need.