Handle S3 compatible storages such as MinIO

[cosmo0920]

This is minimal support for S3 compatible storage such as MinIO.

$ docker run \
  -p 9000:9000 \
  -p 9001:9001 \
  -e "MINIO_ROOT_USER=AKIAEXAMPLEKEY" \
  -e "MINIO_ROOT_PASSWORD=P8AvMjug9c14v/K7MDENG/bPxRfiCYEXAMPLEKEY" \
  minio/minio server /data --console-address ":9001"

With this change, I confirmed that MinIO starts to handle object ingestion:

$ cat store_info.log
321,aaa
356,zzz
859,"a,b c, d"
879,"a\"b\"c\"d\"de"
999,NULL

in:
  type: file
  path_prefix: ./store_info
  parser:
    type: csv
    charset: UTF-8
    newline: LF
    delimiter: ','
    quote: '"'
    escape: '\'
    null_string: 'NULL'
    skip_header_lines: 0
    columns:
    - {name: store_code, type: string}
    - {name: store_name, type: string}

out:
  type: s3
  path_prefix: logs/out
  file_ext: .csv
  bucket: my-minio-bucket
  endpoint: http://127.0.0.1:9000
  access_key_id: AKIAEXAMPLEKEY
  secret_access_key: P8AvMjug9c14v/K7MDENG/bPxRfiCYEXAMPLEKEY
  use_path_style: true
  formatter:
    type: csv

$ embulk --version
embulk 0.9.23

Signed-off-by: Hiroshi Hatake [email protected]

[hiroyuki-sato]

Thank you for contributing to this project!

Ref: Twitter