aws-sdk-pandas icon indicating copy to clipboard operation
aws-sdk-pandas copied to clipboard

Published 20 hours ago • verified publisher icon aws

Timestream write ray support

Open cnfait opened this issue 2 years ago • 35 comments

Feature or Bugfix

  • Feature
  • Refactoring

Detail

  • Ray support for timestream write
  • num_threads argument changed to use_threads to be consistent with the rest of awswrangler + support of os.cpu_count()

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

cnfait avatar Jul 20 '22 13:07 cnfait

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: e43944fccfcb75934ea492e2f37c2030d0751e99
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 20 '22 13:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: dfc91e5a4090e99ed6c2a9ef13806272e2260929
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 20 '22 13:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: dfc91e5a4090e99ed6c2a9ef13806272e2260929
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 20 '22 13:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: af6d8a1d3fb9515f20f3c2709a6fa5cc0d1a0b3f
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 20 '22 14:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: af6d8a1d3fb9515f20f3c2709a6fa5cc0d1a0b3f
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 20 '22 14:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: af6d8a1d3fb9515f20f3c2709a6fa5cc0d1a0b3f
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 20 '22 15:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: efab4f370ae0f99a5e6af725579d58343759f358
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 21 '22 11:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: efab4f370ae0f99a5e6af725579d58343759f358
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 21 '22 11:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: efab4f370ae0f99a5e6af725579d58343759f358
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 21 '22 11:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: 8f26c3c9940cdbddb45bb3ad4a8fd27a31a3e9b6
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 21 '22 18:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 8f26c3c9940cdbddb45bb3ad4a8fd27a31a3e9b6
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 21 '22 18:07 malachi-constant

Looking good. Did you test this in ray and non-ray scenario?

When running standard unit tests there is still an issue, I'm investigating it.

cnfait avatar Jul 22 '22 09:07 cnfait

Looking good. Did you test this in ray and non-ray scenario?

When running standard unit tests there is still an issue, I'm investigating it.

I think you just need to move this argument to become the first one in _write_batch instead of database. This is because the self.exec.map method in threading is submitting boto3_primitives as the first argument

jaidisido avatar Jul 22 '22 09:07 jaidisido

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: e810b7876e28f94ab1bf2ee65c5f459157134e29
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 22 '22 10:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: e810b7876e28f94ab1bf2ee65c5f459157134e29
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Jul 22 '22 10:07 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: fd2c127284d1f34fd0a26277857cab15e2d98c7d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 01 '22 10:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubStandardCodeBuild8C06-llutOAimTATs
  • Commit ID: fd2c127284d1f34fd0a26277857cab15e2d98c7d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 01 '22 10:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: fd2c127284d1f34fd0a26277857cab15e2d98c7d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 01 '22 10:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: fd2c127284d1f34fd0a26277857cab15e2d98c7d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 05 '22 18:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: 964d1095a1297518ac1c1e831ecd576c53844ca8
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 05 '22 18:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: 964d1095a1297518ac1c1e831ecd576c53844ca8
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 05 '22 18:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubStandardCodeBuild8C06-llutOAimTATs
  • Commit ID: 964d1095a1297518ac1c1e831ecd576c53844ca8
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 05 '22 18:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: 03a75bcd059a40da1055d19348ab0b303f462b9d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 05 '22 23:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: 03a75bcd059a40da1055d19348ab0b303f462b9d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 05 '22 23:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubStandardCodeBuild8C06-llutOAimTATs
  • Commit ID: 03a75bcd059a40da1055d19348ab0b303f462b9d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 05 '22 23:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: f306baa7e8d208d25a4092f39f35897c73bfcfd7
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 07 '22 15:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: 3c4ee13c93eee9c1fd50ac2c08f520a600fcbb16
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 07 '22 15:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubLoadTests5656BB24-s6u9F3qN9oFy
  • Commit ID: 3c4ee13c93eee9c1fd50ac2c08f520a600fcbb16
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 07 '22 15:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubStandardCodeBuild8C06-llutOAimTATs
  • Commit ID: 3c4ee13c93eee9c1fd50ac2c08f520a600fcbb16
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 07 '22 15:09 malachi-constant

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: e0b75f200dcf4c695ba015a0e4540f38d0ffba2d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant avatar Sep 07 '22 15:09 malachi-constant