canvas-data-cli icon indicating copy to clipboard operation
canvas-data-cli copied to clipboard

Update historical-requests for new S3 location

Open buckett opened this issue 2 years ago • 2 comments

Thanks for submitting a PR! We want to make contributing to the Canvas Data CLI as easy as possible. Please read these instructions carefully:

  • [x] Explain the motivation for making this change.
  • [x] Provide a test plan demonstrating that the code is solid.
  • [x] Match the code formatting of the rest of the codebase.
  • [x] Make sure to add tests to help keep code coverage up.

Motivation (required)

Fixes the historical-requests subcommand to work with the new file location in S3.

Test Plan (required)

Run canvasDataCli historical-requests and check that it groups files by date range.

Next Steps

  • Small pull requests are much easier to review and more likely to get merged. Make sure the PR does only one thing, otherwise please split it.
  • Make sure all tests pass, we will run this on jenkins but you can run it yourself with the build.sh script.

buckett avatar Feb 14 '23 11:02 buckett

This should fix #40

buckett avatar Feb 14 '23 11:02 buckett

Ok, but it's not that simple as the newer requests are of the format:

https://s3.amazonaws.com/<timestamp>/dw_split/<account>/requests/b%3D1/part-<uuid>.c001.txt.gz

so the code should cope with both options. This does mean you either get a range or a timestamp in the grouping.

buckett avatar Feb 14 '23 15:02 buckett