Support piping DynamoDB query / scan output to another command

[jthomerson]

Is your feature request related to a problem? Please describe. When I use the AWS CLI to query or scan a DynamoDB table, I am unable to pipe that output to another command (effectively) because the JSON structure of the output requires the output to be 100% complete before another command can process it. If you need to whip up a quick-and-dirty 'query this table for data, and send each row to this other command' type job, you can't effectively do so if the output is thousands, tens of thousands, or millions of lines - the entire JSON output will be buffered, resulting in extremely slow processing and a huge load on both the CLI itself and the next command in your pipeline to process that giant JSON.

Describe the solution you'd like Line-delimited JSON for datasets such as DynamoDB queries, scans, S3 lists, etc. Then each line can be output from the CLI as soon as it's processed, and the next command in the pipeline can process that line without waiting for the entire dataset to be complete.

Describe alternatives you've considered The alternative is writing my own scripts with the SDK, removing the flexibility and speed of just using the CLI for one-off tasks.

Additional context See also https://github.com/aws/aws-cli/issues/4703#issuecomment-657343054

[stobrien89]

Hi @jthomerson,

Thanks for the feature request!

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[jthomerson]

Yes, this is still an issue. If someone wanted to point me towards where to start with creating an alternative output format, I'd be happy to look into providing a pull request.