apify-cli icon indicating copy to clipboard operation
apify-cli copied to clipboard

Add command to download dataset

Open drobnikj opened this issue 2 years ago • 3 comments

Users are struggling with downloading large datasets.

We need to introduce new command, that handle downloading dataset for them. see slack https://apifier.slack.com/archives/C010Q0FBYG3/p1696943625090939

drobnikj avatar Oct 10 '23 13:10 drobnikj

Pls let's discuss the syntax before we implement it, to make sure it's future proof and consistent with other commands. For example:

apify download-dataset [ID_OR_NAME] --format csv --limit 100 --offset 50 --output /tmp/file.csv ...

We can start with fewer options and add more later

jancurn avatar Oct 10 '23 13:10 jancurn

Let's check if we have some analytics which formats people use the most, but I think for starters it would be enough to support CSV, JSON and JSONL.

Also we could take the format transformation code that we have on the platform, put it in apify-shared and use it in CLI from there.

fnesveda avatar Oct 11 '23 13:10 fnesveda

Once again we stumbled upon lack of this command, see https://apify.slack.com/archives/C06819MJN9Z/p1710923614723909?thread_ts=1710831604.067589&cid=C06819MJN9Z

jancurn avatar Mar 21 '24 12:03 jancurn