apify-cli icon indicating copy to clipboard operation
apify-cli copied to clipboard

fix: cleanup purging behavior for CLI

Open vladfrangu opened this issue 11 months ago • 7 comments

Closes #590

BREAKING CHANGE:

Purging in projects using Crawlee for SDK v3+ now purge by default, so using --purge flag is no longer necessary. Use --no-purge to keep the storage folder intact. The purge-queue, purge-dataset and purge-key-value-store flags have been removed, and the logic of all three was combined into the purge flag.

vladfrangu avatar Jan 15 '25 19:01 vladfrangu

Hi guys, I've just realized that if we want the CLI to behave consistently with how the Apify platform works, we shouldn't purge entirely by default, but instead the CLI should just keep adding local runs for later introspection and comparison. I know you're making it on par with Crawlee's behavior, but I think this will introduce further divergence.

netmilk avatar Jan 16 '25 08:01 netmilk

I am not sure about that, regular actor developer runs, runs, and runs. They would end up with a lot of junk locally, I am not sure people would like such a default.

Also, I am not sure if this is a "divergence", we are not doing that now either, the CLI never considered multiple storage folders, neither the actual consumers of the storage folder (crawlee/sdk). We could surely just rename the existing storage folder instead of wiping it, but should that really be a default?

B4nan avatar Jan 16 '25 08:01 B4nan

@B4nan Understood. Divergence from the platform behavior is what I'm concerned about. You run it there and the results always persist by default under a new run. But unifying the behavior of our local tooling is definitely a way to go.

netmilk avatar Jan 16 '25 09:01 netmilk

@vladfrangu lets describe the BCs in the PR description

B4nan avatar Jan 16 '25 09:01 B4nan

I also found out you can group flags in the help menu. Hows this look? @netmilk @B4nan

Code - 2025-01-16 at 11 51 25

vladfrangu avatar Jan 16 '25 09:01 vladfrangu

nit: should be Node.js, not node.js

B4nan avatar Jan 16 '25 10:01 B4nan

also i double checked with others, python works the same as JS, auto purge for crawlee or SDK (even without crawlee) v2+

B4nan avatar Jan 16 '25 10:01 B4nan