apify-sdk-python
apify-sdk-python copied to clipboard
fix: Make `ApifyFileSystemStorageClient` fully compatible with the apify cli
Description
- Make sure that storage from
ApifyFileSystemStorageClientdoes not get purged twice due to storage fromFileSystemStorageClientpointing to the same location. (Those storage clients will have the same cache key and thus there can be only one.) - Ensure that
Actorwill open input containing KVS on initialization to ensure that an aware storage client is used. - Support any possible pre-existing input key and file that is defined through
Configuration.input_key. Different input files will have different handling based on their suffix:- ".json" is parsed as json.
- ".txt" is opened as plain text
- everything else is opened as bytes
- without extension is tried to be parsed as json first, but falls back to bytes
- Create a metadata file for the valid pre-existing input file, without modifying the input file (otherwise, cli might detect the change to the input, which would be a false positive)
- Raise an error if two valid pre-existing input files exist in the expected storage directory.
- CLI does not respect env variables with the input key so far. TODO: https://github.com/apify/apify-cli/issues/960
Issues
Closes: #621 Related to: #INPUT.json Automatically Deleted on Each Run (Python SDK Local Storage Issue)
Testing
- Added unit tests.
- Manually tested with [email protected]
- npx [email protected] run -i {"a":"c"} with pre-existing input file or without input and multiple times in a row
- npx [email protected] run with pre-existing input file or without input and multiple times in a row