clp
clp copied to clipboard
[WIP] clp-package: Pass search arguments via msgpack object instead of command line to search processes.
Description
This PR modifies clp-s and clo to pass most search arguments via a msgpack object written to stdin instead of via command line arguments. This allows the fs_search_task
python wrapper to be less smart, and makes it easier to change the arguments passed to our search processes -- now we just need to modify the SearchConfig
object in job_config.py
and make clo/clp-s use the new fields directly.
The diff is somewhat large, but mostly consists of simple changes to the CommandLineArguments
classes of clp-s and clo to take config from a msgpack object instead of from command line arguments.
We also change the search scheduler to make it augment the SearchConfig
object with a mongodb output destination when no other output destination is specified. Having the mongodb output destination configured explicitly before the search job gets passed downstream allows downstream components to be less smart, and adding the mongodb output destination needs to happen inside the scheduler to follow our current convention that the job id is the result collection name.
Validation performed
- Validated that search and aggregation jobs work correctly with clp engine
- Validated that search and aggregation jobs work correctly with clp-s engine