timesketch icon indicating copy to clipboard operation
timesketch copied to clipboard

Configure tasks plaso and run_csv_jsonl Opensearch to use flush interval

Open Zawadidone opened this issue 3 years ago • 1 comments

Is your feature request related to a problem? Please describe. Based on the data ingested to Timesketch users should have the option to configure the amount of documents send to Opensearch in a single bulk request. This could speed up the ingestion of data in Timesketch based on the size of the Opensearch cluster.

Describe the solution you'd like The parameter --flush_interval used by Plaso can be configured from the Timesketch application config with the name OPENSEARCH_FLUSH_INTERVAL.

The DEFAULT_FLUSH_INTERVAL used by the task run_csv_jsonl can be configured from the Timesketch application config with the name OPENSEARCH_FLUSH_INTERVAL.

Describe alternatives you've considered n/a

Additional context https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html#_using_and_sizing_bulk_requests https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#_use_bulk_requests

Zawadidone avatar Sep 04 '22 16:09 Zawadidone

Also added the task run_csv_jsonl to the description

Zawadidone avatar Sep 05 '22 16:09 Zawadidone