quickwit
quickwit copied to clipboard
tool local-ingest fails to ingest from stdin
Describe the bug local-ingest fails to ingest from stdin
$ cat > xaa <<EOF
{"_raw":"[email protected] Access denied from 127.0.0.2","_raw_hash2":-8546123884480137573,"email":"[email protected]", "ip":["127.0.0.2"]}
{"_raw":"[email protected] Login successful from 127.0.0.1",_raw_hash2":-3287367624800626342,"email":"[email protected]","ip":["127.0.0.1"]}
EOF
$ cat xaa | quickwit tool local-ingest --index col5
❯ Ingesting documents locally...
---------------------------------------------------
Connectivity checklist
✔ metastore storage
✔ metastore
✔ index storage
✔ _ingest-cli-source
2025-05-20T17:20:41.475Z ERROR quickwit_indexing::actors::indexing_pipeline: error while spawning indexing pipeline, retrying after some time error=unknown source type `Stdin` (available source types are ingest API v2, ingest API v1, file, void, vec) retry_count=0 retry_delay=2s
2025-05-20T17:20:43.476Z ERROR quickwit_indexing::actors::indexing_pipeline: error while spawning indexing pipeline, retrying after some time error=unknown source type `Stdin` (available source types are ingest API v2, ingest API v1, file, void, vec) retry_count=1 retry_delay=4s
^C
Steps to reproduce (if applicable) Steps to reproduce the behavior:
- create an index
- create a jsonl file
- try to execute
cat JSONLFILE | quickwit tool local-ingest --index INDEX
Expected behavior A clear and concise description of what you expected to happen.
Configuration: Please provide:
- Output of
quickwit --version
$ quickwit --version
Quickwit 0.8.0 (x86_64-unknown-linux-gnu unknown unknown)
$ git rev-parse HEAD
6ec11c555aaf05912e1c7a394ad6ace48d095696
- The index_config.yaml
version: 0.7
index_id: col5
doc_mapping:
field_mappings:
- name: _id
type: text
tokenizer: raw
stored: true
- name: _raw
type: text
record: position
stored: false
- name: _raw_hash2
type: i64
fast: true
- name: username
type: array<text>
record: position
- name: ip
type: array<ip>
fast: true
- name: domain
type: array<text>
fast: true
record: position
#
dynamic_mapping:
tokenizer: default
# type: array<text>
stored: true
indexed: true
search_settings:
default_search_fields: [_raw]
indexing_settings:
commit_timeout_secs: 50
- The config file
version: 0.8
data_dir: /home/me/dev/quickwit/qwdata
indexer:
enable_otlp_endpoint: ${QW_ENABLE_OTLP_ENDPOINT:-false}
ingest_api:
max_queue_memory_usage: 8GiB
max_queue_disk_usage: 40GiB
content_length_limit: 1GiB
shard_burst_limit: 2GiB
# i wish I could put this unlimited
shard_throughput_limit: 20MiB
jaeger:
enable_endpoint: ${QW_ENABLE_JAEGER_ENDPOINT:-false}