quickwit icon indicating copy to clipboard operation
quickwit copied to clipboard

tool local-ingest fails to ingest from stdin

Open tandr opened this issue 6 months ago • 0 comments

Describe the bug local-ingest fails to ingest from stdin

$ cat > xaa <<EOF
{"_raw":"[email protected] Access denied from 127.0.0.2","_raw_hash2":-8546123884480137573,"email":"[email protected]", "ip":["127.0.0.2"]}
{"_raw":"[email protected] Login successful from 127.0.0.1",_raw_hash2":-3287367624800626342,"email":"[email protected]","ip":["127.0.0.1"]}
EOF

$ cat xaa |  quickwit tool local-ingest --index col5
❯ Ingesting documents locally...

---------------------------------------------------
 Connectivity checklist 
 ✔ metastore storage
 ✔ metastore
 ✔ index storage
 ✔ _ingest-cli-source

2025-05-20T17:20:41.475Z ERROR quickwit_indexing::actors::indexing_pipeline: error while spawning indexing pipeline, retrying after some time error=unknown source type `Stdin` (available source types are ingest API v2, ingest API v1, file, void, vec) retry_count=0 retry_delay=2s
2025-05-20T17:20:43.476Z ERROR quickwit_indexing::actors::indexing_pipeline: error while spawning indexing pipeline, retrying after some time error=unknown source type `Stdin` (available source types are ingest API v2, ingest API v1, file, void, vec) retry_count=1 retry_delay=4s
^C

Steps to reproduce (if applicable) Steps to reproduce the behavior:

  1. create an index
  2. create a jsonl file
  3. try to execute cat JSONLFILE | quickwit tool local-ingest --index INDEX

Expected behavior A clear and concise description of what you expected to happen.

Configuration: Please provide:

  1. Output of quickwit --version
$ quickwit --version
Quickwit 0.8.0 (x86_64-unknown-linux-gnu unknown unknown)

$ git rev-parse HEAD
6ec11c555aaf05912e1c7a394ad6ace48d095696
  1. The index_config.yaml

version: 0.7

index_id: col5

doc_mapping:
  field_mappings:
    - name: _id
      type: text
      tokenizer: raw
      stored: true
    - name: _raw
      type: text
      record: position
      stored: false
    - name: _raw_hash2
      type: i64
      fast: true
    - name: username
      type: array<text>
      record: position
    - name: ip
      type: array<ip>
      fast: true
    - name: domain
      type: array<text>
      fast: true
      record: position

 #
  dynamic_mapping:
    tokenizer: default
#    type: array<text>
    stored: true
    indexed: true

search_settings:
  default_search_fields: [_raw]

indexing_settings:
  commit_timeout_secs: 50

  1. The config file
version: 0.8
data_dir: /home/me/dev/quickwit/qwdata

indexer:
  enable_otlp_endpoint: ${QW_ENABLE_OTLP_ENDPOINT:-false}

ingest_api:
  max_queue_memory_usage: 8GiB
  max_queue_disk_usage: 40GiB
  content_length_limit: 1GiB
  shard_burst_limit: 2GiB
# i wish I could put this unlimited
  shard_throughput_limit: 20MiB

jaeger:
  enable_endpoint: ${QW_ENABLE_JAEGER_ENDPOINT:-false}

tandr avatar May 20 '25 17:05 tandr