matano icon indicating copy to clipboard operation
matano copied to clipboard

Zeek log source does not accept TSV format

Open hilt86 opened this issue 10 months ago • 0 comments

Zeek writes logs using TSV / ASCII format by default. These logs are gzipped (by default) by the Zeek log rotation script so most folks will end up uploading gzipped TSV logs by default.

Expected bahaviour : zeek tsv files are un-gzipped and added to the matano data lake

Actual behaviour : the transformer function fails with

INFO transformer: {
    "bytes_processed": 1607225,
    "error": false,
    "failing_log_sources": null,
    "log_sources": [
        "zeek"
    ],
    "matano_log": true,
    "rows_written": 0,
    "service": "transformer",
    "sidelined_lines_count": null,
    "sidelined_log_sources": null,
    "time": 165,
    "type": "matano_service_log"
}    

hilt86 avatar Aug 25 '23 22:08 hilt86