graylog2-server icon indicating copy to clipboard operation
graylog2-server copied to clipboard

Published 20 hours ago • verified publisher icon Graylog2

Stream search finds messages that doesn't belong to its index set

Open 2fa opened this issue 6 years ago • 25 comments

Expected Behavior

Stream search should only find messages that corresponds to its index set

Current Behavior

Stream search finds all messages matching its rules (even if they stored in different index sets, but it's true only for rotated indexes)

Possible Solution

Remove stream tag from the message if that stream doesn't belong to its index set

Steps to Reproduce

  1. Create two streams with similar rule sets that belong to different index sets
  2. Enable checkbox "Remove matches from 'All messages' stream" for the stream
  3. Load messages that should go to both streams
  4. Rotate their indexes
  5. Search

Context

We have two streams with similar rules: D2Requests and D2UserRequests (one of them is a subset of the other one), they belong to different index sets (first one -- default "graylog_", second -- custom with a longer lifetime "lt_graylog_").

stream_d2r stream_d2ur

When you search in D2UserRequests stream there are duplicate messages but only for a rotated "default" indexes:

stream_search

Default index has been rotated at ~11:20.

stream_search_message1

stream_search_message2

Same message, same stream -- different indexes.

If you search "All messages" stream for messages that applied to D2UserRequests rules it finds nothing, so checkbox is working there.

Your Environment

  • Graylog Version: 2.4.3+2c41897
  • Elasticsearch Version: 5.6.4
  • MongoDB Version: 3.4.10
  • Operating System: Linux 4.10.0-42-generic #46~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

2fa avatar Mar 06 '18 09:03 2fa