Conditionals: no way to check if boolean field exists

[bradvido]

In the Logstash config language, I can't determine any way to check if a boolean field exists.

e.g. Using this conditional if[myfield] { ...exists... } else { ...doesn't exist... } The results from testing this conditional statement are:

[myfield] does not exist   --> false
[myfield] exists, is true  --> true
[myfield] exists, is false --> false //expected true because the field exists

I've asked the question here as well to reach out and see if someone else knows a way to check for boolean existence. http://stackoverflow.com/questions/26287082/logstash-config-check-if-boolean-field-exists

[bradvido]

This issue is somewhat related as it points out the documentation isn't clear about what evaluates to true in conditionals https://github.com/elasticsearch/logstash/issues/1692

[jordansissel]

This is a bug of omission in the logstash configuration. You correctly identified that 'field exists?' and 'field is true' both are indistinguishable, and this is unfortunate.

We should fix this, but I'm not sure what the best approach is -

Maybe we should make boolean test explicit, if [field] == true ?

[timbunce]

@jordansissel, I seem to recall talk of adding functions, though I might be imagining that, it's a good idea anyway.

So perhaps if exists [field] { ... } or if exists([field]) { ... }

[javifr]

+1 for @timbunce

[webwurst]

+1

[naisanza]

I'm running into this where I'm looking to use an if statement to see if the field contains data. If true, then:

if [mxIP] !== "(?:%{null})|-|" {
    mutate {
    add_field => { "ip" => "%{mxIP}"}
    }
}

Also, does the current implementation of the if statement allow the use of regex and boolean within a field value? Like such:

if [mxIP] == "((?:%{null})|-|)" {

[alfredocambera]

This feature would be very useful

[groeney]

+1

[missnebun]

+1

[omercnet]

+1

[sahlex]

+1

[GoodMirek]

+1

[geoffroya]

+1

[GoodMirek]

Let me comment.

The issue is that it is not possible to distinguish whether a boolean field is false or does not exist. If a test of a boolean field returns true, then it exists for sure and is true. If a test of a boolean field returns false, then either the field exists and is false or the field does not exist. Ability to distinguish these two states would be greatly appreciated.

Sorry if it was already clear and my comment was surplus.

Mirek Svoboda | +420 608 224 486 | Skype: xsvobo10

On 26 April 2016 at 02:35, DeDe Morton [email protected] wrote:

@jordansissel https://github.com/jordansissel I want to add doc updates related to this issue. When I read through to the end, it sounds like most of the confusion is around the fact that testing for “does field exist?” is indistinguishable from testing for “is field true?” But it sounds like we can also improve the the docs. Can you help me get the doc right by answering the questions that bradvido raises? There are three main ones: What is the order of precedence of the operators, how does the "in" operator behave, and what are the type restrictions for specific operators (for example, do comparison operators for equality only accept numeric types?) Here's where I apologize for not testing this myself, but it would take me longer than it's worth, and it's not worth my time if you have the answers already.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/elastic/logstash/issues/1867#issuecomment-214572507

[jordansissel]

@timbunce I agree functions might be a cool way to solve this. I'm open to this.

[cooniur]

+1

[nescim]

+1

[zyphlar]

Nothing after 3 years?

[BarcoMasile]

Still have this problem

[gentunian]

can you post use cases where the distinction between false and the non-existing field is meaningful? I know that semantics may be in the way but there are a lot of workaround for this being a meaningless use case.

[synFK]

Here you go: You maybe get a field foo of type boolean from some input. If the field actually exists you want to convert it to an integer. If not you want to add it to the event and initialize to value 42. So your code might look like this:

filter {
  if [foo] {
    mutate {
      convert => { "foo" => "integer" }
    }
  } else {
    mutate {
      add_field => { "foo" => 42 }
    }
  }
}

If foo is false it's not converted but 42 is added, which you definitely don't want.

[gentunian]

@darefilz looks that you may rethink the pipeline or control the data ingestion. The field foo is supposed to be an integer but sometimes you may be receiving a boolean value. However, I think you could solve this problem.

I understand that despite anything, you need foo field to be an integer and to be added to your pipeline. In the special case where the field does not exists or the field exists but it's false, then, you want to have the field foo = 42.

You may convert the field anyway despite its existence. You now know that if the field exists and it has the false value, you converted it into the 0 integer.

For instance:

input {
  http { port => 9090 }
}

filter {
  # boolean false is converted to 0
  mutate {
    convert => { "foo" => "integer" }
  }

  # if the field does not exists or the field exists and is false
  if ![foo] or [foo] == 0 {
    mutate {
      replace => { "foo" => 42 }
      # this is needed because logstash will set 42 as String.
      # maybe this is related to this issue: https://github.com/elastic/logstash/issues/2987
      convert => { "foo" => "integer" }
    }
  }
}

output {
  stdout {}
}

I know something like if [foo] == false would be better, but also there are a lot of approaches to accomplish it.

Test cases

1. Foo exists and it's a string number. We expect to be converted: OK

curl localhost:9090 -H 'Content-Type: application/json' -d '{"foo": "12"}'
{
           "foo" => 12,
}

2. Foo exists and it's an integer. We expect nothing.

curl localhost:9090 -H 'Content-Type: application/json' -d '{"foo": 12}'
{
           "foo" => 12,
}

3. Foo does not exists. We expect to be created with the integer 42: OK, but note the _jsonparsefailure tag:

curl localhost:9090 -H 'Content-Type: application/json' -d '{"bar": "meh}'
{
          "tags" => [
        [0] "_jsonparsefailure"
    ],
           "foo" => 42,
}

4. Foo is false. We expect to be converted into the integer 42: OK

curl localhost:9090 -H 'Content-Type: application/json' -d '{"foo": false}'
{
           "foo" => 42,
}

[gentunian]

Well, I forgot your last line:

If foo is false it's not converted but 42 is added, which you definitely don't want.

which makes the pipeline simpler:

input {
  http { port => 9090 }
}

filter {
  # boolean false is converted to 0
  mutate {
    convert => { "foo" => "integer" }
  }

  # now we already know that if the field was
  # false, it will be converted to 0. So we check
  # only if the field does not exists
  if ![foo] {
    mutate {
      add_field => { "foo" => 42 }
      # this is needed because logstash will set 42 as String.
      # maybe this is related to this issue: https://github.com/elastic/logstash/issues/2987
      convert => { "foo" => "integer" }
    }
  }
}

output {
  stdout {}
}

[nescim]

can you post use cases where the distinction between false and the non-existing field is meaningful? I know that semantics may be in the way but there are a lot of workaround for this being a meaningless use case.

Suppose you have some records, but not all, marked with a boolean field that is used later in the pipeline and that for unmarked records one needs to invoke a process to make a decision. Maybe that process is expensive. Maybe it's not even 100% accurate.

[gentunian]

@nescim you could use [@metadata][myTemporaryField] for those scenarios. I don't quite understand your example, but for those records marked with a boolean field used in the pipeline you may change your pipeline to use your [@metadata][marked] when they are marked. And if ![@metadata][marked] when they are not.

[dodo5575]

I would really want to have the "exist" function!

My use case: If the boolean field "foo" exists, put it in an S3 good_data bucket, else, put it in another bad_data bucket

output { if [foo] { s3 {good_data} } else { s3 {bad_data} }
}

Now, if the foo == false, this record will be put in the bad_data bucket, which is not what I want. And, I have a list of field to check. Some of them are boolean while others are not. Hard-coding to put the case where foo == false to the good_data bucket is not a clean solution.

[MakoWish]

This bug has been open for nearly 6 years?! I am running into this same issue. I have a boolean field that does not exist in all events, but I need to check if it exists, and parse differently whether true or false. I have tried these different ways of testing

if [field] == true or [field] == false {
 --- do work ---
}

Logstash will to fail to start with Expected one of [ \\t\\r\\n], \"#\", \"(\" at line...

if [field] == 'true' or [field] == 'false' {
 --- do work ---
}

Never recognizes that the field exists whether or not it does. Guessing this is because the condition is checking for a string value which will not work for a boolean value.

if "" in [field] {
 --- do work ---
}

Also does not work. Again, looking for a string value; which a boolean is not?

I cannot for the life of me figure out how to handle this.

[jessequinn]

@MakoWish i am in the same boat. I cannot get conditional testing with boolean field to work!!!

Is there an update on this? i am using 7.6.2.

[MakoWish]

Still looking for a solution to this issue. There needs to be a way to test test for the presence of a boolean field. This issue has been open now for more than six years with no resolution. Would appreciate some movement on this.

[yodog]

+1

bumping so the thread wont die.