fix(elasticsearch): Encode bulk action parameters as JSON

[jszwedko]

They are currently using string templating which, if there are special characters in the value, will end up creating an invalid JSON payload; an issue that can be difficult to track down. This happened in https://github.com/vectordotdev/vector/discussions/21288.

Signed-off-by: Jesse Szwedko [email protected]

[datadog-vectordotdev[bot]]

Datadog Report

Branch report: jszwedko/fix-elasticsearch-message-encoding Commit report: 509120c Test service: vector

:x: 9 Failed (0 Known Flaky), 2223 Passed, 0 Skipped, 1m 31.64s Total Time

:x: Failed Tests (9)

This report shows up to 5 failed tests.

• sinks::elasticsearch::tests::allows_using_except_fields - vector - Details

  Expand for error

  hread 'sinks::elasticsearch::tests::allows_using_except_fields' panicked at src/sinks/elasticsearch/tests.rs:494:5:
  ssertion \`left == right\` failed
   left: "{\"index\":{\"_type\":\"_doc\",\"_index\":\"purple\"}}\n{\"foo\":\"bar\",\"message\":\"hello there\"}\n"
  right: "{\"index\":{\"_index\":\"purple\",\"_type\":\"_doc\"}}\n{\"foo\":\"bar\",\"message\":\"hello there\"}\n"
  tack backtrace:
    0:     0x5575c3570165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x5575c3570165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x5575c3570165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
  ..
  

• sinks::elasticsearch::tests::allows_using_only_fields - vector - Details

  Expand for error

  hread 'sinks::elasticsearch::tests::allows_using_only_fields' panicked at src/sinks/elasticsearch/tests.rs:529:5:
  ssertion \`left == right\` failed
   left: "{\"index\":{\"_type\":\"_doc\",\"_index\":\"purple\"}}\n{\"foo\":\"bar\"}\n"
  right: "{\"index\":{\"_index\":\"purple\",\"_type\":\"_doc\"}}\n{\"foo\":\"bar\"}\n"
  tack backtrace:
    0:     0x557e6a475165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x557e6a475165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x557e6a475165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
  ..
  

• sinks::elasticsearch::tests::encode_datastream_mode - vector - Details

  Expand for error

  hread 'sinks::elasticsearch::tests::encode_datastream_mode' panicked at src/sinks/elasticsearch/tests.rs:251:5:
  ssertion \`left == right\` failed
   left: "{\"create\":{\"_type\":\"_doc\",\"_index\":\"synthetics-testing-default\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"namespace\":\"default\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  right: "{\"create\":{\"_index\":\"synthetics-testing-default\",\"_type\":\"_doc\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"namespace\":\"default\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  tack backtrace:
    0:     0x557f7366b165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x557f7366b165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x557f7366b165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
  ..
  

• sinks::elasticsearch::tests::encode_datastream_mode_no_routing - vector - Details

  Expand for error

  hread 'sinks::elasticsearch::tests::encode_datastream_mode_no_routing' panicked at src/sinks/elasticsearch/tests.rs:306:5:
  ssertion \`left == right\` failed
   left: "{\"create\":{\"_type\":\"_doc\",\"_index\":\"logs-generic-something\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"namespace\":\"something\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  right: "{\"create\":{\"_index\":\"logs-generic-something\",\"_type\":\"_doc\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"namespace\":\"something\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  tack backtrace:
    0:     0x55dabe5bb165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x55dabe5bb165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x55dabe5bb165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
  ..
  

• sinks::elasticsearch::tests::encode_datastream_mode_no_sync - vector - Details

  Expand for error

  hread 'sinks::elasticsearch::tests::encode_datastream_mode_no_sync' panicked at src/sinks/elasticsearch/tests.rs:458:5:
  ssertion \`left == right\` failed
   left: "{\"create\":{\"_type\":\"_doc\",\"_index\":\"synthetics-testing-something\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  right: "{\"create\":{\"_index\":\"synthetics-testing-something\",\"_type\":\"_doc\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  tack backtrace:
    0:     0x55ae51faf165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x55ae51faf165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x55ae51faf165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
  ..
  

[pront]

Note: Tests fail due the order of keys in the objects. The data look identical otherwise.

[datadog-vectordotdev[bot]]

Datadog Report

Branch report: jszwedko/fix-elasticsearch-message-encoding Commit report: dfadc0b Test service: vector

:white_check_mark: 0 Failed, 7 Passed, 0 Skipped, 25.26s Total Time