Processors DSL

[gegen07]

trafficstars

Is there a solution to DSL for processors? Using low-client to create dynamic processors hurts a lot, a mess of kwargs and if/else.

[gegen07]

@sethmlarson Any plans to add IngestClient to DSL package?

[sethmlarson]

Not currently, what parts of the ingest pipeline are tough to use and warrant a declarative structure?

[gegen07]

I was using the enrich processor, but the enrichment process could support an array of objects or a simple JSON object. So, I wrote two repetitive blocks of code like these ones below:

For the array of objects:

self._json = {
  "description": description,
  "processors": [
      {
          "foreach": {
              "field": kwargs.get("field_array"),
              "processor": {
                  "enrich": {
                      "policy_name": policy_name,
                      "field": "_ingest._value." + match_field,
                      "target_field": "_ingest._value."
                      + target_field_name,
                      "ignore_missing": True,
                  }
              },
              "ignore_failure": True,
          }
      }
  ],
}

For simple JSON objects:

self._json = {
    "description": description,
    "processors": [
        {
            "enrich": {
                "policy_name": policy_name,
                "field": match_field,
                "target_field": target_field_name,
                "ignore_missing": True,
            }
        }
    ],
}

For the remove_field processor, I have to do the same copy and change minimal pieces with the same previous logic.

Sometimes there's some adjustment that I could do to be more generic but I didn't think of another manner to do it.

[thedeadliestcatch]

I'm also looking for pipeline settings support and found nothing in the documentation.