[Meta] Upgrading from 7.x to 8.0

[roaksoax]

This issue tracks all the items that need to be handled for the 8.0 release.

• [ ] Support Breaking Changes in Stack Components:
  • [ ] Elasticsearch finalizing Types Removal (upstream: https://github.com/elastic/elasticsearch/issues/41059)
    • [x] Elasticsearch Output no longer sends type when target ES >= 8.0 https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/892
    • [x] Elasticsearch Output no longer sends type when target ES >= 7.0 and user don't customize it with document_type https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/994
    • [ ] X-Pack Internal Monitoring uses type, but is possibly slated for removal in 8.0
    • [x] Investigate _type usage in elasticsearch output https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/915
  • [x] [kaise] Elasticsearch migration to composable templates and removal of _template API (upstream: https://github.com/elastic/elasticsearch/issues/53101)
    • [x] https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/944
  • [x] Switch ES plugins to Logstash specific APIs (Following System Indices https://github.com/elastic/logstash/issues/12291)
  • [ ] ~Drop _xpack API endpoint usage (and settings) https://github.com/elastic/logstash/issues/11382~
  • [ ] ~Agent Migration~
  • [x] GeoIP transition to using service.
• [ ] Logstash Breaking Changes
  • [ ] ~Internal Collection - Deprecation in favor of agent (or drop sending data directly to ES cluster) (https://github.com/elastic/logstash/issues/11169)~
  • [x] [kaise] Execution Engine - Remove Ruby Execution Engine (https://github.com/elastic/logstash/issues/11236)
  • [x] ECS - Pipelines in ECS-Compatibility mode as the default (meta: https://github.com/elastic/logstash/issues/11623)
  • [x] DOC - Document breaking changes
  • [x] [kaise] Settings - Remove "field-reference-parse" https://github.com/elastic/logstash/issues/12355: INPROGRESS
  • [ ] ~[andsel] Modules - Remove modules infrastructure and existing modules (Azure/Netflow)~
  • [x] Elasticsearch Output under default distribution requires licensed Elasticsearch (https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/875)
  • [ ] Operating System support changes in 8.0.0 https://github.com/elastic/logstash/issues/12511
  • [x] Auditing for terminology: master/slave, *list, and others https://github.com/elastic/logstash/issues/12052
  • [x] [andsel] Use LS_JAVA_HOME instead of JAVA_HOME #12725 (See elastic/elasticsearch#69149 for references in ES)
  • [ ] Review/evaluate other breaking changes (https://github.com/elastic/logstash/labels/breaking%20change) and open issues targeted for 8.0.0
• [ ] Other (e.g. Stack Features Logstash should Support)
  • [x] Elasticsearch Introduction of Hidden Indices (upstream: https://github.com/elastic/elasticsearch/pull/50452)
  • [ ] System Indices - Clean-up old implementation (https://github.com/elastic/logstash/issues/12291)
  • [ ] Runtime Fields
  • [ ] Logstash Breaking Changes
    • [ ] avoid Ruby Event wrapper - just rely on the Java org.logstash.Event in Ruby land
  • [ ] ? stack update JRuby to 9.3 (would resolve https://github.com/elastic/logstash/issues/12077)
    • [ ] Java collection patches https://github.com/elastic/logstash/issues/12800
    • [ ] actively (CI) testing against jruby-head
  • [ ] New 'major' release of Ruby filter (https://github.com/logstash-plugins/logstash-filter-ruby/pull/56)
  • [x] Document upgrade instructions

[HenkPoley]

Is the 'migration to composable templates' scripting still planned? (for setups without clashing inheritance)

Or otherwise, some kind of guidance how to move parts around. Documentation on v2 and legacy templates is currently quite separate.

Is it literally: If you don't have inheritance, then ignore PUT _component_template and "composed_of": [..] ? Then it seems pretty straightforward.

---

E.g. for the no-inheritance case, with jq on a directory of legacy template json files:

jq '{index_patterns: [."index_patterns"], template: {settings: ."settings", mappings: { properties: ."mappings"."properties"}, aliases: ."aliases"}, priority: 49, composed_of: [], version: ."version"|0}' *

It doesn't transfer "_source": { "enabled": false },. Maybe somebody knows more jq-foo than I do.

Potentially add , _meta: { description: "TODO: customize"} before the last } if you want to have a description field metadata, etc.

[yaauie]

@HenkPoley while Logstash does persist the templates it is given and it will need to change how we compose our own templates internally, more likely your question would be better addressed by the Elasticsearch folks. I'd recommend starting a thread in the Elasticsearch forum.

[yaauie]

@roaksoax I do not believe that the "Hidden Indices" feature of Elasticsearch should be classified with "Breaking Changes". It is a new feature, by which a user can now explicitly configure an index to not be available to wildcard queries. Logstash does not use this feature when creating indices (yet?) and any wildcard queries that are performed in the Elasticsearch Input and Filter plugins will continue to work as-is unless a user explicitly configures an index to not be returned.

[HenkPoley]

Ah oops, I just followed a link from the huge Elasticsearch-project Composable Templates meta ticket to here, and didn't want to spew in there.

But now I see this is a different, but related, project.

[jsvd]

Adding a terminology audit task that will result in breaking changes https://github.com/elastic/logstash/issues/12052

[karenzone]

@roaksoax From the output-elasticsearch doc: "Using the transport protocol to communicate with the cluster has been deprecated in Elasticsearch 7.0.0 and will be removed in 8.0.0." Should that effort be on this list?

[yaauie]

@karenzone neither Logstash core not any of our plugins use the deprecated Elasticsearch Transport Protocol -- we use the HTTP APIs.

[karenzone]

In that case, I'll pull that line out. Thanks, @yaauie.

[roaksoax]

This issue no longer seems relevant. I'm gonna go ahead and close it.