Fix for Errors when Installing Content Packs with Streams

[ryan-carroll-graylog]

Fixes the errors that were occurring when installing of a Content Pack containing Event Definitions or Dashboards that use default Streams.

Description

Underlying issue: Prior to this PR, when a Content Pack entity referenced a Stream, the installation logic expected the referenced Stream to be included in the Content Pack as well.

However, System Streams have intentionally been left out when a Content Pack is created since this PR (For including Pipelines with default Streams in Content Packs): https://github.com/Graylog2/graylog2-server/pull/6873

This PR follows a similar approach used with Pipelines in the PR above: checking if the Content Pack Entity's referenced Stream is a System Stream, and if it is adding it without trying to resolve it to a Native Content Pack Entity.

Motivation and Context

Several customers were reporting that they were unable to install Content Packs that contained Event Definitions that use default Stream. More details in Issue: https://github.com/Graylog2/graylog2-server/issues/7212

Further investigation revealed that this same issue was occurring for Content Packs with Dashboards with Widgets that use System Streams.

How Has This Been Tested?

Tested in local development environment by creating Content Packs including Event Definitions and Dashboards that use System Streams.

Steps to reproduce:

1. Create an Event Definition and/or Dashboard that with "All Messages" Stream selected for the Entity.
2. Create a Content Pack including the Entity created above.
3. Attempt to Install the Content Pack Created above.

Alternatively:

1. Upload the following Content Pack containing an Event Definition and a Dashboard using the "All Messages" Stream (".txt" added for GitHub to allow upload): content-pack-with-event-def-and-dashboard-with-default-stream.json.txt
2. Attempt to Install the Content Pack uploaded above.

Screenshots (if appropriate):

Types of changes

• [X] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Refactoring (non-breaking change)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• [X] My code follows the code style of this project.
• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [X] I have read the CONTRIBUTING document.
• [X] I have added tests to cover my changes.

[ryan-carroll-graylog]

I went with this approach because it was similar to what was done with Pipelines but am open to other suggestions.

A few concerns I had:

• Is it possible that a System Stream referenced in a Content Pack wont exist when the Pack is installed?
• Are there other Entity's where this needs to be addressed? (I looked but may have missed some)

[boosty]

Hi @ryan-carroll-graylog, thanks for working on this! I removed myself as a reviewer – this should better be done by the engineers.

[ryan-carroll-graylog]

@ryan-carroll-graylog It looks like in addition to the default All Messages stream, there are 3 more default streams (events, system events, and indexing failures):

https://github.com/Graylog2/graylog2-server/blob/dd13876f41e6b1826475ef01c53d28b27b1d1ba9/graylog2-server/src/main/java/org/graylog2/plugin/streams/Stream.java#L34-L46

I think the changes here cover uses of all of those streams, but I am seeing this error when attempting to import a content pack that contains a dashboard with widget referencing the Events system stream. Seems to work with the All Messages stream.

Here's the content pack I am using:

content-pack-50a0ddc5-e023-4a3b-8ccc-3aaa377b77c5-1.json.txt

And the stacktrace:

2022-09-12 10:45:49,949 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
org.graylog2.contentpacks.exceptions.ContentPackException: Failed to install content pack <50a0ddc5-e023-4a3b-8ccc-3aaa377b77c5/1>
	at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:159) ~[classes/:?]
	at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:100) ~[classes/:?]
	at org.graylog2.rest.resources.system.contentpacks.ContentPackResource.installContentPack(ContentPackResource.java:295) ~[classes/:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [jersey-server-2.32.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [jersey-common-2.32.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [jersey-common-2.32.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [jersey-common-2.32.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [jersey-common-2.32.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [jersey-common-2.32.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [jersey-common-2.32.jar:?]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [jersey-server-2.32.jar:?]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680) [jersey-server-2.32.jar:?]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [jersey-container-grizzly2-http-2.32.jar:?]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [grizzly-http-server-2.4.4.jar:2.4.4]
	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [metrics-core-4.1.9.jar:4.1.9]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.graylog2.contentpacks.exceptions.ContentPackException: Could not find matching stream id: 000000000000000000000002
	at org.graylog2.contentpacks.model.entities.QueryEntity.lambda$shallowMappedFilter$4(QueryEntity.java:156) ~[classes/:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at org.graylog2.contentpacks.model.entities.QueryEntity.lambda$shallowMappedFilter$5(QueryEntity.java:161) ~[classes/:?]
	at java.util.Optional.map(Optional.java:260) ~[?:?]
	at org.graylog2.contentpacks.model.entities.QueryEntity.shallowMappedFilter(QueryEntity.java:147) ~[classes/:?]
	at org.graylog2.contentpacks.model.entities.QueryEntity.toNativeEntity(QueryEntity.java:174) ~[classes/:?]
	at org.graylog2.contentpacks.model.entities.SearchEntity.lambda$toNativeEntity$1(SearchEntity.java:126) ~[classes/:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.util.Iterator.forEachRemaining(Iterator.java:133) ~[?:?]
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?]
	at org.graylog2.contentpacks.model.entities.SearchEntity.toNativeEntity(SearchEntity.java:127) ~[classes/:?]
	at org.graylog2.contentpacks.facades.ViewFacade.decode(ViewFacade.java:141) ~[classes/:?]
	at org.graylog2.contentpacks.facades.ViewFacade.createNativeEntity(ViewFacade.java:127) ~[classes/:?]
	at org.graylog2.contentpacks.ContentPackService.installContentPack(ContentPackService.java:150) ~[classes/:?]
	... 29 more

Ah nice catch! I don't think the dashboards I was testing with had QueryEntitys, but it looks like the System Streams need to be handled there as well.

[ryan-carroll-graylog]

@danotorrey @kingzacko1 Instead of trying to try and handle each facade individually I decided to move the logic to a ContentPackService, adding the system streams before the facades are called during content pack installation. Essentially reverting what's done during export here: https://github.com/Graylog2/graylog2-server/blob/385f035a867c402bbc77bf43324a79b7ca0a058e/graylog2-server/src/main/java/org/graylog2/contentpacks/ContentPackService.java#L365-L367

It seems to be working in all the cases that were failing before and should work for any entities added in the future. Lmk if you see any issues with this approach or have suggestions for improvement.