kibana icon indicating copy to clipboard operation
kibana copied to clipboard

[Spaces] Add Scout tests using Saved Objects API `_import` endpoint in different spaces

Open csr opened this issue 3 weeks ago • 2 comments

This PR adds new Scout tests using the Saved Objects _import endpoint in different spaces (default space and custom spaces).

While the initial goal was to migrate all FTR tests belonging to the x-pack/platform/test/saved_object_api_integration/spaces_only/config.ts test config to Scout, it turned out that:

  • Several Saved Object APIs operations are deprecated, except for some (e.g. _import and _export). I didn't deem worth our time to move
  • The FTR test suite is quite involved and it is importing an archive with outdated saved objects types. A 1:1 migration would involve rearchitecting test suite.

With this said, and despite the limited coverage, I think it's still worth to add these Scout tests increase the number of Scout tests in the Kibana Security area. This will let us additional data points to see how Scout performs with custom spaces. This PR is no longer a "migration" because no FTR test is deleted.

csr avatar Nov 26 '25 11:11 csr

@csr Thank you for putting this together! I have some questions regarding the scope and how we should plan to continue the effort of migrating tests to scout.

This suite focuses purely on space isolation and namespace behavior, testing how objects are created, retrieved, updated, and deleted across different Kibana spaces. ... most Saved Object API operations are deprecated so they aren't used in the new Scout tests, except for the bulk delete operation. The import and export APIs aren't deprecated so these tests focus especially on those.

I noticed that this PR covers testing the import API exclusively, and the description might just be out of date.

  • Can we update the title and description of the PR for the sake of posterity?
  • Are we limited to testing only public APIs with Scout?
  • How would you recommend we move forward with Scout test coverage knowing that the public APIs are deprecated, but the operations of the SO repository (SOR) are not? It will be important for us to keep or improve the coverage we have now with the pre-scout FTR to ensure that any changes made in the SOR do not introduce regressions. Maybe a good topic for our next cross-team sync.

jeramysoucy avatar Dec 09 '25 13:12 jeramysoucy

Hey @jeramysoucy, thank you very much for your feedback! We've synced offline, but I also wanted to follow up here with our discussion:

Can we update the title and description of the PR for the sake of posterity?

Good point, I have updated both the title and description. Feel free to suggest further improvements 👍

Are we limited to testing only public APIs with Scout?

We're not limited to testing public APIs with Scout. We encourage teams to test both public and private APIs.

How would you recommend we move forward with Scout test coverage knowing that the public APIs are deprecated, but the operations of the SO repository (SOR) are not?

I agree we should maintain the FTR test suite to not lose coverage 💯

I assume Saved Object Operations are accessible via the kbnClient (correct me if I'm wrong). In that case, Scout API tests aren't likely the way to go to test them as they shouldn't rely on kbnClient to test the final endpoint. Perhaps the solution here is to test Saved Object Operations using Jest unit integration tests (those spin up clusters).

csr avatar Dec 10 '25 17:12 csr

:yellow_heart: Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #1 / GroupedAlertsTable resets all levels pagination when global query updates
  • [job] [logs] Jest Tests #1 / GroupedAlertsTable resets all levels pagination when selected group changes
  • [job] [logs] Jest Tests #1 / GroupedAlertsTable resets innermost level's current page when that level's page size updates
  • [job] [logs] Jest Tests #1 / GroupedAlertsTable resets only most inner group pagination when its parent groups open/close
  • [job] [logs] Jest Tests #1 / GroupedAlertsTable resets outermost level's current page when that level's page size updates
  • [job] [logs] Jest Tests #1 / useQueryAlerts fetch alerts data
  • [job] [logs] Jest Tests #1 / xy style settings Axis settings should pass in endzone visibility setter and current sate for time chart
  • [job] [logs] Jest Tests #1 / xy style settings Axis settings should pass in extent information

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/scout 216 221 +5
Unknown metric groups

API count

id before after diff
@kbn/scout 567 572 +5

History

  • :broken_heart: Build #371306 failed 5c7270d816faf616a8c16e3b7082ac6c01f18cbf
  • :broken_heart: Build #371169 failed 050908b4e29682681c2df96d43d8992b917d27c9
  • :broken_heart: Build #367870 failed 2d47a96258aeea5db27e35d8988ac9454d3b0503
  • :broken_heart: Build #367820 failed 857acd13429dd89db4beb0d947ee30394152ce99
  • :broken_heart: Build #367039 failed 960498156a8d7205f1c2b8cbfab6d5948302e63c

cc @csr

elasticmachine avatar Dec 11 '25 13:12 elasticmachine

Starting backport for target branches: 8.19, 9.1, 9.2

https://github.com/elastic/kibana/actions/runs/20139115681

kibanamachine avatar Dec 11 '25 15:12 kibanamachine

💔 All backports failed

Status Branch Result
8.19 Backport failed because of merge conflicts
9.1 Backport failed because of merge conflicts
9.2 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 244323

Questions ?

Please refer to the Backport tool documentation

kibanamachine avatar Dec 11 '25 16:12 kibanamachine

Friendly reminder: Looks like this PR hasn’t been backported yet. To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label. You can also create backports manually by running node scripts/backport --pr 244323 locally cc: @csr

kibanamachine avatar Dec 12 '25 16:12 kibanamachine

💚 All backports created successfully

Status Branch Result
9.2
9.1
8.19

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

csr avatar Dec 15 '25 08:12 csr

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. cc: @csr

kibanamachine avatar Dec 15 '25 16:12 kibanamachine

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. cc: @csr

kibanamachine avatar Dec 16 '25 17:12 kibanamachine

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. cc: @csr

kibanamachine avatar Dec 17 '25 18:12 kibanamachine

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. cc: @csr

kibanamachine avatar Dec 18 '25 18:12 kibanamachine

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. cc: @csr

kibanamachine avatar Dec 19 '25 19:12 kibanamachine