Normation
Fixes #24789: Archive API for import/export doesn't know about technique
https://issues.rudder.io/issues/24789
This PR introduces import and export of categories for technique.
It adds a new file descriptors for them, category.json which mimics category.xml but in JSON for consistency with other file format in archive.
The logic to parse each format was moved to TechniqueCategoryMetadata companion object so that everything is in one place.
Since category information where not updatable by API (internal or rest) up until now, I needed to also add the relevant logic to actually save changes in /var/rudder/configuration-repository in TechniqueArchiverImpl#saveTechniqueCategory.
For the reading part, I was similarly forced to add new logic into TechniqueRevisionRepository to retrieve information corresponding to a specific revision id.
The change in ArchiveApi that ensue is a bit big, but rather systematic: I needed to add logic similar than technique/directive/etc for technique category using the added methods above.
And finally, we need to keep path information during the process to be able to reconstruct the technique category hierarchy and where techniques go.
NOTE: the revision history is horrible because the CI wasn't being nice with me. First, I encounter a system group management different on the CI and on my linux (even with docker) which lead to permission error not directly related to theses changes (just on a path that was not tested before). The solution was to retrieve the group with which tests are launched and use it as a target for perm, so that we are sure it does not break (also: it's useless on that case). Then, I encoutered a (real) bug that seems to be that the sorting of map is not the same on CI and on local machine, certainly due to different version of java using different seeds/hash algo configuration. The solution for that one is to sort the result in chunks.
Finally, it specifies that we should use jenkins user in docker to run tests, which is not directily linked to archiving, but to battle with the CI part.
There's a test in error on CI that I don't seem to have locally, it needs a new pass of work:
[2024-06-26T12:22:30.325Z] Tests in error:
[2024-06-26T12:22:30.325Z] The test lib should::save a category should::create a new file and commit if the category
does not exist(com.normation.cfclerk.services.JGitRepositoryTest): Chained(error when committing technique
category 'techniques/systemSettings/myNewCategory/category.xml',SystemError(An error
occurred,java.nio.file.FileSystemException: /tmp/test-jgit-2024-06-26T14:22:13.809+02:00/techniques/systemSettings/
myNewCategory: Operation not permitted))
[2024-06-26T12:22:30.326Z] The test lib should::save a category should::does nothing when the category already
exsits(com.normation.cfclerk.services.JGitRepositoryTest): Chained(error when committing technique category
'techniques/systemSettings/myNewCategory/category.xml',SystemError(An error
occurred,java.nio.file.FileSystemException: /tmp/test-jgit-2024-06-26T14:22:13.809+02:00/techniques/systemSettings/
myNewCategory: Operation not permitted))
So, this wasn't working for a long time because I used to get operation not permitted when setting group owner.
This was certainly due to the fact that on the CI, the default group for current user doesn't exists (and so it was not linked to a docker restriction or whatever, as initially thought).
Now I'm getting that error:
[2024-07-26T00:48:18.210Z] [error] x uploading an archive
[2024-07-26T00:48:18.210Z] [error] 'Meta Techniques' != 'System settings updated' (ArchiveApiTest.scala:638)
[2024-07-26T00:48:18.210Z] [error] Actual: Meta Techniques
[2024-07-26T00:48:18.210Z] [error] Expected: System settings updated
[2024-07-26T00:48:18.210Z] [info]
[2024-07-26T00:48:18.210Z] 02:48:18.195 [specs2-6] ERROR application.archive - Error when processing uploaded archive: Accumulated: Inconsistency: Error: technique in archive at path 'ncf_techniques/a_simple_yaml_technique/1.0' should have ID from a_simple_yaml_technique/1.0 based on pattern 'category/techniqueName/techniqueVersion' but the technique descriptor contains id: bad_bad_technique_id/1.0. You need to make both match.
[2024-07-26T00:48:18.210Z] 02:48:18.195 [specs2-6] ERROR api-processing - Accumulated: Inconsistency: Error: technique in archive at path 'ncf_techniques/a_simple_yaml_technique/1.0' should have ID from a_simple_yaml_technique/1.0 based on pattern 'category/techniqueName/techniqueVersion' but the technique descriptor contains id: bad_bad_technique_id/1.0. You need to make both match.
[2024-07-26T00:48:18.210Z] [info] + uploading an archive with a yaml technique with a mismatch between id and path stop everything
[2024-07-26T00:48:18.490Z] [info] + correctly build an archive with past revision items
[2024-07-26T00:48:18.490Z] [info]
[2024-07-26T00:48:18.490Z] [info]
[2024-07-26T00:48:18.490Z] [info] Total for specification ArchiveApiTest
[2024-07-26T00:48:18.490Z] [info] Finished in 682 ms
[2024-07-26T00:48:18.490Z] 17 examples, 18 expectations, 1 failure, 0 error
[2024-07-26T00:48:18.490Z] [info]
[2024-07-26T00:48:18.490Z] Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.285 sec <<< FAILURE!
[2024-07-26T00:48:18.490Z] uploading an archive(com.normation.rudder.rest.ArchiveApiTest) Time elapsed: 0.004 sec <<< FAILURE!
[2024-07-26T00:48:18.490Z] java.lang.Exception: 'Meta Techniques' != 'System settings updated'
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.ArchiveApiTest.$anonfun$new$219(ArchiveApiTest.scala:638)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.ArchiveApiTest.$anonfun$new$218(ArchiveApiTest.scala:639)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.ArchiveApiTest.$anonfun$new$217(ArchiveApiTest.scala:640)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.ArchiveApiTest.$anonfun$new$216(ArchiveApiTest.scala:641)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.RestTest.$anonfun$execRequestResponse$1(RestTestSetUp.scala:1037)
[2024-07-26T00:48:18.490Z] at net.liftweb.mockweb.MockWeb$.testReq(MockWeb.scala:106)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.RestTest.$anonfun$doReq$2(RestTestSetUp.scala:1016)
[2024-07-26T00:48:18.490Z] at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.RestTest.$anonfun$doReq$1(RestTestSetUp.scala:1016)
[2024-07-26T00:48:18.490Z] at net.liftweb.util.ThreadGlobal.doWith(ThreadGlobal.scala:71)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.RestTest.doReq(RestTestSetUp.scala:1015)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.RestTest.execRequestResponse(RestTestSetUp.scala:1027)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.RestTest.testBinaryPOSTResponse(RestTestSetUp.scala:1128)
[2024-07-26T00:48:18.490Z] at com.normation.rudder.rest.ArchiveApiTest.$anonfun$new$211(ArchiveApiTest.scala:633)
https://ci.normation.com/jenkins/blue/organizations/jenkins/rudder/detail/PR-5747/56/pipeline/23 Which at least seems relatated whith the PR. Still, I don't understand what's going on here for now, and it is not reproducible on local so :shrug: Will have to try again local docker etc
This PR is not mergeable to upper versions.
Since it is "Ready for merge" you must merge it by yourself using the following command:
rudder-dev merge https://github.com/Normation/rudder/pull/5747
-- Your faithful QA
Kant merge: "All our knowledge begins with the senses, proceeds then to the understanding, and ends with reason. There is nothing higher than reason."
(https://ci.normation.com/jenkins/job/merge-accepted-pr/89653/console)
