Generating Manifests and Metadata page `--kustomize` flag not available

[Jaanki]

trafficstars

I am trying to create a bundle for submariner-operator by following the official "Generating manifests and metadata" guide. The page describes about passing --kustomize=false to disable the interactive prompt. However, when running the command generate kustomize manifests --kustomize=false, it throws an unknown flag error.

$ ./bin/operator-sdk generate kustomize manifests --kustomize=false
Error: unknown flag: --kustomize
Usage:
  operator-sdk generate kustomize manifests [flags]

Examples:

  $ operator-sdk generate kustomize manifests

  Display name for the operator (required):
  > memcached-operator
  ...

  $ tree config/manifests
  config/manifests
  ├── bases
  │   └── memcached-operator.clusterserviceversion.yaml
  └── kustomization.yaml

  # After generating kustomize bases and a kustomization.yaml, you can generate a bundle or package manifests.

  # To generate a bundle:
  $ kustomize build config/manifests | operator-sdk generate bundle --version 0.0.1

  # To generate package manifests:
  $ kustomize build config/manifests | operator-sdk generate packagemanifests --version 0.0.1


Flags:
      --apis-dir string     Root directory for API type defintions
  -h, --help                help for manifests
      --input-dir string    Directory containing existing kustomize files
      --interactive         When set to false, if no kustomize base exists, an interactive command prompt will be presented to accept non-inferrable metadata
      --output-dir string   Directory to write kustomize files
      --package string      Package name
  -q, --quiet               Run in quiet mode

Global Flags:
      --plugins strings   plugin keys to be used for this subcommand execution
      --verbose           Enable verbose logging

FATA[0000] unknown flag: --kustomize      

operator-sdk version: 1.23.0

$ ./bin/operator-sdk version                                       
operator-sdk version: "v1.23.0", commit: "1eaeb5adb56be05fe8cc6dd70517e441696846a4", kubernetes version: "1.24.2", go version: "go1.18.5", GOOS: "linux", GOARCH: "amd64"

kustomize version: 3.10.0 kubebuilder version: v3

[varshaprasad96]

@Jaanki generate kustomize manifests does not have --kustomize flag. You can use --interactive=false to skip the prompts. Here is more on the command: https://v1-23-x.sdk.operatorframework.io/docs/cli/operator-sdk_generate_kustomize_manifests/

[everettraven]

@Jaanki I took a look at this and using the --interactive=false flag this worked fine for me. I even cloned the submariner-io/submariner-operator source. After updating the PROJECT config version to 3 instead of 3-alpha I was able to successfully run operator-sdk generate kustomize manifests --interactive=false.

Since I was unable to replicate the issue, would you mind sharing all the steps to reproduce this issue? Thanks!

[Jaanki]

@everettraven hey, thanks for trying it. I have this PR which updates the PROJECT config to v3 and the operator-sdk version to 1.23.0. After these changes when I run operator-sdk generate kustomize manifests --interactive=false (there is also a make kustomization target for it), it runs indefinitely.

[Jaanki]

Also another doc related issue: kustomize files section in the official guide suggests that kustomization.yaml be under config/manifests/bases/ but shouldn't it be under config/manifests/?

[Jaanki]

PROJECT config being different in these 2 scenarios might be a problem.

What I have is (generated by running operator-sdk init --domain submariner.io --repo github.com/submariner-io/submariner-operator with sdk v1.21.1 (and then adding the APIs)

domain: submariner.io
layout:
- go.kubebuilder.io/v3
plugins:
  manifests.sdk.operatorframework.io/v2: {}
  scorecard.sdk.operatorframework.io/v2: {}
projectName: submariner-operator
repo: github.com/submariner-io/submariner-operator
resources:
- api:
    crdVersion: v1
    namespaced: true
  controller: true
  domain: submariner.io
  kind: Broker
  path: github.com/submariner-io/submariner-operator/api/v1alpha1
  version: v1alpha1
- api:
    crdVersion: v1
    namespaced: true
  controller: true
  domain: submariner.io
  kind: Submariner
  path: github.com/submariner-io/submariner-operator/api/v1alpha1
  version: v1alpha1
- api:
    crdVersion: v1
    namespaced: true
  controller: true
  domain: submariner.io
  kind: ServiceDiscovery
  path: github.com/submariner-io/submariner-operator/api/v1alpha1
  version: v1alpha1
version: "3"

and what operator-sdk alpha config-3alpha-to-3 generates is

domain: submariner.io
layout: go.kubebuilder.io/v2
projectName: submariner
repo: github.com/submariner-io/submariner-operator
version: "3"
multigroup: true
plugins:
  go.sdk.operatorframework.io/v2-alpha: {}

plugins section and kubebuilder based layout are different.

[everettraven]

Also another doc related issue: kustomize files section in the official guide suggests that kustomization.yaml be under config/manifests/bases/ but shouldn't it be under config/manifests/?

Yep, thanks for catching that!

PROJECT config being different in these 2 scenarios might be a problem.

Oh yes, that definitely seems like a problem. for now, I would recommend attempting to update the version in the PROJECT file manually to version: "3" instead of using operator-sdk alpha config-3alpha-to-3 for now and seeing how that does.

@Jaanki All of these are valid issues, and once we figure out the solution to get this working for you we can create a few different new issues to make tracking each of the fixes for these issues a bit easier. Thanks for reporting your findings!

[everettraven]

After these changes when I run operator-sdk generate kustomize manifests --interactive=false (there is also a make kustomization target for it), it runs indefinitely.

I am not sure why it would be running indefinitely. As I mentioned I was able to clone the submariner project and it worked fine for me with the 1.23.0 release. I wonder if it has anything to do with needing to migrate to the newest version of SDK and potentially missing a migration step? If you are upgrading from a different SDK version, did you go through all the migration docs here?: https://v1-23-x.sdk.operatorframework.io/docs/upgrading-sdk-version/

[everettraven]

@Jaanki were you able to get this resolved?

[Jaanki]

@everettraven yes and no. I renamed csv file from submariner.clusterserviceversion.yaml to submariner-operator.clusterserviceversion.yaml and corresponding kustomization.yaml changes. After this, operator-sdk generate kustomize manifests -q runs non-interactively even without --interactive=false flag but it never completes. So the never completes part is what is not working now. This is the PR that has these changes.

[everettraven]

@Jaanki thanks for the update and sharing the PR. I will take a look at it and see what I can find!

[everettraven]

@Jaanki so I took a little bit of time to debug this issue in the subcommand, and from what I can tell the internal portion of code that gets reached and is infinitely looping is here: https://github.com/operator-framework/operator-sdk/blob/8fc9626cf19d6f8ea8a4251b22928dbb200deda0/internal/generate/clusterserviceversion/bases/definitions/ast.go#L104-L174

I haven't had the chance to diagnose exactly what is causing this loop to not exit specifically when attempting to generate the submariner bundle but at least I now know what is infinitely looping :)

I will put it on my list of todos to take a more detailed look into this and diagnose the exact reason why the loop is running infinitely.

[Jaanki]

@everettraven Thanks for taking a look at this. I also want to point out that my csv file has empty customresourcedefinitions. Even with non-empty customresourcedefinitions, there is infinite looping.

[Jaanki]

could any of the config/* files misformed? I am just trying to list things that could be wrong. This is a high priority task to get the bundle generated. :)

[everettraven]

@Jaanki Doing a little more debugging using the submariner project I was able to find that the loop is happening when trying to process the following fields in the SubmarinerStatus struct: https://github.com/submariner-io/submariner-operator/blob/dc55d7bee82eba362e744672d1c5a18b32e9ac65/api/v1alpha1/submariner_types.go#L85-L88

If I comment those lines out in the SubmarinerStatus then operator-sdk generate kustomize manifests succeeds. When they are uncommented the command will loop infinitely.

I will continue digging to see if I can figure out exactly where it is hitting an issue with this and update you once I am able to figure it out.

[everettraven]

@Jaanki So I believe I managed to resolve the issue locally. It seems that the loop for the operator-sdk generate kustomize manifests command wasn't able to properly handle non-array based struct fields when processing the XxxSpec and XxxStatus fields. I am going to work on getting a PR up for this problem after verifying this with another operator.

[everettraven]

@Jaanki So after attempting to create a fix I did a little bit more digging and discovered that the problem is that the code that processes child elements isn't able to tell the difference between types with the same name in different packages. So for example, the custom DaemonSetStatus type that submariner operator defines in the v1alpha1 package is the same name as the appsv1.DaemonSetStatus type that is used in the custom type. Due to this it keeps triggering the processing loop of the DaemonSetStatus type and entering an infinite loop.

The solution seems to be renaming the types in the API that you are using to wrap other types of the same name. So in this case you could change:

• https://github.com/submariner-io/submariner-operator/blob/dc55d7bee82eba362e744672d1c5a18b32e9ac65/api/v1alpha1/submariner_types.go#L123-L128 to something like DaemonSetStatusWrapper
• https://github.com/submariner-io/submariner-operator/blob/dc55d7bee82eba362e744672d1c5a18b32e9ac65/api/v1alpha1/submariner_types.go#L119-L121 to something like LoadBalancerStatusWrapper

I tested making changes like the above to my local clone of the submariner operator and that seemed to work for running operator-sdk generate kustomize manifests against it. I even tested with the submariner-operator's make bundle target and the manifest generation step succeeded.

I hope this helps!

As a side note, what seems like needs to be done to close out this issue is to update the documentation to mention that a local type can not have the same name of an imported type when using operator-sdk generate kustomize manifests

[Jaanki]

@everettraven hey, thanks for looking into this. Yes, after renaming the fields, operator-sdk generate kustomize manifests completes and does not loop any longer. I am able to create to the bundle.