Integration tests fail after change in devfile registry

[feloy]

/kind bug

Context

This PR changes metadata in Devfiles. https://github.com/redhat-developer/odo/pull/6137#issuecomment-1251953720

These changes break the integration tests

Acceptance criteria

See https://github.com/redhat-developer/odo/issues/6138#issuecomment-1270171350

• [ ] Spin up our own Devfile registry, that we control. Odo tests should test odo features, not that the Devfile stacks are valid/usable with odo, which should be the scope of Devfile registry tests
• [ ] Have one or just a few tests that validate that that odo can communicate with the default Devfile registry. They should not test other odo features, but could just check that the right Devfile has been downloaded.
• [ ] Remove the NGINX server used for caching Devfiles (cf. #5922)
• [ ] We should not use starter projects because we might have issues there as well. Ensure that we use projects in our local code base.

[rm3l]

https://github.com/redhat-developer/odo/pull/6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

[kadel]

There is also another thing. This change only broke our tests, and not odo itself. But next time there could be a change that breaks odo. We should consider running our tests against https://registry.stage.devfile.io/ every change in devfile registry gets first deployed into staging registry before going to production, we should try to catch as much as we can before changes are in production. Using staging could help with that.

[rm3l]

There is also another thing. This change only broke our tests, and not odo itself. But next time there could be a change that breaks odo.

That's a good point indeed.

We should consider running our tests against registry.stage.devfile.io every change in devfile registry gets first deployed into staging registry before going to production, we should try to catch as much as we can before changes are in production. Using staging could help with that.

I think we are already using the staging registry in our tests: https://github.com/redhat-developer/odo/blob/main/tests/helper/helper_generic.go#L326-L334 This is why we caught this issue today (https://github.com/devfile/registry/pull/123 was merged yesterday); I think this change has not been deployed yet to the production registry.

[kadel]

I think we are already using the staging registry in our tests: https://github.com/redhat-developer/odo/blob/main/tests/helper/helper_generic.go#L326-L334 This is why we caught this issue today (devfile/registry#123 was merged yesterday); I think this change has not been deployed yet to the production registry.

I forgot about this. That is good. So that is good that it broke our tests, it worked as expected :-D, it just came in the worst possible time it could.

[valaparthvi]

#6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

I remember we discussed some robust solution to this, but I can't recall what. Can you help me recall it? @rm3l @feloy

[feloy]

#6140 introduced a quick fix to unblock the current (v3.0.0-rc2) release process, but we need to find a more robust solution for such issues that can arise again at any later time.

I remember we discussed some robust solution to this, but I can't recall what. Can you help me recall it? @rm3l @feloy

We discussed two possibilities:

• pass a devfile version when running odo init (are different versions stored in registry?)
• during integration tests, read the content of the devfile to get the values to assert, instead of hardcoding them in the test source code

[valaparthvi]

pass a devfile version when running odo init (are different versions stored in registry?)

I don't think this would work. There is only one version at a time. I say this by checking odo registry, if that is what you are pointing at.

odo registry output

$ odo registry
 NAME                          REGISTRY         DESCRIPTION                                 
 dotnet50                      StagingRegistry  Stack with .NET 5.0                         
 dotnet60                      StagingRegistry  Stack with .NET 6.0                         
 dotnetcore31                  StagingRegistry  Stack with .NET Core 3.1                    
 go                            StagingRegistry  Go is an open source programming languag... 
 java-maven                    StagingRegistry  Upstream Maven and OpenJDK 11               
 java-openliberty              StagingRegistry  Java application Maven-built stack using... 
 java-openliberty-gradle       StagingRegistry  Java application Gradle-built stack usin... 
 java-quarkus                  StagingRegistry  Quarkus with Java                           
 java-springboot               StagingRegistry  Spring Boot® using Java                     
 java-vertx                    StagingRegistry  Upstream Vert.x using Java                  
 java-websphereliberty         StagingRegistry  Java application Maven-built stack using... 
 java-websphereliberty-gradle  StagingRegistry  Java application Gradle-built stack usin... 
 java-wildfly                  StagingRegistry  Upstream WildFly                            
 java-wildfly-bootable-jar     StagingRegistry  Java stack with WildFly in bootable Jar ... 
 nodejs                        StagingRegistry  Stack with Node.js 16                       
 nodejs-angular                StagingRegistry  Angular is a development platform, built... 
 nodejs-nextjs                 StagingRegistry  Next.js gives you the best developer exp... 
 nodejs-nuxtjs                 StagingRegistry  Nuxt is the backbone of your Vue.js proj... 
 nodejs-react                  StagingRegistry  React is a free and open-source front-en... 
 nodejs-svelte                 StagingRegistry  Svelte is a radical new approach to buil... 
 nodejs-vue                    StagingRegistry  Vue is a JavaScript framework for buildi... 
 php-laravel                   StagingRegistry  Laravel is an open-source PHP framework,... 
 python                        StagingRegistry  Flask is a web framework, it’s a Pytho...   
 python-django                 StagingRegistry  Django is a high-level Python web framew... 

during integration tests, read the content of the devfile to get the values to assert, instead of hardcoding them in the test source code

I am inclined towards this solution, but then again, this cannot be done for all the test cases. We have been using local devfiles for most of our cases, but some tests in odo init and odo registry still call the command with devfile name to fetch it.

[rm3l]

Summary of Cabal and grooming discussions (2022-10-06):

• Spin up our own Devfile registry, that we control. Odo tests should test odo features, not that the Devfile stacks are valid/usable with odo, which should be the scope of Devfile registry tests
• Have one or just a few tests that validate that that odo can communicate with the default Devfile registry. They should not test other odo features, but could just check that the right Devfile has been downloaded.
• Remove the NGINX server used for caching Devfiles (cf. #5922)
• We should not use starter projects because we might have issues there as well. Ensure that we use projects in our local code base.

[valaparthvi]

Since this issue has come up recently way more frequently than we'd like, following changes have been implemented so far:

Spin up our own Devfile registry, that we control. Odo tests should test odo features, not that the Devfile stacks are valid/usable with odo, which should be the scope of Devfile registry tests

• Setup our custom devfile registry on IBM cloud for our CI, this ensures that any issue caused by the Staging Registry does not block us from continuous development. See: https://github.com/redhat-developer/odo/pull/6705. On IBM cloud, we export the custom devfile registry via DEVFILE_REGISTRY and our tests capture this env var if available and use this custom registry.

Have one or just a few tests that validate that that odo can communicate with the default Devfile registry. They should not test other odo features, but could just check that the right Devfile has been downloaded.

• We still want to catch any problems from the Staging Registry before they propagate to production, and so we continue to run our nightly tests with Staging Registry. See: https://github.com/redhat-developer/odo/pull/6725

Remove the NGINX server used for caching Devfiles (cf. https://github.com/redhat-developer/odo/pull/5922)

• This was implemented, but then reversed because of frequent flaky behavior, IIRC.

We should not use starter projects because we might have issues there as well. Ensure that we use projects in our local code base.

• This has been taken care of.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 90 days. Stale issues will be closed after an additional 30 days of inactivity.

[github-actions[bot]]

This issue was closed because it has been inactive for 30 days since being marked as stale.

[rm3l]

/remove-lifecycle stale /remove-lifecycle rotten /reopen

Reopening because it is partially implemented, and there remains a few tasks still open.

[openshift-ci[bot]]

@rm3l: Reopened this issue.

In response to this:

/remove-lifecycle stale /remove-lifecycle rotten /reopen

Reopening because it is partially implemented, and there remains a few tasks still open.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.