Repository not assigned to project

[martinm82]

Describe the bug

We are using Artifactory Cloud 7.34.4 and encountered the problem that the repository is not assigned to a project. After running terraform apply it says all changes were applied but running it again it shows that the changes (project_key) will be applied again.

In the TF logs we see the following error:

2022-02-22T15:25:48.792+0100 [WARN]  Provider "provider[\"registry.terraform.io/jfrog/artifactory\"]" produced an unexpected new value for artifactory_local_helm_repository.del_team_x_helm_local, but we are tolerating it because it is using the legacy plugin SDK.                    The following problems may be the cause of any confusing errors from downstream operations:
      - .project_key: was cty.StringVal("del"), but now cty.StringVal("")

Requirements for and issue

• [ ] A fully functioning terraform snippet that can be copy&pasted (no outside files or ENV vars unless that's part of the issue)
• [x] Your version of artifactory (you can curl it at $host/artifactory/api/system/version: 7.34.4
• [x] Your version of terraform: 1.0.11

Expected behavior Repository is assigned to project.

Additional context Add any other context about the problem here.

[alexhung]

@martinm82 I have not been able to reproduce this issue. I am able to create a project with no repos and then create a local repo with project_key. Then after running terraform apply the first time, all the resources are there.

terraform {
  required_providers {
    artifactory = {
      source  = "registry.terraform.io/jfrog/artifactory"
      version = "2.20.1"
    }

    project = {
      source  = "registry.terraform.io/jfrog/project"
      version = "1.0.2"
    }
  }
}

provider "artifactory" {
  //  supply ARTIFACTORY_USERNAME, _PASSWORD and _URL as env vars
}

provider "project" {
  //  supply ARTIFACTORY_USERNAME, _PASSWORD and _URL as env vars
}

resource "project" "myproject" {
  key          = "myproj"
  display_name = "My Project"
  description  = "My Project"
  admin_privileges {
    manage_members   = true
    manage_resources = true
    index_resources  = true
  }
  max_storage_in_gibibytes   = 10
  block_deployments_on_limit = false
  email_notification         = false
}

resource "artifactory_local_generic_repository" "myproj-generic-repo" {
  key         = "myproj-test-generic-repo"
  project_key = "myproj"
  project_environments = ["DEV"]
  property_sets        = []

  depends_on = [project.myproject]
}

When I run terraform plan next, I see the following with project resource state need to be updated with the new repo with terraform apply -refresh-only:

alexh@alexh-mac-2 terraform-provider-artifactory % terraform plan
project.myproject: Refreshing state... [id=myproj]
artifactory_local_generic_repository.myproj-generic-repo: Refreshing state... [id=myproj-test-generic-repo]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # artifactory_local_generic_repository.myproj-generic-repo has changed
  ~ resource "artifactory_local_generic_repository" "myproj-generic-repo" {
        id                   = "myproj-test-generic-repo"
      + property_sets        = []
        # (9 unchanged attributes hidden)
    }

  # project.myproject has changed
  ~ resource "project" "myproject" {
        id                         = "myproj"
      + repos                      = [
          + "myproj-test-generic-repo",
        ]
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these
changes.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only plan:
  terraform apply -refresh-only
╷
│ Warning: Argument is deprecated
│
│   with provider["registry.terraform.io/jfrog/artifactory"],
│   on sample.tf line 15, in provider "artifactory":
│   15: provider "artifactory" {
│
│ Xray and projects functionality will not work with any auth method other than access tokens (Bearer)
╵

[martinm82]

@alexhung which Terraform version are you using? Did you try to first create a repository and afterwards add the project_key?

[martinm82]

Also you are using some 2.20.1 version which is not yet released.

[alexhung]

@martinm82 Our local dev install script increment the patch version automatically, even without any code change. 2.20.0 is released version which is what I used to test

[alexhung]

@martinm82 You're right. Updating an existing repo with project_key doesn't assign to the project. Looks like an Artifactory API bug.

The provider calls the repo update API with project key in projectKey field. We receive 200 status code from Artifactory but the repo does not actually got assigned to the project!

This project assignment works using repo creation API though.

Thanks for the report. I'll escalate this to our R&D team.

[alexhung]

@martinm82 An alternative is to add the repo key to project.repos list in HCL. This will assign existing repos to that project.

[martinm82]

@alexhung Thanks for diving into the problem. We will for now use the project.repos list although we wanted to keep it outside of the project configurations.

[martinm82]

Hi @alexhung,

we had at the beginning created 300+ repositories without the project_key. In the next step we started to use the project_key on all of the repository resources. Due to the above issue we were not able to assign the repositories to the projects.

Therefore we started now to assign the repositories directly in the projects but now are facing some issues that not all repositories are assigned to projects even though Terraform states that all have been updated.

Apart from that we see as well changes like these:

  # artifactory_local_pypi_repository.pypi_local will be updated in-place
  ~ resource "artifactory_local_pypi_repository" "pypi_local" {
        id                   = "pypi-local"
      - project_key          = "svc" -> null
        # (9 unchanged attributes hidden)
    }

These changes pop up after each terraform apply. It looks like somehow the project key is assigned and cannot be removed.

The project itself references this repository as well:

resource "project" "service_project" {
  key          = "svc"
  display_name = "Service Project"
  description  = ""

  admin_privileges {
    manage_members   = false
    manage_resources = false
    index_resources  = false
  }

  max_storage_in_gibibytes   = 10
  block_deployments_on_limit = false
  email_notification         = true

  repos = [
    artifactory_local_pypi_repository.pypi_local.key
  ]
}

Any idea what is wrong?

[martinm82]

After running it against our Artifactory SaaS environment I get now the following error:

│ Error:
│ 400 POST https://tomtom.jfrog.io/artifactory/api/repositories/my-repository
│ {
│   "errors" : [ {
│     "status" : 400,
│     "message" : "Could not merge and save new descriptor [org.jfrog.common.ExecutionFailed: Last retry failed: Should update revision 2149. Not trying again]"
│   } ]
│ }

We have in total 21 projects and 340 repositories.

[chb0github]

The error you get basically means that you've overwhelmed your RT instance and one of the necessary resources hasn't been fully created yet. We've built in retry logic into several endpoints and this was one of the ones we've handled. When you ran this, did you see any log output about retrying? It attempts to create the resource 5 times before failing.

if so, can you send that? It's set to 5, and that's always been enough, but maybe not this time

[chb0github]

To be clear, we have 2 problems here:

1. You can't assign repos to projects
2. Your getting that 400 error (which is un related and we do need logs).

[martinm82]

Regarding 2.) We have no logs anymore due to SaaS environment. Also since this task was quite heavy and caused additional stress on the service we decided for now not to "reproduce" it again. Once we sort out things we will come back to this point.

Regarding 1.) I think there is another issue. If you have once started using project_key attribute on the repository resources and you decide to use the workaround and assign repos in projects, Terraform will constantly report changes on the repository project_key

- project_key              = "mykey" -> null

We haven't figured out how to get rid of this change in a clean way. Only way so far was to use lifecycle and ignore_changes on project_key.

Should this issue be tracked in a separate issue?

[alexhung]

Should this issue be tracked in a separate issue?

Let's continue tracking problem #1 with this issue.

@dasmanas is working on problem #2. We can create a separate issue for that.

[alexhung]

@martinm82 @dasmanas See https://github.com/jfrog/terraform-provider-project/issues/27

[oallauddin]

@alexhung Is there a ticket somewhere to fix the repo update API ?

[alexhung]

@oallauddin There's an internal ticket for the R&D team for this issue.

[mcaulifn]

@alexhung Do you have an update on the internal ticket?

[alexhung]

@mcaulifn No update. I'll try to follow up with the internal engineering team.

[coreyjackwilson]

@alexhung is there any update on this issue?

I am experiencing a similar issue where getting repos created via the provider will randomly result with the repository not being associated to a project, even though the project_key is specified

[alexhung]

@mcaulifn @coreyjackwilson @oallauddin FYI, I've opened this PR which will remove the parallel requests to Projects API that causes system error. Now the requests to move repos into a project is performed sequentially with the consequences that the Terraform apply process takes longer.

As for this issue, there is no plan to implement support for associating a repo with projectKey with Artifactory update repo API. Therefore, I'll implement the extra projects API call to do that when the project_key attribute changes for the repo.