Add offload policies in the pulsar_namespace resource

[shodo]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Hi all! In the company I work in, we are willing to enable infinite retention on some namespace and we are planning to offload them on S3 if they reach a certain size. We want to specify different threshold quotas for each namespace and have seen the possibility of doing that using Pulsar admin as described here (https://pulsar.staged.apache.org/docs/en/2.9.0/tiered-storage-aws/#example-3). Seems that there is no property available on the pulsar_namespace resource to do the same through Terraform. Do you have any plans to add it?

Impacted resources:

pulsar_namespace

Potential Terraform Configuration

resource "pulsar_namespace" "test" {
  tenant    = pulsar_tenant.test_tenant.tenant
  namespace = "eternals"

  enable_deduplication = true

  // If defined partially, plan would show difference
  // however, none of the mising optionals would be changed
  namespace_config {
    anti_affinity                  = "anti-aff"
    max_consumers_per_subscription = "50"
    max_consumers_per_topic        = "50"
    max_producers_per_topic        = "50"
    message_ttl_seconds            = "86400"
    replication_clusters           = ["standalone"]
  }

  dispatch_rate {
    dispatch_msg_throttling_rate  = 50
    rate_period_seconds           = 50
    dispatch_byte_throttling_rate = 2048
  }

  retention_policies {
    retention_minutes    = "1600"
    retention_size_in_mb = "10000"
  }
  
  // --- NEW OFFLOAD POLICIES HERE -- //
  offload_policies {
    threshold_size_in_mb = "10000"
  }

  backlog_quota {
    limit_bytes  = "10000000000"
    limit_seconds = "-1"
    policy = "consumer_backlog_eviction"
    type = "destination_storage"
  }

  persistence_policies {
    bookkeeper_ensemble                   = 1   // Number of bookies to use for a topic, default: 0
    bookkeeper_write_quorum               = 1   // How many writes to make of each entry, default: 0
    bookkeeper_ack_quorum                 = 1   // Number of acks (guaranteed copies) to wait for each entry, default: 0
    managed_ledger_max_mark_delete_rate   = 0.0 // Throttling rate of mark-delete operation (0 means no throttle), default: 0.0
  }

  permission_grant {
    role    = "some-role"
    actions = ["produce", "consume", "functions"]
  }
}

References

[shodo]

Hi everyone! I've opened a PR to add the threshold config: https://github.com/streamnative/terraform-provider-pulsar/pull/109

It's a little different from what I've requested in this issue, since I've noticed that it was better to add the configuration on the namespace properties instead of creating a dedicated offload_policies.