New SDK breaks centralized resource schemas

[Doridian]

Module version

0.17.0

Use-cases

My Terraform provider offers most resources both as a data source and resource. Before this version, I could easily have one place these schemas were in code, to ensure they are both consistent and up to date.

With this new version it seems I have to pretty much duplicate all that code (quite literally, because 99% of the difference is the package the schema comes from)

An example of my use-case can be found here: https://github.com/Doridian/terraform-provider-hexonet/blob/a965c1984e9ef5264f09a97ffd50fd785dbdb24d/hexonet/shared_domain.go#L24

My current attempt can be seen here: https://github.com/Doridian/terraform-provider-hexonet/blob/fix-for-new-sdk/hexonet/shared_domain.go#L30

This of course does not work as data sources cannot accept this type in their schema at all.

Attempted Solutions

• Tried using common base type, which is internal and therefor unusable (fwschema)
• Duplicated code, which is ugly

Proposal

Allow some way to create one schema that can be applied to both resources and data sources

[bflad]

Hi @Doridian 👋 Thank you for raising this concern. It appears that you're looking to simplify your provider development in cases where a data source lookup is fairly symmetrical to managing a resource in data terms. To support this, you are taking the "shared" schema and slightly modifying it for the data source to ensure you have the correct schema. Is that understanding correct?

The purpose of splitting the schemas along concept boundaries was to introduce compiler-level guardrails for functionality that does not exist in certain places (such as plan modifiers only existing for resources) and therefore prevent developer confusion. This is to help newer provider developers, especially those unfamiliar with Go, to successfully create their providers without overwhelming documentation or trial-and-error burden that a global schema type brings. However, this is certainly at odds with some provider use cases which generally don't rely on concept-specific schema functionality.

As you discovered, the internal/fwschema interfaces are not intended for external consumption. They are there to define a standard interface between the various external schema implementations and the framework's internal logic. There may be a few different provider-side options in this situation though.

The newer schema model, the datasource/schema, provider/schema, and resource/schema packages now provide exported interfaces, such as schema.Attribute, which are implemented by the exported type-specific implementations, such as schema.StringAttribute. Over time, it is expected that each of these type-specific implementations will gain additional type-specific functionality. Since those interface and implementation types are exported, it should be possible to create provider-defined attribute types that satisfy both datasource/schema.Attribute and resource/schema.Attribute. This is really powerful (you can make your own custom tailored schema types to match a domain), but potentially quite a large amount of work for your intended goal here.

A simpler option may be to have the schema conversion for datasources accept a resource/schema.Attribute and use a type switch, for example:

// Quick sketch of resource to datasource attribute conversion
import (
  datasourceschema "github.com/hashicorp/terraform-plugin-framework/datasource/schema"
  resourceschema "github.com/hashicorp/terraform-plugin-framework/resource/schema"
)

func convertAttribute(resourceAttribute resourceschema.Attribute) datasourceschema.Attribute {
  switch attr := resourceAttribute.(type) {
  case resourceschema.BoolAttribute:
    return datasourceschema.BoolAttribute{
      Computed: attr.Computed,
      // ... other fields ...
    }
  // ... other types ...
  default:
    panic(fmt.Sprintf("unknown resource attribute type: %T", resourceAttribute))
  }
}

Another option that feels worth mentioning is that the schema generation could potentially happen outside the framework in this case rather than using a framework-defined type as the source. This removes the Go static typing problem. For example, creating a go generate based code generator that creates both the datasource and resource schema Go code from another source, such as a JSON file. For additional context, providing this sort of generator is a medium to long term goal for the framework maintainers, we just needed to finalize the first major version details of the framework itself beforehand.

As for adding something to the framework itself in this case, there are some sticky design considerations. For example, providers may require differing conversion behaviors based on the API (e.g. not all APIs make sense to convert all configurable attributes to Computed except one). Generally, the framework design has leaned to offer extensibility for customization, rather than pre-described solutions, and I personally feel that offering something specifically to solve this use case could be a risky endeavor on this side. Something in this area would likely be better suited as a separate Go module unless there is an overwhelmingly consistent ecosystem expectation.

The framework could also opt to keep the "global" tfsdk.Schema handling available, however having two options for schema definitions feels like it would be more confusing for provider developers.

The prior terraform-plugin-sdk did offer a reverse solution to this problem (helper/schema.DataSourceResourceShim), however it was never utilized much in practice because ultimately there are too many changes required for many providers. Solutions like these may be more confusing to understand over some potential code duplication.

---

I don't believe there is a good place to link you to for tracking details about provider code generation efforts yet, but if you are interested in meeting with the product team about this, maybe we can get something setup.

[Doridian]

Hi @Doridian 👋 Thank you for raising this concern. It appears that you're looking to simplify your provider development in cases where a data source lookup is fairly symmetrical to managing a resource in data terms. To support this, you are taking the "shared" schema and slightly modifying it for the data source to ensure you have the correct schema. Is that understanding correct?

That is exactly correct. The main reason I want to avoid duplication is to prevent drift.

Of course it could be solved the inverse way with a lint-type tool that ensures all fields are the same between the two schemas-in-code, but, that doesn't feel like something that should be developed for a specific provider, since looking at the current TF landscape, a lot of providers indeed offer both data sources and resources for a lot of their objects.

A simpler option may be to have the schema conversion for datasources accept a resource/schema.Attribute and use a type switch, for example:

I thought about doing that. I might do that in the interim, but I just feel like it will look really repetitive and lots of duplicated code (as for most cases of the switch it will set the exact same things), and was hoping there might be a cleaner solution.

Another option that feels worth mentioning is that the schema generation could potentially happen outside the framework in this case rather than using a framework-defined type as the source. This removes the Go static typing problem. For example, creating a go generate

That option would be (as expected) the preferred outcome. That way I can write my schemas once and have them work. This might pose some issues when it comes to things like the custom validators if not carefully taken into consideration when designing the schema of course, but I'm sure that such considerations are part of the design anyway.

[rschmied]

I'm running into the exact same issue... I have a rather complicated, nested schema which I use both for resources and data sources. Basically, my provider deals with network topologies which consist of nodes which consists of interfaces (among other things). Such a topology can be a resource as well as a data source. Up to 1.0.0, I had a relatively elegant schema, serving both purposes. 1.0.0 has thrown a big wrench into this... 😭

And, tbh, at this point I don't really see the benefit of strictly separating (and thus requiring to duplicate) all schema attributes into provider, resource and data source attributes. Providing too many guard rails here, imo. Feels a bit like the omni-present "Caution: Contents Hot" label, well... of course, you ordered the hot drink, so you wouldn't be surprised if it actually was hot.

[remyleone]

Would it be possible to have in addition to datasource/schema, provider/schema, and resource/schema a any/schema that could be used in all those contexts? By doing so we could have attributes that are the same everywhere. I won't mind importing this any/schema defined schema and adjusting it for either a data or resource use case. Because the underlying types are mostly the same maybe it could be a way to have some reuse between all those different use-cases. Maybe go generics could be of some help here?

[gaetanars]

To deal with centralized schema, we have start to write a little "superschema" plugin. We also add some tweaks to add Validator and Planmodifier MarkdownDescription because there are not catch by the doc plugin at this time.

If you are interested, you can check this plugin