magic-modules decoupling from filesystem ops: phase 2

Use fs.FS to load templates and various third_party files, instead of expecting those files to be available relative to the repository root.

This is useful in a Blaze/Bazel environment, where local filesystem operations may not make those files available unless they're included in the BUILD files.

Note that this PR cannot be merged until internal work (cr/837417036) is submitted first ( :heavy_check_mark: ).

See also https://github.com/GoogleCloudPlatform/magic-modules/pull/15881 for related work in this space.

Use explicit FS to load templates instead of filesystem read operations. This breaks direct `loader` usage: you will need to pass a `fs.FS` if you're using this API directly.

Nov 28 '25 13:11 nicdumz

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@zli82016, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

Nov 28 '25 16:11 github-actions[bot]

/gcbrun

Dec 01 '25 17:12 zli82016

@nicdumz , can you split this PR into two PRs:

Upgrade go to 1.25 by following the doc
Use embed.fs

Thanks.

Dec 01 '25 20:12 zli82016

OK, https://github.com/GoogleCloudPlatform/magic-modules/pull/15842

Dec 01 '25 21:12 nicdumz

If you need providers to be aware of mmv1 location you could adopt a similar pattern to my loader where I pass a BaseDirectory.

That way providers would be aware of where to read/write. I think the provider paradigm needs a bit of a rework in general tbh, but that design needs to be more considered.

Dec 01 '25 23:12 ScottSuarez

This will break me in certain contexts because I use mmv1 as a API and need to load and reload configs when serializing. We should set up time to talk if this needs to be done.

Good to know, where is this code so I can have a look at how to change it?

I am thinking a global fileio package or something similar that has a global embedded fs it looks through first instead of passing the embedded filesystem around a bunch.

The latest version now adds an OverlayFS which is pretty much like this.

Note that it most likely cannot be a global however, since it needs to be populated from mmv1's main.go (or... main.go would have to set a Singleton, which... would feel sad :-1: )

Perhaps consider how passing an OverlayFS through allows for easy extensiibility of the API. For instance, for my upcoming internal work I'll be able to add templates to an OverlayFS programmatically, via the mmv1 API, without having to bother and find ways to add the templates somewhere (in overrides repositories or otherwise).

Also, as is this would break override behavior since the templates for override files are not stored in mmv1.

Actually I believe this simply works. Tested locally, the generation of internal providers looks to be working. There are minor comment header diffs ("This code is generated by Magic Modules using the following:") but both the before and after versions anyhow produce incorrect links because the override files do not exist in mmv1 repo.

If you need providers to be aware of mmv1 location you could adopt a similar pattern to my loader where I pass a BaseDirectory.

I think that doesn't work.

Taking a step back, the requirements in a Blaze world will look like:

The mmv binary is pre-built, using BUILD files, and we have control on what's embedded or not.
A product.yaml and associated Resource.yaml files are being built as part of other build rules, and do not come from the mmv1 repo.
A build genrule/macro puts the yaml files into a tempdir, and then runs something akin to {mm} --overrides=$$TMPDIR --output={output} --provider={provider}.
When running as part of the macro, the mmv1 binary runs in a environment where os.GetCwd() is not meaningful and does not have a way to find mmv1 template files or other third party files since they're not declared/exposed to the build env.

And I think that for internal work I need the mmv1 API to allow injecting additional templates easily, without relying on file reads/writes.

The way I solve this in this PR is by:

embedding the templates via go/embed inside of mmv1
overlaying overrides directory on top of it
(Because of go/embed constraints mentioned above I need to move around go.mod which I do not love)

There may be a second alternative, which I would have yet to test:

Create a blaze/bazel filegroup containing all mmv1 files
Change the build rule to pass --base_directory=path/to/filegroup/dir to main.go (and add that flag). This is assuming that filegroups can be exposed as fully fledged directories when executing actions, something I'd have to verify
Overlay the overrides directory on top of it

Comparison vs current approach

:heavy_plus_sign: does not require moving around go.mod
:question: : needs more experimenting to validate filegroup behavior
:heavy_minus_sign: requires exposing third_party/ and templates/ filegroup contents to the build rule, which is somewhat leaky; whereas the previous approach hides this as an implementation detail inside of the mmv1 binary.

Dec 02 '25 14:12 nicdumz

Can you ping me internally. Lets chat a little.

Dec 02 '25 17:12 ScottSuarez

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

Dec 04 '25 08:12 modular-magician

After our discussion, I'm now using the second approach.

In particular this does not require the use of embeds, the migration to go1.25 anymore, and should be more flexible if in the future inputs have to be split around.

I've sent https://github.com/GoogleCloudPlatform/magic-modules/pull/15881 as a first PR to make disallow-large-prs happier.

@ScottSuarez PTAL

Dec 04 '25 12:12 nicdumz

@zli82016 This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

Dec 05 '25 09:12 github-actions[bot]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2853 files changed, 2853 insertions(+), 2853 deletions(-)) google-beta provider: Diff ( 3043 files changed, 3043 insertions(+), 3043 deletions(-)) terraform-google-conversion: Diff ( 1192 files changed, 1192 insertions(+), 1193 deletions(-))

Dec 05 '25 15:12 modular-magician

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2853 files changed, 2853 insertions(+), 2853 deletions(-)) google-beta provider: Diff ( 3043 files changed, 3043 insertions(+), 3043 deletions(-)) terraform-google-conversion: Diff ( 1192 files changed, 1192 insertions(+), 1193 deletions(-))

Dec 05 '25 15:12 modular-magician

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

terraform-google-conversion: Diff ( 1 file changed, 1 deletion(-))

Dec 05 '25 17:12 modular-magician

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

terraform-google-conversion: Diff ( 1 file changed, 1 deletion(-))

Dec 05 '25 17:12 modular-magician

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

terraform-google-conversion: Diff ( 1 file changed, 1 deletion(-))

Dec 05 '25 18:12 modular-magician

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

Dec 05 '25 19:12 modular-magician

Tests analytics

Total tests: 5226 Passed tests: 4768 Skipped tests: 448 Affected tests: 10

Click here to see the affected service packages

cloudtasks

cloudasset

accesscontextmanager

appengine

documentai

iambeta

servicedirectory

bigquery

apihub

bigqueryconnection

iam2

notebooks

osconfigv2

securitycenter

sql

binaryauthorization

datalossprevention

edgecontainer

firebaseapphosting

firebasedatabase

gkeonprem

modelarmor

osconfig

biglake

dataform

sourcerepo

storage

identityplatform

storageinsights

alloydb

backupdr

deploymentmanager

dialogflow

siteverification

activedirectory

firebasestorage

healthcare

networkconnectivityv1

parametermanagerregional

pubsublite

developerconnect

migrationcenter

bigquerydatatransfer

dialogflowcx

cloudids

apphub

kms

logging

spanner

beyondcorp

bigtable

secretmanagerregional

workbench

accessapproval

networkservices

clouddeploy

filestore

lustre

networksecurity

tags

composer

gemini

memcache

storagetransfer

clouddomains

firebaseextensions

gkehub

apigee

cloudidentity

datacatalog

firebasehosting

corebilling

bigquerydatapolicy

chronicle

dataprocgdc

observability

parametermanager

cloudscheduler

bigquerydatapolicyv2

discoveryengine

edgenetwork

looker

orgpolicy

privateca

resourcemanager

artifactregistry

compute

datapipeline

dataplex

datastream

firebaseappcheck

firebasedataconnect

modelarmorglobal

firestore

integrations

securitycenterv2

storagecontrol

vmwareengine

vpcaccess

containeranalysis

datafusion

mlengine

securityposture

securityscanner

serviceusage

storagebatchoperations

bigqueryreservation

cloudfunctions2

resourcemanager3

vertexai

gkehub2

redis

workflows

containerattached

documentaiwarehouse

iamworkforcepool

servicemanagement

billing

blockchainnodeengine

certificatemanager

cloudbuildv2

eventarc

iam3

monitoring

netapp

apigateway

oracledatabase

servicenetworking

transcoder

workstations

bigqueryanalyticshub

cloudquotas

dns

gkebackup

networkmanagement

cloudbuild

parallelstore

cloudrun

contactcenterinsights

dataproc

essentialcontacts

integrationconnectors

networkconnectivity

pubsub

securitycentermanagement

managedkafka

secretmanager

ces

cloudfunctions

cloudsecuritycompliance

databasemigrationservice

firebase

memorystore

oslogin

privilegedaccessmanager

colab

dataprocmetastore

iap

publicca

runtimeconfig

saasruntime

securesourcemanager

tpuv2

cloudrunv2

Action taken

Found 10 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccAccessContextManager

TestAccComputeFutureReservation_sharedFutureReservationExample

TestAccComputeRegionBackendService_withNetworkPassThroughLbTrafficPolicy

TestAccDataFusionInstance_dataFusionInstanceCmekExample

TestAccDialogflowCXTestCase_dialogflowcxTestCaseFullExample

TestAccDialogflowCXTestCase_update

TestAccDiscoveryEngineCmekConfig_discoveryengineCmekconfigDefaultExample

TestAccDiscoveryEngineCmekConfig_discoveryengineCmekconfigDefaultExample_update

TestAccGKEHubFeature_FleetDefaultMemberConfigPolicyController

TestAccHealthcarePipelineJob_healthcarePipelineJobBackfillExample

Get to know how VCR tests work

Dec 06 '25 01:12 modular-magician

Tests analytics

Total tests: 5224 Passed tests: 4766 Skipped tests: 448 Affected tests: 10

Click here to see the affected service packages

cloudbuildv2

dataplex

parametermanagerregional

workbench

apigee

dataprocgdc

deploymentmanager

gkehub

privilegedaccessmanager

bigtable

dataprocmetastore

kms

managedkafka

sql

apigateway

apphub

cloudids

edgecontainer

parallelstore

storageinsights

storagetransfer

cloudrun

datapipeline

iam2

pubsub

securitycentermanagement

spanner

cloudquotas

cloudtasks

compute

dataproc

documentaiwarehouse

healthcare

networkconnectivity

clouddeploy

containerattached

essentialcontacts

firebaseextensions

oslogin

secretmanager

securitycenter

storagecontrol

bigquery

contactcenterinsights

corebilling

publicca

servicemanagement

workstations

bigquerydatapolicyv2

binaryauthorization

cloudasset

cloudidentity

datastream

looker

memcache

containeranalysis

dns

iam3

memorystore

osconfigv2

vmwareengine

networkmanagement

appengine

cloudrunv2

developerconnect

dialogflow

networkconnectivityv1

colab

discoveryengine

orgpolicy

securesourcemanager

storagebatchoperations

certificatemanager

cloudbuild

firebasedataconnect

securityposture

serviceusage

clouddomains

composer

filestore

oracledatabase

tags

iamworkforcepool

runtimeconfig

activedirectory

bigquerydatapolicy

cloudfunctions2

documentai

notebooks

privateca

saasruntime

tpuv2

firebasedatabase

osconfig

bigqueryanalyticshub

ces

gkehub2

logging

modelarmorglobal

datacatalog

firebaseappcheck

gkebackup

servicedirectory

artifactregistry

bigqueryreservation

firebasestorage

netapp

resourcemanager

vpcaccess

apihub

firebase

lustre

beyondcorp

bigqueryconnection

resourcemanager3

siteverification

sourcerepo

accessapproval

dialogflowcx

firebaseapphosting

firestore

integrations

securityscanner

accesscontextmanager

datafusion

identityplatform

monitoring

transcoder

vertexai

biglake

billing

chronicle

cloudfunctions

dataform

networksecurity

observability

servicenetworking

edgenetwork

gemini

iambeta

iap

networkservices

bigquerydatatransfer

mlengine

modelarmor

redis

secretmanagerregional

workflows

alloydb

databasemigrationservice

datalossprevention

gkeonprem

cloudscheduler

integrationconnectors

blockchainnodeengine

cloudsecuritycompliance

firebasehosting

migrationcenter

parametermanager

storage

backupdr

eventarc

pubsublite

securitycenterv2

Action taken

Found 10 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccAccessContextManager

TestAccComputeFutureReservation_sharedFutureReservationExample

TestAccComputeRegionBackendService_withNetworkPassThroughLbTrafficPolicy

TestAccDataFusionInstance_dataFusionInstanceCmekExample

TestAccDialogflowCXTestCase_dialogflowcxTestCaseFullExample

TestAccDialogflowCXTestCase_update

TestAccDiscoveryEngineCmekConfig_discoveryengineCmekconfigDefaultExample

TestAccDiscoveryEngineCmekConfig_discoveryengineCmekconfigDefaultExample_update

TestAccGKEHubFeature_FleetDefaultMemberConfigPolicyController

TestAccHealthcarePipelineJob_healthcarePipelineJobBackfillExample

Get to know how VCR tests work

Dec 06 '25 01:12 modular-magician

🟢 Tests passed during RECORDING mode: TestAccAccessContextManager__access_level [Debug log] TestAccAccessContextManager__access_level_condition [Debug log] TestAccAccessContextManager__access_policy [Debug log] TestAccAccessContextManager__access_policy_scoped [Debug log] TestAccAccessContextManager__authorized_orgs_desc [Debug log] TestAccAccessContextManager__service_perimeter_dry_run_ingress_policy [Debug log] TestAccAccessContextManager__service_perimeters [Debug log] TestAccComputeRegionBackendService_withNetworkPassThroughLbTrafficPolicy [Debug log]

🔴 Tests failed when rerunning REPLAYING mode: TestAccComputeRegionBackendService_withNetworkPassThroughLbTrafficPolicy [Error message] [Debug log]

Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.

Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.

🔴 Tests failed during RECORDING mode: TestAccAccessContextManager__access_level_custom [Error message] [Debug log] TestAccAccessContextManager__access_level_full [Error message] [Debug log] TestAccAccessContextManager__access_levels [Error message] [Debug log] TestAccAccessContextManager__gcp_user_access_binding [Error message] [Debug log] TestAccAccessContextManager__service_perimeter [Error message] [Debug log] TestAccAccessContextManager__service_perimeter_dry_run_egress_policy [Error message] [Debug log] TestAccAccessContextManager__service_perimeter_update [Error message] [Debug log] TestAccComputeFutureReservation_sharedFutureReservationExample [Error message] [Debug log] TestAccDataFusionInstance_dataFusionInstanceCmekExample [Error message] [Debug log] TestAccDialogflowCXTestCase_dialogflowcxTestCaseFullExample [Error message] [Debug log] TestAccDialogflowCXTestCase_update [Error message] [Debug log] TestAccDiscoveryEngineCmekConfig_discoveryengineCmekconfigDefaultExample [Error message] [Debug log] TestAccDiscoveryEngineCmekConfig_discoveryengineCmekconfigDefaultExample_update [Error message] [Debug log] TestAccGKEHubFeature_FleetDefaultMemberConfigPolicyController [Error message] [Debug log] TestAccHealthcarePipelineJob_healthcarePipelineJobBackfillExample [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

Dec 06 '25 01:12 modular-magician

🟢 Tests passed during RECORDING mode: TestAccAccessContextManager__access_level [Debug log] TestAccAccessContextManager__access_level_condition [Debug log] TestAccAccessContextManager__access_level_custom [Debug log] TestAccAccessContextManager__access_level_full [Debug log] TestAccAccessContextManager__access_levels [Debug log] TestAccAccessContextManager__access_policy [Debug log] TestAccAccessContextManager__access_policy_scoped [Debug log] TestAccAccessContextManager__authorized_orgs_desc [Debug log] TestAccAccessContextManager__service_perimeter [Debug log] TestAccAccessContextManager__service_perimeter_dry_run_egress_policy [Debug log] TestAccAccessContextManager__service_perimeter_dry_run_ingress_policy [Debug log] TestAccAccessContextManager__service_perimeter_update [Debug log] TestAccAccessContextManager__service_perimeters [Debug log] TestAccComputeRegionBackendService_withNetworkPassThroughLbTrafficPolicy [Debug log]

🔴 Tests failed when rerunning REPLAYING mode: TestAccComputeRegionBackendService_withNetworkPassThroughLbTrafficPolicy [Error message] [Debug log]

Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.

Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.

🔴 Tests failed during RECORDING mode: TestAccAccessContextManager__gcp_user_access_binding [Error message] [Debug log] TestAccComputeFutureReservation_sharedFutureReservationExample [Error message] [Debug log] TestAccDataFusionInstance_dataFusionInstanceCmekExample [Error message] [Debug log] TestAccDialogflowCXTestCase_dialogflowcxTestCaseFullExample [Error message] [Debug log] TestAccDialogflowCXTestCase_update [Error message] [Debug log] TestAccDiscoveryEngineCmekConfig_discoveryengineCmekconfigDefaultExample [Error message] [Debug log] TestAccDiscoveryEngineCmekConfig_discoveryengineCmekconfigDefaultExample_update [Error message] [Debug log] TestAccGKEHubFeature_FleetDefaultMemberConfigPolicyController [Error message] [Debug log] TestAccHealthcarePipelineJob_healthcarePipelineJobBackfillExample [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

Dec 06 '25 01:12 modular-magician

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 1 file changed, 12 insertions(+), 12 deletions(-)) google-beta provider: Diff ( 1 file changed, 12 insertions(+), 12 deletions(-))

Dec 08 '25 13:12 modular-magician

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 1 file changed, 12 insertions(+), 12 deletions(-)) google-beta provider: Diff ( 1 file changed, 12 insertions(+), 12 deletions(-))

Dec 08 '25 13:12 modular-magician

@zli82016 This PR has been waiting for review for 3 weekdays. Please take a look! Use the label disable-review-reminders to disable these notifications.

Dec 11 '25 09:12 github-actions[bot]

@GoogleCloudPlatform/terraform-team @zli82016 This PR has been waiting for review for 1 week. Please take a look! Use the label disable-review-reminders to disable these notifications.

Dec 15 '25 09:12 github-actions[bot]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR hasn't generated any diffs, but I'll let you know if a future commit does.

Dec 18 '25 16:12 modular-magician