RFC: Combine and organize metadata file locations across lifecycle

[sambhav]

Readable

It's my first time creating an rfc so not sure if I did everything right here.

[sambhav]

I updated the rfc with the sub-directory proposal instead of the prefix one as discussed in the last WG meeting.

[jromero]

I updated the rfc with the sub-directory proposal instead of the prefix one as discussed in the last WG meeting.

This makes it sound like the sub-directory structure would be the primary idea but I'm still seeing prefixes as the primary proposal. Am I missing something?

[sambhav]

I updated the rfc with the sub-directory proposal instead of the prefix one as discussed in the last WG meeting.

This makes it sound like the sub-directory structure would be the primary idea but I'm still seeing prefixes as the primary proposal. Am I missing something?

Urgh - I think I forgot to push. Should be fixed now.

[jromero]

Really like the directory restructure. The term buildpack-workspace is a bit of a mouthful so I may bikeshed on it a bit but it totally makes sense IMO.

[natalieparellano]

Really like the directory restructure. The term buildpack-workspace is a bit of a mouthful so I may bikeshed on it a bit but it totally makes sense IMO.

buildspace?

[ekcasey]

Capturing a discussion from WG, Stephen proposed:

1. Moving "top-level" files to a different "config" directory so that the layers dir provided to the buildpack is exclusively for layers and their metadata.
2. Providing the location of both the layers and config directories as environment variables, rather than positional arguments

Elaborating on this idea and digging the details we would need to work out:

current directory structure:

`/layers` (configurable with `CNB_LAYERS_DIR` or -layers) 
 ├── group.toml
 ├── plan.toml
 └── <escaped_buildpack_id> (buildpack arg `$1` )
    ├── build.toml
    ├── launch.toml
    ├── store.toml
    ├── <layer>.toml
    └── <layer>
  └── config
    └── metadata.toml

Assuming we want to reusing the existing layers volume for the new files instead of introducing a new required volume, I think these are the possible future directory structures? Option 1:

`/layers` (configurable with `CNB_LAYERS_DIR` or -layers) 
 ├── group.toml
 ├── plan.toml
 └── <escaped_buildpack_id> (`CNB_BP_LAYERS_DIR` )
    ├── <layer>.toml
    └── <layer>
  └── config
     └── metadata.toml
     └── <escaped_buildpack_id> (`CNB_BP_CONFIG_DIR` )
          ├── build.toml
          ├── launch.toml
          └── store.toml

pros:

• doesn't break layer reuse
• same-length for layer path names cons:
• calling the top-level layers even though it is layers + config

Option 2:

`/build` (configurable with `CNB_BUILD_DIR` or -build-dir) 
 ├── group.toml
 ├── plan.toml
 └── l ( or `layers?`)
      └── <escaped_buildpack_id> (`CNB_BP_LAYERS_DIR` )
          ├── <layer>.toml
          └── <layer>
 └── c ( or `config`?)
      ├── metadata.toml
      └── <escaped_buildpack_id> (`CNB_BP_CONFIG_DIR`  )
          ├── build.toml
          ├── launch.toml
          └── store.toml

pros:

• doesn't incorrectly call the entire top level directory layers (it's really too bad we used workspace for the app dir). cons:
• breaks layer reuse
• layers/layers? layers/l? (both are bad, one is shorter)
• longer path names
• right now we export layers/config, if we implement this before eliminating metadata.toml we would need to partially export /layers/config.

In the above examples names of specific directories or env vars could change. E.g:

1. Do we want to reuse CNB_LAYERS_DIR to mean something different in the buildpack context instead of a creating a new CNB_BP_LAYERS_DIR?
2. build is just a placeholder. workspace actually seems the most logical to me but we already used it for the app dir.

But I think the overall structures represent the two obvious options.

Other possibilities that I won't take as much time to elaborate on:

1. add a layers and config dir under each buildpack ID
2. unite layers and the app dir in a single mega workspace (e.g. workspace/app and workspace/<bp_id> or workspace/config, and workspace/layers) -- this one would be nice but it would also require a lot of changes...

[jkutner]

I'm +1 on @ekcasey's "option 1". I'm also fine with /layers/config. I don't think it's exceptionally weird.

[jabrown85]

I like @ekcasey's Option 2 - with a name change from /build to (/output, /result, /out, /cnb/out) or something along those lines.

I am also ok with Option 1.

[sambhav]

I also prefer option 2 with the same suggestions that @jabrown85 gave. option 1 may conflict with buildpack ids named config?

[jkutner]

fwiw, I'm fine with option 2 also

[jabrown85]

Here is my take after our recent discussions.

/cnb
├── buildpacks
│   └── <buildpacks>
├── lifecycle
│   └── <creator/builder/etc>
├── build-workspace (configurable with `CNB_WORKSPACE_DIR`)
│   ├── app (configurable with `CNB_APP_DIR ` or -app)
│   │    └── <app source code to transform>
│   ├── platform-config (configurable with `CNB_PLATFORM_CONFIG_DIR`)
│   │   └── order.toml (written by a platform, optionally - prior to `lifecycle` execution `platform/prepare`)
│   ├── lifecycle-config (configurable with `CNB_LIFECYCLE_CONFIG_DIR`)
│   │   └── group.toml
│   │   └── plan.toml
│   ├── layers (configurable with `CNB_LAYERS_DIR`)
│   │   └── new-buildpack (`CNB_BP_LAYERS_DIR` )
│   │       ├── <layer>.toml
│   │       └── <layer>
│   │   └── old-buildpack (buildpack arg `$1` )
│   │       ├── build.toml
│   │       ├── launch.toml
│   │       ├── store.toml
│   │       ├── <layer>.toml
│   │       └── <layer>
│   └── layers-config (configurable with `CNB_LAYERS_CONFIG_DIR`)
│       ├── metadata.toml
│       └── new-buildpack (`CNB_BP_CONFIG_DIR`  )
│           ├── build.toml
│           ├── launch.toml
│           └── store.toml

Platforms can mount a volume at /build-workspace or mount different volumes at each sub-directory if they have a need to do so.

[jabrown85]

/builder
├── buildpacks
│   └── <buildpacks>
├── lifecycle
│   └── <creator/builder/etc>
├── order.toml
└── stack.toml
/cnb (configurable with `CNB_WORKSPACE_DIR`)
├── app (configurable with `CNB_APP_DIR ` or -app)
│    └── <app source code to transform>
├── platform-config (configurable with `CNB_PLATFORM_CONFIG_DIR`)
│   └── order.toml (written by a platform, optionally - prior to `lifecycle` execution `platform/prepare`)
├── lifecycle-config (configurable with `CNB_LIFECYCLE_CONFIG_DIR`)
│   └── group.toml
│   └── plan.toml
├── layers (configurable with `CNB_LAYERS_DIR`)
│   └── new-buildpack (`CNB_BP_LAYERS_DIR` )
│       ├── <layer>.toml
│       └── <layer>
│   └── old-buildpack (buildpack arg `$1` )
│       ├── build.toml
│       ├── launch.toml
│       ├── store.toml
│       ├── <layer>.toml
│       └── <layer>
├── layers-config (configurable with `CNB_LAYERS_CONFIG_DIR`)
│    ├── metadata.toml
│    └── new-buildpack (`CNB_BP_CONFIG_DIR`  )
│        ├── build.toml
│        ├── launch.toml
│        └── store.toml

How does this look?

[jabrown85]

@samj1912 can you fetch upstream on your fork?

I am not sure how to properly take over this PR 😬 . I checked it out via gh pr checkout 145 and pushed...but I think it only lives in this repo somehow. I noticed the fork has a "fetch upstream" button but I don't have write permissions to do it.

[hone]

Putting on the backburner for now due to other active RFCs.

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[hone]

@natalieparellano suggested of pairing this with deprecation of other APIs.