turbo icon indicating copy to clipboard operation
turbo copied to clipboard

Allow inputs and globalDependencies outside of workspace root

Open qdozaq opened this issue 3 years ago • 3 comments

Describe the feature you'd like to request

Currently integrating turbo into a repo that has its typescript monorepo in a subdirectory that is part of a much larger monolithic app. example:

# typescript monorepo
/frontend
#other backend logic written mostly in PHP
/api
/controllers
/views
/routes

turbo is only being implemented in /frontend but some package builds rely on files external to the workspace root such as the openapi.yaml spec which is consumed to generate typescript types and api typed hooks.

Currently if I were to try targeting an external file in inputs I get the following error:

failed to calculate global hash: the path you are attempting to specify (<path>) is outside of the root

I see in the code that this is something that is explicitly checked for https://github.com/vercel/turborepo/blob/0d51a1bdfc101aa48294777aef7432384f391966/cli/internal/globby/globby.go#L25-L37

So wondering what the limitation is there with including files external to the root.

Describe the solution you'd like

Essentially extending the current syntax to work with files outside the root workspace:

// turbo.json
{
  "globalDependencies": ["../my-global-file"],
  "pipeline": {
      "api-types#build": {
          "inputs": ["../../../open-api.yaml"],
          ...
     }
  }
}

Describe alternatives you've considered

At the moment I am able to get around this by performing a pre step before running turbo that copies the required files into the workspace so that they can be cached and referenced properly. This is however not nearly as ergonomic and requires devs to be aware of these build steps declared outside the turbo.json file.

qdozaq avatar Sep 12 '22 18:09 qdozaq

This is an interesting use case, where turbo is responsible for a subtree in a larger repository.

There are some workarounds possible (move turbo.json and your package.json / workspace definitions to the monorepo root), but that does pollute the rest of your repository with turbo-related items that are only relevant to /frontend.

Out of curiosity, would you also want to outputs outside of the root of where you're using turbo?

gsoltis avatar Sep 14 '22 23:09 gsoltis

@gsoltis

There are some workarounds possible (move turbo.json and your package.json / workspace definitions to the monorepo root), but that does pollute the rest of your repository with turbo-related items that are only relevant to /frontend.

Yeh unfortunately not an option atm just due to the way our over arching build system is working and our desire to keep this folder somewhat isolated from the rest of the app

Out of curiosity, would you also want to outputs outside of the root of where you're using turbo?

That would also be great! Currently having to run a task as cache:false because we copy output to another location for deployment

I guess the whole restriction seems somewhat arbitrary unless there's a particular performance reason it's not done.

qdozaq avatar Sep 15 '22 00:09 qdozaq

Some internal comments on this track the conversation here just about 1:1. 😅

Possible options:

  • Promote turbo.json to the root, add a package.json at the root and move the workspaces definition there.
  • Some API to set a root that is separate from the npm workspace root?
  • inputs can be from anywhere, but outputs have to be scoped to the turbo.json rooted directory?

Eventually this type of design is absolutely one where they're going to ask for outputs to be somewhere outside of the frontend monorepo because of some weird combination of things which will absolutely break our assumptions. So I'm tempted to encourage them to promote it to root.

We're all on the same page here about this being interesting and worth supporting, but it violates a few core assumptions about how we function in terms of cache creation and restoration (which are all predicated on the repository root).

In terms of a short-term fix, the workaround is the only option that we can propose. We do believe that there's something valuable to figuring out how to enable incremental adoption that isn't top-down and, as we explore the problem, we will try to "discover" what that design should be.

nathanhammond avatar Sep 15 '22 08:09 nathanhammond

Hey guys, any momentum on this? Another use case for this existing apps. We have a microfrontend-based app that started out with many isolated repos by design (11 in total). I'd like to keep that repo structure but would like to add some pipelining (orchestration) via turbo rather than building custom scripts. We don't want to merge all repos into one, or have developers create a root parent folder and move all their repos there. Would be nice to have another repo that would "act as root".

I get that the ideal is to have a parent folder (root) and have all your apps/ and your packages/ inside. But an existing app won't have this. What is the limitation with forbidding repo to reach outside of the root?

Thanks!

luijar avatar Feb 17 '23 19:02 luijar

Hey guys, any plans to move this forward? My use case:

I have a multilanguage project with C++, Kotlin, and JS monorepo parts. The proto files are located in the root of the project (parent root of the monorepo) and I can't move proto files to the monorepo, because the C++ and Kotlin code requires those files also.

OleksiiZubko avatar Oct 19 '23 21:10 OleksiiZubko