Introduce Ballerina workspaces support

[azinneera]

Description

Ballerina currently supports only a single package per project, which limits its ability to manage multiple interdependent packages in a monorepo-style structure. Many other ecosystems already support multi-package workspaces, enabling developers to manage dependencies directly from the local filesystem rather than relying solely on remote or local package repositories.

Problem

Without workspace support, developers must depend on a package repository (remote or local) for dependency management, even when related packages are available locally. This creates inefficiencies in the development workflow, particularly for teams following a monorepo approach. In such cases, developers are forced to manage each package independently, increasing complexity and reducing productivity.

Solution

Introduce native support for managing multiple Ballerina packages in a single workspace. This will allow developers to define a workspace containing multiple packages, enabling seamless dependency resolution between them. Core package commands such as new, build, pack, and test will be enhanced to operate in a workspace context, making it easier to organize, build, and test interdependent packages together.

[gimantha]

Can we improve [workspace] header to contain a name field? What i feel is, this is more like the profiles in maven and we should give a name to refer to each workspace. In a mono-repo style, devs might not want all the packages to be build (for example, monorepo for hr operations, we will have leave app, then par app, but they are not dependant. So can be two separate workspaces)

e.g.

[workspace]
name="workspace1
packages=["service-a, uti/lib-commons]

[workspace]
name="workspace2
packages=["service-b, uti/lib-commons]

[sanjiva]

Doesn't this overlap with the consolidator thing that was done for BI? That's similar but supports packages in different directories I think? Is that the difference?

[azinneera]

Doesn't this overlap with the consolidator thing that was done for BI? That's similar but supports packages in different directories I think? Is that the difference?

The consolidator tool requires packages to be published locally or remotely using bal push before they can be consolidated. Multi-package workspace support removes this prerequisite since it resolves packages in the same workspace OOTB.

Workspace support goes beyond consolidation. The feature originated from https://github.com/wso2-enterprise/integration-product-management/issues/338. With workspaces, BI can discover and interact with multiple Ballerina packages in a monorepo. This improves the DX by enabling direct inter-package resolution and tooling support without any publishing step.

[azinneera]

Can we improve [workspace] header to contain a name field? What i feel is, this is more like the profiles in maven and we should give a name to refer to each workspace. In a mono-repo style, devs might not want all the packages to be build (for example, monorepo for hr operations, we will have leave app, then par app, but they are not dependant. So can be two separate workspaces)

e.g.

[workspace] name="workspace1 packages=["service-a, uti/lib-commons]

[workspace] name="workspace2 packages=["service-b, uti/lib-commons]

The user can build the entire workspace or a specific package in the workspace (e.g., leave app). When a specific package is built, all its dependencies are built first. Therefore, for workspaces with multiple independent packages, the user can generate the artifact by specifying the app.

Nonetheless, this could be useful for monorepos with microservices. For example, a group of services needs to be built for the leave app, and another group of services for the par app. @sameerajayasoma WDYT?

[sanjiva]

Hey Asma ref overlap with consolidator- yes I understand the comment but we will have two ways to do the same thing in a monorepo situation now right?

We have to find a way to this in a way that is consistent with that feature IMO.

[azinneera]

Hey Asma ref overlap with consolidator- yes I understand the comment but we will have two ways to do the same thing in a monorepo situation now right?

We have to find a way to this in a way that is consistent with that feature IMO.

Workspaces and the consolidator tool are orthogonal. Workspaces are about organizing multiple packages in the same directory. Users still have to use the consolidator tool to generate the consolidated executable for these packages. So I don't think that there is an overlap. Rather, the consolidator tool only works for single-package projects at the moment. We should improve it to support multi-package projects as well.

[sameerajayasoma]

@sanjiva, as Asma said, workspaces and the consolidator tool are kinda orthogonal things.

Workspaces or multi-package projects is all about giving you a seamless editing experience across all your packages in the workspace, without having to publish your dependencies to the local repo.

The Consolidator tool is designed to link multiple packages into a single executable. IMO, workspaces will drastically improve the Cosolidator tool experience.

[sameerajayasoma]

Can we improve [workspace] header to contain a name field? What i feel is, this is more like the profiles in maven and we should give a name to refer to each workspace. In a mono-repo style, devs might not want all the packages to be build (for example, monorepo for hr operations, we will have leave app, then par app, but they are not dependant. So can be two separate workspaces) e.g. [workspace] name="workspace1 packages=["service-a, uti/lib-commons] [workspace] name="workspace2 packages=["service-b, uti/lib-commons]

The user can build the entire workspace or a specific package in the workspace (e.g., leave app). When a specific package is built, all its dependencies are built first. Therefore, for workspaces with multiple independent packages, the user can generate the artifact by specifying the app.

Nonetheless, this could be useful for monorepos with microservices. For example, a group of services needs to be built for the leave app, and another group of services for the par app. @sameerajayasoma WDYT?

I like the requirement. Shall we do this in the next phase?

[sanjiva]

@sanjiva, as Asma said, workspaces and the consolidator tool are kinda orthogonal things.

Workspaces or multi-package projects is all about giving you a seamless editing experience across all your packages in the workspace, without having to publish your dependencies to the local repo.

The Consolidator tool is designed to link multiple packages into a single executable. IMO, workspaces will drastically improve the Cosolidator tool experience.

@sameerajayasoma - so a you guys both say, the difference is that we don't need to publish to a local repo. However, remember that's a pain point of the consolidator, not a nice feature.

Let me ask it differently- when this feature is done, when and why would you use the consolidator for anything new?

The answer can be "its when you can't have a mono repo". Then should we not make it possible for a workspace to have a path outside the current dir as well? Do we allow symlinks to be a valid package dir?

[sameerajayasoma]

The current workspace design does not allow packages to be outside of the workspace dir.

my-workspace/
│
├── Ballerina.toml # Defines the workspace
├── service-a/
│     └── Ballerina.toml
│     └── modules/
├── service-b/
│     └── Ballerina.toml
│     └── modules/
└── util
      └── lib-common/
              └── Ballerina.toml
              └── modules/

Here is a sample Ballerina.toml file that defines a directory as a workspace.

[workspace]
packages = ["service-a", "service-b", "util/lib-common"] ## packages in the workspace

[build-options] ## common build options
observability-included = true
sticky = true

In this example, service-a and service-b depend on the common package, and unlike earlier, now you don't need to push the common package to the local repo. When you build service-a, Ballerina now automatically builds (if needed) the common package first.

---

If you want to create a single binary that consolidates service-a and service-b, you can use the following options:

1. Create a separate package in the workspace (say app) and add a bal file that imports service-a and service-b

import myorg/service-a as _;
import myorg/service-b as _;

2. Use the consolidate tool to generate this package.

$ bal consolidate-packages new --package-path=app myorg/service-a,myorg/service-b

This command creates a Ballerina package with the following Ballerina.toml. You can add this package to the workspace by adding the package name to the workspace toml file.

[package]
org = "app"
name = "myApp"
version = "0.1.0"

[[tool.consolidate-packages]]
id = "consolidateSvc"
options.services = ["myorg/service-a", "myorg/service-b"]

When you build this package, the consolidate tool generates a bal file that imports these two packages.

---

If the app package is not in a workspace and you want to consolidate packages in your local repo, then you have to specify local package dependencies as follows.

[package]
org = "app"
name = "myApp"
version = "0.1.0"

[[dependency]]
org = "myorg"
name = "service-a"
version = "1.1.0"
repository = "local"

[[dependency]]
org = "myorg"
name = "service-b"
version = "1.1.0"
repository = "local"

[[tool.consolidate-packages]]
id = "consolidateSvc"
options.services = ["myorg/service-a", "myorg/service-b"]

---

If the dependencies are available in Ballerina central, then the following should just work.

[package]
org = "app"
name = "myApp"
version = "0.1.0"

[[tool.consolidate-packages]]
id = "consolidateSvc"
options.services = ["myorg/service-a", "myorg/service-b"]

[sameerajayasoma]

If you don't want to write and maintain a bal file that imports packages for consolidation, then use the consolidator. Most probably for BI use cases.

@sanjiva, as Asma said, workspaces and the consolidator tool are kinda orthogonal things. Workspaces or multi-package projects is all about giving you a seamless editing experience across all your packages in the workspace, without having to publish your dependencies to the local repo. The Consolidator tool is designed to link multiple packages into a single executable. IMO, workspaces will drastically improve the Cosolidator tool experience.

@sameerajayasoma - so a you guys both say, the difference is that we don't need to publish to a local repo. However, remember that's a pain point of the consolidator, not a nice feature.

Let me ask it differently- when this feature is done, when and why would you use the consolidator for anything new?

The answer can be "its when you can't have a mono repo". Then should we not make it possible for a workspace to have a path outside the current dir as well? Do we allow symlinks to be a valid package dir?

Not only when you can't have a mono repo. If all your packages to be consolidated are available in Ballerina central or any other custom repository, then you will have to use the consolidator.

Allowing workspaces to have packages outside the workspace dir won't solve this problem, IMO.