[Bug]: `ballerina/http` not resolving to the latest version during `bal build` in 2201.7.x

[gayaldassanayake]

Description

When we run bal build without the import ballerinax/github, and with a test directory, the following error occurs.

ERROR [ballerina/http/2.6.2::http_request_context.bal:(88:17,88:41)] invalid return type: members of a dependently-typed union type with an inferred typedesc parameter should have disjoint basic types

Steps to Reproduce

1. Create a new ballerina project with bal new -t main resolution-issue.
2. Run bal build once.
3. Add ballerinax/github as an import and add the below line to the main.bal main() method.

github:Client ghClient = check new ({auth: {token: "xxx"}});

4. bal run again.

Affected Version(s)

At least 2201.7.x

OS, DB, other environment details and versions

No response

Related area

-> Compilation

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[gayaldassanayake]

The latest version of ballerinax/github matching for U7 is 4.6.0 (although there is a newer 5.0.0, it is built with U8, hence incompatible with U7). ballerina/http is a transitive dependency in ballerinax/github with http 2.6.2 locked in it.

If we directly build the user package with the github import, we do not have a Dependencies.toml, therefore making the PackageResolutionMode=SOFT. This will allow to pick the latest compatible http version 2.9.7. As a result it will be successfully built.

However, the issue arises when we first build the project without the github import and then add the import and build again. Here, in the first build, a Dependencies.toml is generated. Due to the presence of a Dependencies.toml, during the next build, PackageResolutionMode=MEDIUM. This allows only the patch version bumps of dependencies. However, the latest 2.6.x patch of http is 2.6.2 and it will be the result of the resolution. This leads to the compile time error.

As a solution for this, we need to look into how we can make the resolution mode = soft in similar cases. A possible solution is to make the resolution mode soft, if the dependency is not recorded in the previous build.

[gayaldassanayake]

Had an offline discussion with @azinneera. With this fix, there are various scenarios that we need to reconsider, especially related to --sticky and predictable builds. So it was decided to document the current dependency resolution approach and discuss/ revise the overall design again. Hence the fix is on hold for now.

[gayaldassanayake]

The initial draft PR was for setting the package locking mode = SOFT only the sub-graph with the new direct import as the root.

However during the project API discussion with @sameerajayasoma @azinneera @ShammiL @Dilhasha @Thevakumar-Luheerathan on this issue, along with the above-mentioned approach, we discussed the need to evaluate the possibility of doing the entire resolution with the locking mode SOFT (rather than only the subgraph).

As per the drafted solution,

Current Impl

Dependencies added

New Package

SOFT mode

Added a dependency during the 24 hour period

For new direct dependencies, SOFT For existing dependencies, HARD New transitive dependencies, MEDIUM Pick the higher version for all conflicts.

Added a dependency after the 24 hour period

For existing dependencies, MEDIUM For new direct dependencies, SOFT New transitive dependencies, MEDIUM Pick the higher version for all conflicts.

New Impl

Dependencies added

New Package

SOFT mode

Added a dependency during the 24 hour period

For new direct dependencies, SOFT Pick the higher version for all conflicts. IO 1.1.1*, IO 1.2.1*, IO 1.3.1(central) For existing dependencies, HARD -> SOFT? New transitive dependencies, MEDIUM -> SOFT (Suggested)

Added a dependency after the 24 hour period

For existing dependencies, MEDIUM For new direct dependencies, SOFT New transitive dependencies, MEDIUM Pick the higher version for all conflicts.

However, a final decision was not taken, hence need to evaluate this approach further.