DifferentialEquations.jl icon indicating copy to clipboard operation
DifferentialEquations.jl copied to clipboard

The great restructuring, episode 15: the monorepo strikes back, single SciML version

Open ChrisRackauckas opened this issue 8 months ago • 44 comments

As SciML has grown, so has its structural needs. Back in 2017, we realized that DifferentialEquations.jl was too large and so it needed to be split into component packages for OrdinaryDiffEq, StochasticDiffEq, etc. in order to allow for loading only a part of the system as DiffEq itself was too big. In 2021, we expanded SciML to have LinearSolve, NonlinearSolve, etc. as separate interfaces so they can be independently used, and as such independent packages. In 2024 we split many of the solver packages into independent packages like OrdinaryDiffEqTsit5, in order to accomodate users who only wanted lean dependencies to simple solvers. And now we have hundreds of packages across hundreds of repos, though many of the subpackages are in the same repo.

If we had started this system from scratch again, I don't think we'd have so many repos. With the subrepo infrastructure of modern Julia, keeping OrdinaryDiffEq and StochasticDiffEq together would have been nice. In fact, DelayDiffEq.jl touches many of the OrdinaryDiffEqCore internals, so it should be versioned together with OrdinaryDiffEq. This leads to issues like https://github.com/JuliaLang/julia/issues/55516 being asked of the compiler team.

The core issue here is that the module structure does not necessarily reflect the public/private boundaries, but rather the module system is designed for separate loading, separate compilation, conditional dependencies, and ultimately handling startup/latency issues. This means that it really does not make sense for OrdinaryDiffEqTsit5 and OrdinaryDiffEqCore to have different versions, since they really should be handled in lock-step, and any versioning that isn't matching is always suspect. This leads to odd issues with bumping, and nasty resolution of manifests. The issue is that the semver system is designed around package boundaries being built around public interfaces with announcing public breakage, but this simply does not work out when the common breakage is non-public internals simply because the package boundary is at some internal function, again not because it's a generally good idea for the packages but instead because it's a requirement to achieve lower startup times.

So what can we do? The proposal is not so simple, but it can work. Instead of waiting on a fix for https://github.com/JuliaLang/julia/issues/55516, we can do some clever tricks on the tools that we now have in Julia 2025 land in order to pull this off. It would look like this:

  • We can have a single basic repository that contains all of the core code. For example, OrdinaryDiffEq contains what is now the OrdinaryDiffEqCore code
  • We then make all of the add on code into extensions. For example, OrdinaryDiffEqTsit5 would effectively be an empty package, that only exists to have a Project.toml for declaring new dependencies, and an OrdinaryDiffEqOrdinaryDiffEqTsit5 extension to OrdinaryDiffEq would be created to trigger on using OrdinaryDiffEqTsit5 that actually loads the extension code.
  • Importantly, this means that the code lives in OrdinaryDiffEq.jl, and thus the solver code only has one version, the OrdinaryDiffEq version. All solver code is then single versioned and updated in lock step. The only reason to bump the version on the pseudo packages is to bump dependency versions. Now because the code does not actually live in OrdinaryDiffEqTsit5, the only way to do a code update is to first make a version of OrdinaryDiffEqTsit5 that is a major bump with the new dependency versions, and then update OrdinaryDiffEq.jl to allow the new OrdinaryDiffEqTsit5 major. This means that every release of the psuedo packages is a considered a breaking release, but the only times they need a release are for compat bumps.

With this, the user code would look like:

using OrdinaryDiffEq, OrdinaryDiffEqTsit5
function lorenz!(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
u0 = [1.0; 0.0; 0.0]
tspan = (0.0, 100.0)
prob = ODEProblem(lorenz!, u0, tspan)
sol = solve(prob, Tsit5())
using Plots;
plot(sol, idxs = (1, 2, 3))

Note that using OrdinaryDiffEqTsit5 would be a requirement, as OrdinaryDiffEq would no longer trigger any solvers. If this isn't nice, we could have the single version package be OrdinaryDiffEqCore, with a higher level OrdinaryDiffEq that just uses a few solvers.

Small Questions

  1. The issue "Note that using OrdinaryDiffEqTsit5 would be a requirement, as OrdinaryDiffEq would no longer trigger any solvers. If this isn't nice, we could have the single version package be OrdinaryDiffEqCore, with a higher level OrdinaryDiffEq that just uses a few solvers. " - Is that to janky?
  2. Any potential issues not considered?

Big Question

If we go to monorepo, what's in and what's out? The advantage of having more things in a single repo is clear. Most of the issues around downstream testing, keeping package versions together, etc. are all gone. You know your code is good to go if it passes tests in this repo because that would have "everything".

However, what is everything? Should we have the following:

  • DifferentialEquations
  • NonlinearSolve
  • LinearSolve
  • ...

Or if we're going to do this, should we just have a single SciML?

using SciML, OrdinaryDiffEqTsit5
function lorenz!(du, u, p, t)
    du[1] = 10.0(u[2] - u[1])
    du[2] = u[1] * (28.0 - u[3]) - u[2]
    du[3] = u[1] * u[2] - (8 / 3) * u[3]
end
u0 = [1.0; 0.0; 0.0]
tspan = (0.0, 100.0)
prob = ODEProblem(lorenz!, u0, tspan)
sol = solve(prob, Tsit5())
using Plots;
plot(sol, idxs = (1, 2, 3))

Then you can just ask, "what version is your SciML?"

I think that would actually be really nice because splitting the interface between SciMLOperators, SciMLBase, DiffEqBase, NonlinearSolveBase, OptimizationBase, OrdinaryDiffEqCore, etc. are also somewhat arbitrary distinctions and moving code at the interface level there has always been a multi-repo mess due to the aritificiality of the split and release process w.r.t. semver at this level. Additionally, SciMLSensitivity is a package adding sensitivity analysis to all solvers, even NonlinearSolve.jl, so it would neatly fit as an extension to all of the packages. If that's the case, is the core package here just SciMLBase, and everything else is an extension to SciMLBase?

That said, how far do we go? Is ModelingToolkit in there? Is Catalyst in there? Symbolics.jl and SymbolicUtils.jl? Do we then restructure the whole SciML documentation as a single Vitepress doc? Or build many different Documenter docs from one repo?

Some CI questions

  1. Testing the subpackages: do we put the tests all in the main repo, or associate them with the psuedo packages? That way users could trigger subsets of tests?
  2. Could the tests know/understand the dependency graph, so say a change to SciMLBase itself triggers all tests, while changing something in OrdinaryDiffEqCore only triggers the OrdinaryDiffEq tests and other downstream of that? That would be pretty essential for managing the growth of the CI budget, since more tests is more costs.
  3. Are there downsides with precompilation, download sizes, and system image building to consider?
  4. How will the VS Code Language Server feel about this?

Conclusion

This is going to be a lot of work, so I'm looking for comments before commencing.

@devmotion @oscardssmith @isaacsas @TorkelE @thazhemadam @asinghvi17

ChrisRackauckas avatar Apr 02 '25 18:04 ChrisRackauckas

A drawback to extensions is that you cannot really access symbols defined in them. So they have to work pretty much purely by method extensions. Does all the code that is desired to be put into extensions do that here?

In the code given as:

With this, the user code would look like: sol = solve(prob, Tsit5())

does the Tsit5 symbol come from OrdinaryDiffEqTsit5, which is then dispatched on in the extension? What about solvers that need more complicated constructors, you then start to need to put more code into the packages that are supposed to be empty, which works against the idea of having all code in the base package.

Note that using OrdinaryDiffEqTsit5 would be a requirement, as OrdinaryDiffEq would no longer trigger any solvers.

I think the requirement is so that you get access to the Tsit5 symbol, no?

KristofferC avatar Apr 02 '25 19:04 KristofferC

What about packages that extend SciML packages but are not part of SciML? We cannot depend on package extensions, can we?

Moreover, I would have expected Tsit5() to be defined in OrdinaryDiffeqTsit5.jl. What if we change the algorithm interface? This will still require synchronizing versions across packages. Edit: Just saw that a related comment was posted above.

ranocha avatar Apr 02 '25 19:04 ranocha

Keeping a look at this, but given the magnitude of things I will have to think through things before I can contribute any useful opinion.

From experience, working with extensions is a proper pain. E.g. If I have function f in CatalystBifurcationKitExt, I cannot do CatalystBifurcationKitExt.f or Catalyst.f or something to reach it. I do agree that coupling versions would make things a lot easier though, and maybe given the niche use case of extensions it can work out.

I guess we have no idea when a solution to https://github.com/JuliaLang/julia/issues/55516 could drop?

TorkelE avatar Apr 02 '25 19:04 TorkelE

regarding "Small question" #1: Say I was still in the troubleshooting phase of a project and getting my infrastructure in place and I'm trying out different solvers to see which is most appropriate to use. Would the workflow in this suggestion be to 1) test with OrdinaryDiffEqCore because it has a bunch of the solvers packaged in, then 2) when I figure out which solver I need I would switch to OrdinaryDiffEq and import only that solver?

hodgensc avatar Apr 02 '25 19:04 hodgensc

does the Tsit5 symbol come from OrdinaryDiffEqTsit5, which is then dispatched on in the extension? What about solvers that need more complicated constructors, you then start to need to put more code into the packages that are supposed to be empty, which works against the idea of having all code in the base package.

Well there's two different choices that could be made here.

  1. It's still defined in the core package, in which case we'd need to manually guard against solve(prob, Tsit5()) not having loaded the extension. The downside here is that these manual load checks will be nasty to get right and have to be manually maintained (it's somewhat what's already in place in LinearSolve.jl)
  2. It's defined in the extension, in which case yes it's not just an empty piece of code. The nice thing here is that would handle some dependency issues, since yes some algorithms may have a default argument that depends on a dependency (like threads = Threads.PolyesterThreads() or something of the sort), and so it needs to be in the subpackage in order to not take the dependency.

I would be inclined to go with 2 for a mostly different reason than those, which is that we know large files and just loading large amounts of code is one of the biggest latency issues right now, and that's not necessarily going to get any smaller. So having a giant file of algorithm struct constructors would be something that would add up in terms of load time. IIRC the OrdinaryDiffEq algorithms page was like 0.15 seconds itself, so all solvers together could be a floor of like 0.4 seconds to loading (in its current implementation) just from the algorithm structs. So, putting those into the package shims could be the right thing to do not just for handling dependencies but also to make the loading floor lower.

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

I think I'm generally in favour of this. One thing I do want to mention though is that on this issue:

Small Questions

  1. The issue "Note that using OrdinaryDiffEqTsit5 would be a requirement, as OrdinaryDiffEq would no longer trigger any solvers. If this isn't nice, we could have the single version package be OrdinaryDiffEqCore, with a higher level OrdinaryDiffEq that just uses a few solvers. " - Is that to janky?

Why not keep the same behaviour we currently have? using OrdinaryDiffEqCore loads no solvers, and using OrdinaryDiffEq loads the (ordinary) kitchen sink.


Beyond that, I want to flag the fact that there's still currently a lot of UX issues with extensions, e.g. various things are broken with them in 1.10 (crytic errors about being unable to merge manifests when running test suites for example), and most users don't know how to actually access an extension module (i.e. Base.get_extension is kinda wonky to use).

I don't think any of that are reasons to not do it, but they are maybe reasons to be cautious about this.

I would suggest perhaps starting with a smaller set of consolidations. For instance, we could turn OrdinaryDiffEq back into a single-version monorepo using the techniques described above, and see how that goes. Or we could try this as a project to separate out the solvers in e.g. StochasticDiffEq.

Doing this on a smaller scale might help us find sharp edges and work out solutions to them before we go all in and consoldiate everything into one mega-repo.

MasonProtter avatar Apr 02 '25 19:04 MasonProtter

I guess we have no idea when a solution to https://github.com/JuliaLang/julia/issues/55516 could drop?

I was told today that it's unlikely to happen in v1.13, and so probably not within the next ~2 years. I don't think we can keep this version hell going on for that long without having a riot, so given that's not happening with new Base Julia tooling, we need to figure out a solution with duck tape and bubble gum.

That said, one non extension way to do this that was proposed is a purely CI solution. What @oscardssmith had sent me last night is that could do this through a CI-based lock step versioning system. Quoting:

  1. New compat bot that when triggered makes a commit that bumps the versions of all packages in a repo and and bumps all compats so that the compats are in lockstep.
  2. JuliaRegister bot support for bulk registration. (likey @JuliaRegistrator register_all() or something similar)
  3. Tagbot and Dependabot support for multiple projects in a repo

This approach has a few advantages over introducing subpackages:

  1. We can use this ~now rather than in ~2 years when we are comfortable lower bounding on 1.13
  2. This isn't a Pkg/language change so if it doesn't work out, we haven't committed to a new feature

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

Error while trying to register: Action not recognized: register_all

JuliaRegistrator avatar Apr 02 '25 19:04 JuliaRegistrator

Go away registrator, not now. Read the room.

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

Error while trying to register: Action not recognized: register_all

JuliaRegistrator avatar Apr 02 '25 19:04 JuliaRegistrator

I would suggest perhaps starting with a smaller set of consolidations. For instance, we could turn OrdinaryDiffEq back into a single-version monorepo using the techniques described above, and see how that goes. Or we could try this as a project to separate out the solvers in e.g. StochasticDiffEq.

Doing this on a smaller scale might help us find sharp edges and work out solutions to them before we go all in and consoldiate everything into one mega-repo.

That's definitely the plan. I think we'd do just the DiffEq stuff first and see how that goes. But I'm curious then for the next step if people would prefer NonlinearSolve separate or together with it. At least I know I want all of DiffEq together, since DelayDiffEq makes no sense in a different repo and that has been an issue for at least 7 years at this point.

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

https://github.com/SciML/DifferentialEquations.jl/pull/1: Say I was still in the troubleshooting phase of a project and getting my infrastructure in place and I'm trying out different solvers to see which is most appropriate to use. Would the workflow in this suggestion be to 1) test with OrdinaryDiffEqCore because it has a bunch of the solvers packaged in, then 2) when I figure out which solver I need I would switch to OrdinaryDiffEq and import only that solver?

Somewhat other way around. using OrdinaryDiffEq as a top level would give you all of the solvers. You REPL play with that. And then when you realize you just want Tsit5, you do using OrdinaryDiffEqCore, OrdinaryDiffEqTsit5 in the package and now it just has the one dependency.

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

#1: Say I was still in the troubleshooting phase of a project and getting my infrastructure in place and I'm trying out different solvers to see which is most appropriate to use. Would the workflow in this suggestion be to 1) test with OrdinaryDiffEqCore because it has a bunch of the solvers packaged in, then 2) when I figure out which solver I need I would switch to OrdinaryDiffEq and import only that solver?

Somewhat other way around. using OrdinaryDiffEq as a top level would give you all of the solvers. You REPL play with that. And then when you realize you just want Tsit5, you do using OrdinaryDiffEqCore, OrdinaryDiffEqTsit5 in the package and now it just has the one dependency.

Ah! Okay, that makes sense and seems manageable for someone like me. Thanks.

hodgensc avatar Apr 02 '25 19:04 hodgensc

What about packages that extend SciML packages but are not part of SciML? We cannot depend on package extensions, can we?

Yes, those couldn't be extensions. They would just have to be separate packages as always, with the same warts as before. But normally external packages should just be relying on public interfaces. Really the big issues are for example the solver packages that rely on the exact implementation of ODEIntegrator and such.

Though https://github.com/NumericalMathematics/PositiveIntegrators.jl is this kind of edge case that's using OrdinaryDiffEqCore internals... I'd prefer to just get that into the system at that point as its hard to ever make it fully robust if separate. We could at least keep the same downstream test infra for this kind of thing.

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

One thing I want to draw attention to with a monorepo approach is testing. I have similar problems with DI, where you could argue that I run 14 downstream tests every time I bat an eyelid. Now imagine if every docstring modification triggered a full run of every single SciML test ever written. This would be nightmarish for user experience and energy consumption alike. If we go down that road, we'll need a clever way to be selective on the test suites that run. It could probably be done through a mixture of PR labels and automatic GitHub Actions magic, for instance to detect which files have been modified and decide which subpackages should be tested as a result. But even for DI alone it's far from easy to set up.

As a side note, if we're discussing custody of the kids, I'd like to bring ADTypes into the DI repo, for exactly the same lockstep versioning reasons. These days, it is DI which defines the semantics of ADTypes, so it makes no sense that we're able to add something to ADTypes without having it immediately implemented in DI.

gdalle avatar Apr 02 '25 19:04 gdalle

One other thought: I think only reasonably mature and stable packages should be included in this monorepo approach. For example, in Neuroblox.jl we have a hard upper bound on ModelingToolkit.jl that we only increment when we've vetted that new versions don't break our stuff. This is partially our fault for using some MTK internals, but also partially the 'fault' of MTK, where it can be very hard for people working on that library to fully know and understand all the possible negative downstream affects of their changes.

I'd be a little nervous that an approach like this would mean that I'd have to hold back and not update stuff from the entire SciML ecosystem just because I want to protect myself from MTK breakage. On the other hand, maybe that's a good thing, and I can definitely imagine that I'll run into problems caused by having an old MTK and a new version of other stuff.

MasonProtter avatar Apr 02 '25 19:04 MasonProtter

Beyond that, I want to flag the fact that there's still currently a lot of UX issues with extensions, e.g. various things are broken with them in 1.10 (crytic errors about being unable to merge manifests when running test suites for example), and most users don't know how to actually access an extension module (i.e. Base.get_extension is kinda wonky to use).

I don't disagree, which is why this is a bit scary.

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

It's still defined in the core package, in which case we'd need to manually guard against solve(prob, Tsit5()) not having loaded the extension.

You kind of run into the problem described here, then https://youtu.be/TiIZlQhFzyk?t=1106. Basically, solve(prob, Tsit5()) will work even if a transitive dependency has loaded using OrdinaryDiffEqTsit5. For that to not be the case you kind of need to "prove" that you have a direct dependency on OrdinaryDiffEqTsit5 somehow, either by providing the module or some other "token" provided by that package that can be verified in the extension.

KristofferC avatar Apr 02 '25 19:04 KristofferC

It's still defined in the core package, in which case we'd need to manually guard against solve(prob, Tsit5()) not having loaded the extension.

You kind of run into the problem described here, then https://youtu.be/TiIZlQhFzyk?t=1106. Basically, solve(prob, Tsit5()) will work even if a transitive dependency has loaded using OrdinaryDiffEqTsit5. For that to not be the case you kind of need to "prove" that you have a direct dependency on OrdinaryDiffEqTsit5 somehow, either by providing the module or some other "token" provided by that package.

I don't think that'd be a bad thing. The only reason we don't want solve(prob, Tsit5()) to work without loading the extension is that we don't want to pay the cost of loading the solver code.

If someone already paid the cost, there's not really any reason to make it error.

MasonProtter avatar Apr 02 '25 19:04 MasonProtter

  1. It's defined in the extension, in which case yes it's not just an empty piece of code. The nice thing here is that would handle some dependency issues, since yes some algorithms may have a default argument that depends on a dependency (like threads = Threads.PolyesterThreads() or something of the sort), and so it needs to be in the subpackage in order to not take the dependency.

So is this the structure you are thinking of approximately?

# module OrdinaryDiffEqCore.jl
abstract type Solver end
function solve(prob, alg::Solver)
    throw(BackendNotLoadedError(alg))
end
# module OrdinaryDiffEqTsit5.jl
struct Tsit5 <: Solver
...
end
# module OrdinaryDiffEqCore/OrdinaryDiffEqTsit5Ext.jl
function OrdinaryDiffEqCore.solve(prob, alg::Tsit5)
...
end

If so I don't really see the value of adopting this structure when we could have a bot / Registrator based solution that will bump all versions of packages (regardless of whether they have changed or not) in lockstep with the OrdinaryDiffEq / SciML version.

One issue here, no matter which path we go down, is that a release of SciML + all constituent packages will be required even for a minor bugfix, which could create a lot of noise. We could potentially maintain per-package or per-folder changelogs, which would decrease the noise you have to wade through in a single file, and have a summary in SciML.jl's main changelog. But in general there would be a lot of "irrelevant" releases. You could version each package separately but that doesn't help the release situation for SciML.jl in general.

asinghvi17 avatar Apr 02 '25 19:04 asinghvi17

As a side note, if we're discussing custody of the kids, I'd like to bring ADTypes into the DI repo, for exactly the same lockstep versioning reasons. These days, it is DI which defines the semantics of ADTypes, so it makes no sense that we're able to add something to ADTypes without having it immediately implemented in DI.

Sure, but SciML gets to keep the dog.

But yes since MTK, OrdinaryDiffEq, NonlinearSolve, etc. has now moved to DI, ADTypes is no longer as much of a common interface of DI and SciML, since it's now really just SciML. DI should probably pull it in and downstream test SciMLSensitivity on its changes. We should follow up on that separately. Integrals.jl is I think the last straggler.

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

Unfortunately I don't know how to write my thoughts down without some duplication of what's already been written here and at https://github.com/JuliaLang/julia/issues/55516 .

The fundamental benefit of monorepos is that they linearize changes. This lets the developers not have to think about compatibility when developing. For monorepos with multiple packages (which we want for many reasons), this is also their fundamental problem: from the perspective of users and julia tooling these packages can skew to versions which don't work with each other.

Keeping everything in lockstep -- (e.g. forcing all versions to be the same and setting the compat bounds to that version) is a solution, and the one proposed here, right?

It does have downsides though: bumping anything bumps everything, with all the follow on from that: duplicated downloads, precompilation, etc. It also linearizes the entire ecosystem around SciML. Consider the cases where libraries build on disparate sciml packages, and a user wants to use two of these libraries that are stuck at different versions of SciML that nonetheless would have worked together. Letting versions float from each other is actually useful!

One approach I have not seen mentioned that would make this easier is to formalize even internal interfaces, i.e. versioning them with nearly empty packages that act like tags. (Like, but not completely the same as the common use of ...Core packages.) But this is of course much more error-prone than the lockstep approach.

wnoise avatar Apr 02 '25 19:04 wnoise

One issue here, no matter which path we go down, is that a release of SciML + all constituent packages will be required even for a minor bugfix, which could create a lot of noise. We could potentially maintain per-package or per-folder changelogs, which would decrease the noise you have to wade through in a single file, and have a summary in SciML.jl's main changelog. But in general there would be a lot of "irrelevant" releases. You could version each package separately but that doesn't help the release situation for SciML.jl in general.

Yes, the noise would be... substantial. Stefan said he was calculating statistics of the General registry and noticed that I personally averaged around 2-4 package releases per day. SciML has 200 packages now. So if we did a single SciML.jl with lockstep CI releasing per this suggestion, I personally would open 400-800 general registry PRs every single day. 🤷

But it does seem like it's coming ahead as the leading solution.

ChrisRackauckas avatar Apr 02 '25 19:04 ChrisRackauckas

If someone already paid the cost, there's not really any reason to make it error.

Yes there is because you don't control the set of transitive dependencies that end up getting loaded. You don't want your code to break because some random package in your dependency graph restructured and maybe started using a different diffew solver. By structuring it like that you make users vulnerable to code breakage without an associated breaking change.

KristofferC avatar Apr 02 '25 19:04 KristofferC

Do we then restructure the whole SciML documentation as a single Vitepress doc? Or build many different Documenter docs from one repo?

I don't think that's helpful for beginners. Someone that starts with Julia and wants to learn how to solve an ODE numerically has no reason to first go through the SciML castle until they trickle down to the OrdinaryDiffEq pillar. Separate docs are more accessible in my opinion. The current system does a good job of supporting this by having separate docs that are connected by the MultiDocumenter.jl top-page-header.


I have to say that I read this conversation but it is not transparent to me what is the currently favored approach. Is it the approach proposed in the very first comment, with empty extension packages?


  1. The issue "Note that using OrdinaryDiffEqTsit5 would be a requirement, as OrdinaryDiffEq would no longer trigger any solvers. If this isn't nice, we could have the single version package be OrdinaryDiffEqCore, with a higher level OrdinaryDiffEq that just uses a few solvers. " - Is that to janky?

This doesn't sound like an issue at all to me actually. It just warrants a major version bump. As a user I have already removed entirely all using OrdinaryDiffEq from my code: there is never a project where I would need all solvers. I use directly the Default or Verner packages.

Datseris avatar Apr 02 '25 21:04 Datseris

If we go to monorepo, what's in and what's out? The advantage of having more things in a single repo is > clear. Most of the issues around downstream testing, keeping package versions together, etc. are all > > > gone. You know your code is good to go if it passes tests in this repo because that would have "everything".

However, what is everything? Should we have the following:

DifferentialEquations NonlinearSolve LinearSolve ... Or if we're going to do this, should we just have a single SciML?

If everything would be in one big SciML, wouldn't it be harder for people "from the sidelines" to contribute ? A docstring update in LinearSolve triggering a new version of "everything" might scare people away. In some sense this would be opposite of the trend in Julia core - moving out SparseArrays etc into separate repos in order to be able to develop these independent from the core. I guess we need to have a good balance here, and the boundaries could be just as stated - defined by the different subfields of numerical methods as ODE solvers, linear solvers, nonlinear solvers. And it would be sufficient to be competent in just one of those in order to be not scared to contribute.

j-fu avatar Apr 02 '25 21:04 j-fu

The fundamental benefit of monorepos is that they linearize changes. This lets the developers not have to think about compatibility when developing. For monorepos with multiple packages (which we want for many reasons), this is also their fundamental problem: from the perspective of users and julia tooling these packages can skew to versions which don't work with each other.

Keeping everything in lockstep -- (e.g. forcing all versions to be the same and setting the compat bounds to that version) is a solution, and the one proposed here, right?

It does have downsides though: bumping anything bumps everything, with all the follow on from that: duplicated downloads, precompilation, etc. It also linearizes the entire ecosystem around SciML. Consider the cases where libraries build on disparate sciml packages, and a user wants to use two of these libraries that are stuck at different versions of SciML that nonetheless would have worked together. Letting versions float from each other is actually useful!

As mentioned, there would be clear advantages and disadvantages to this. Julia is extremely composable as a language, which itself is a double-edged sword, and I feel SciML is a living example of it. As a package developer strongly relying on the SciML ecosystem, it is amazing to be able to access each individual package and to have infinite granularity, but at the same time this highly complicates managing compatibilities, and you can often end up with weird package combinations which make it harder to debug issues. From my point of view, if this new monolithic package approach could still offer some decent level of granularity while not imposing precompilation penalization to downstream users (i.e. having to precompile lots of stuff that you won't need), then this could strike a nice balance. I like the idea of "what SciML version are you using?", although that perimeter should be properly discussed, since there will be different expectations for different people.

JordiBolibar avatar Apr 03 '25 08:04 JordiBolibar

Most of this conversation is above my pay grade, but I wanted to say that however you decide to structure it the user experience should be consistent across the entire ecosystem. I am mainly thinking of structuring things in some sort of hierarchy like the OrdinaryDiffEqCore and OrdinaryDiffEq discussion above. I think it is important to have some high-level package people can just load and run with access to a "reasonable" set of solvers. This is currently not the case for something like Optimization.jl. Doing R&D and REPL work with it is annoying and a PITA when you want to experiment with solvers. Obviously a tertiary concern.

agerlach avatar Apr 03 '25 14:04 agerlach

Right now I am partial to something like Oscar proposed, where we keep everything in CI and checks for OrdinaryDiffEq only. That seems the less risky while we figure stuff out (I am still a bit scared about the extensions, in my experience those are always incredibly messy to develop).

TorkelE avatar Apr 04 '25 09:04 TorkelE

The SciML ecosystem has often been at the forefront of pushing Julia to its limits (and beyond), and being wildly successful while doing it. Considering the myriads of repos and the associated maintenance hell, I absolutely do see the need for change.

However, the approach as proposed in the OP seems to (ab)use the extension system for something that it was not designed to do. IMHO this is a classic recipe for conservation of pain: It would resolve one set of issues, but only while creating a whole set of new issues.

Given that the SciML ecosystem is too vast and important for such kinds of shenanigans, I kindly suggest to use one of the other - more band-aid-like - solutions that have been proposed (e.g., enhancing the CI infrastructure). This should buy some time to develop a proper solution for the underlying issues, which can then be pushed upstream to Julia, and from which other projects might benefit as well.

Note: I am not an active SciML developer, only a heavy user. I thus might have easily missed some boundary conditions/constraints. Also, I am aware that proper solutions require (much) more time, possibly exceeding the realm of possibilities in an volunteer-driven project such as SciML.

sloede avatar Apr 04 '25 11:04 sloede