Oscar.jl
Oscar.jl copied to clipboard
oscar-system
Make the user aware of that they are using code from `experimental`
I decided to merge #1999 and relocate the discussion on the separation of experimental from the rest of the source here.
- We all seemed to agree that the user should in some way be made aware of that they are using code from
experimental. Just for the sake of completeness: Please disagree here if you disagree. - Should code from experimental be available by default? As already stated, if not its discoverability goes down, but that is not necessarily a bad thing. In my pov code should not stay "experimental" for too long.
These were the ideas of @fingolfin :
Indeed. Some options come to mind:
* maybe accessing code in experimental should be "guarded" and require `using Oscar.Experimental`, or require writing `Experimantal.FOO` (so `Experimental.galois_group(...)` * pro: obviously experimental * con: less discoverable * or maybe the primary functions should have a `@warn_experimental()` call in them, which prints a warning about them being experimental (possibly only the first time they are called) * ... other ideas?
So I'll cc some people who might be interested: @micjoswig @fieker @benlorenz . Maybe we can discuss a bit here and then decide during one of the next Friday meetings, preferably when we are all there, unless we immediately find a consensus here.
On Fri, Mar 03, 2023 at 08:36:09AM -0800, Lars Kastner wrote:
I decided to merge #1999 and relocate the discussion on the separation of
experimentalfrom the rest of the source here.
- We all seemed to agree that the user should in some way be made aware of that they are using code from
experimental. Just for the sake of completeness: Please disagree here if you disagree. I do not agree here - but do not strongly disagree- Should code from experimental be available by default? As already stated, if not its discoverability goes down, but that is not necessarily a bad thing. In my pov code should not stay "experimental" for too long.
yep
Question: What do we want to achieve/ avoid?
- lower the barrier to share code
- make it available immediately
- do not promise no further (interface) changes
- ???
- s.o. getting wrong results (yes, we want to avoid that!!)
So if I write a function CoolStuff in experimental (OK, I will write cool_stuff), and expoert it, then
- anyone can play with it
- there is a slight danger that the code will immediately be used programmatically which might break later This would also indicate a more experienced user who might be able to discover the experimental nature alone
- there is a slightly higher chance that s.o. will just interactively play with it - in which the discoverability is important and the breaking of th API irrelevant
- there are/ will be stuff in experimental where the API is rock-solid (e.g. any grobener_basis function will return a groebner basis, any factorisation will return a factorisation, ...) here the experimental part indicated that the algorithm might be improved/ changed - based on feedback, or hooked up automatically or not or ...
How would anyone "discover" or use stuff in experimental - unless its automatically there? If they are on a mailing list, there might be an announcemet: cool_stuff available in Oscar now - and hopefully they would read further to disciver that they have to actively allow it first. My experience: they will just call cool_stuff and then be annoyed since it was not there and hopefull go back to read more.
So: what are the goals/ what are the pitfalls?
Claus
These were the ideas of @fingolfin :
Indeed. Some options come to mind:
* maybe accessing code in experimental should be "guarded" and require `using Oscar.Experimental`, or require writing `Experimantal.FOO` (so `Experimental.galois_group(...)` * pro: obviously experimental * con: less discoverable * or maybe the primary functions should have a ***@***.***_experimental()` call in them, which prints a warning about them being experimental (possibly only the first time they are called) * ... other ideas?So I'll cc some people who might be interested: @micjoswig @fieker @benlorenz . Maybe we can discuss a bit here and then decide during one of the next Friday meetings, preferably when we are all there, unless we immediately find a consensus here.
-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011 You are receiving this because you were mentioned.
Message ID: @.***>
In response to Claus' question: "What are the goal?"
Maybe the intention is to avoid wrong results?
If so, I like the idea from a user perspective. Either having a warning (or having to invoke using Oscar.experimental) would then alert me that the results that I am computing might not be accurate. So the computed results must be used with care.
In case we agree to go ahead with this, I like both the warnings and using Oscar.experimental. The latter is easier to maintain, while the former might be better to inform that some results were computed with experimental code (hence, whenever one sees an "experimental code was used"-warning, the following result and results computed based on this must be used with care), while the methods in src/ would not trigger such a warning.
From the user perspective I prefer the @warn_experimental() solution. But putting @warn_experimental() in each function seems annoying. Are there ways to automate it?
On Wed, Mar 15, 2023 at 06:47:53AM -0700, simonbrandhorst wrote:
From the user perspective I prefer the @.***_experimental()
solution. But putting ***@***.***_experimental()in each function seems annoying. Are there ways to automate it?
But why? So me, code in experimental is expected to work - and "never" return wrong results. So why put warnings in place? Careful: the next time it might be faster? The output (format) might change? Philosohically, I can understand the problems, but is this really a problem or a hypothetical one?
-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1470041724 You are receiving this because you were mentioned.
Message ID: @.***>
I would be hesitant to use experimental code to prove a theorem unless I wrote the code myself and know exactly what I am doing.
On Wed, Mar 15, 2023 at 09:03:53AM -0700, simonbrandhorst wrote:
I would be hesitant to use experimental code to prove a theorem unless I wrote the code myself and know exactly what I am doing.
Than we need to document our intend better. To me, experimental is supposed to be correct - as normal code, however, not in oits final shape and with the wrong method, wrong API, but mathematically correct. Code that is not correct, maybe should be in example or in a draft PR, but my intentions is that correctness is guaranteed
-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1470314257 You are receiving this because you were mentioned.
Message ID: @.***>
Where would you put code for a n algorithmic paper that is submitted but not yet published?
What is your goal? Do you want to keep that code secret? What is special about that code?
On Wed, Mar 15, 2023 at 11:37:07PM -0700, Michael Joswig wrote:
What is your goal? Do you want to keep that code secret? What is special about that code?
Why is it secret? It is as easy to find as everything else... Given that we are spread over >= 5 repositories, browsing a single directory will never show everything. Given that we also make use of sub directories and that many functions do not have a "natural" place (e.g. operation on ideals: part of groups or of ideals?, conversions (Ring -> additive group, ...) this is also unavoidable.
I find key new features mostly using word-of-mmouth and tab-expansion. Code usually by "methods", @.***" or "git grep". Eventually possibly in the "News" section of the "Release Notes" - once we have them...
As I said before: the goal to me is make imperfect (but correct! (up to the usual numner of errors)) available as soon as possible, while indicating to programmers that the APIs and such may be still subject to change. Futhermore, I'd like it as easy and painless as possible for all parties involved. I got annoyed (Magma, Maple) too frequently when trying some announced new feature to learn that it is only available after some secret activation (usually mentioned in the original posting, but usually also lost in the trickling down to me)
Annotating every (exported) function with a warning manually will just result in people not doig this. If you can do it automatically, then maybe.
What is the problem with experimental? Bad name? Wong concept? Abuse?
-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1471399555 You are receiving this because you were mentioned.
Message ID: @.***>
If there is no difference between using code from experimental to normal code, there is no point of having experimental.
Either the code in experimental is different for the user (unstable API, maybe wrong results), then the user should be made aware of this explicitly. Or it is not, then this whole discussion is moot and experimental should just be removed.
What is your goal? Do you want to keep that code secret? What is special about that code?
That was meant as questions for @simonbrandhorst
@micjoswig One of my goals is to put my code somewhere other people can use it [at least the parts that are of general interest] and where I myself can use it in 3 years. Because if I keep it private it starts to rot.
Let's say if one of us convinced that the code is correct this is good enough for us.
But what if someone external believes their code is correct and wants to submit to experimental although his algorithm is only in preprint form. Probably we do not have the time to review it in detail. Should we reject that? Put it in examples? Experimental? src?j
On Thu, Mar 16, 2023 at 12:27:57PM -0700, simonbrandhorst wrote:
@micjoswig One of my goals is to put my code somewhere other people can use it [at least the parts that are of general interest] and where I myself can use it in 3 years. Because if I keep it private it starts to rot.
Let's say if one of us convinced that the code is correct this is good enough for us. But what if someone external believes their code is correct and wants to submit to experimental although his algorithm is only in preprint form. Probably we do not have the time to review it in detail. Should we reject that? Put it in examples? Experimental? src?j
That cannot be decided in general - this will, like in GAP I assume involve a reviewing process. Similar to papers, here the author is trying to convince us it is correct - and there will be few formal rules about the process....
-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1472623325 You are receiving this because you were mentioned.
Message ID: @.***>
On Thu, Mar 16, 2023 at 12:27:57PM -0700, simonbrandhorst wrote:
@micjoswig One of my goals is to put my code somewhere other people can use it [at least the parts that are of general interest] and where I myself can use it in 3 years. Because if I keep it private it starts to rot.
Let's say if one of us convinced that the code is correct this is good enough for us. But what if someone external believes their code is correct and wants to submit to experimental although his algorithm is only in preprint form. Probably we do not have the time to review it in detail. Should we reject that? Put it in examples? Experimental? src?j
... but even then: unless we are convinced it is correct (up the usual...), it is geing to be at most in examples...
-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1472623325 You are receiving this because you were mentioned.
Message ID: @.***>
I am okay with code in experimental to be "correct" so that the only basic difference is an unstable api.
But right now when I code, I don't really know myself which part of the schemes is experimental and which in src.
Perhaps it will help when we have all our exports accessible via grep?But it seems that @which experimental_foo returns the path and solves this issue too.
So I start seeing your point @fieker .
On Thu, Mar 16, 2023 at 12:18:32AM -0700, Tommy Hofmann wrote:
If there is no difference between using code from experimental to normal code, there is no point of having experimental.
Either the code in experimental is different for the user (unstable API, maybe wrong results), then the user should be made aware of this explicitly. Or it is not, then this whole discussion is mood and experimental should just be removed. For the interactive user, there is no difference at all For the programming user, they will read the code and know about it
For the contributer, I still maintain the psychological hurdle is lower to have experimental....
-- Reply to this email directly or view it on GitHub: https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1471435268 You are receiving this because you were mentioned.
Message ID: @.***>
A bit late to the game, but here is my view:
- code in experimental should strive to be correct like all our code (but of course bugs happen)
- but it may be incomplete (
error("this case is not yet implemented")), unpolished, API not yet settled, etc. - thus I am fine with people using it, but I want them to be aware there are sharp edges and also that they might have to adapt to API changes in the future
- in fact code in experimental also might go away again instead of graduating to
src - of course right now, we are at a 0.x release of OSCAR anyway, so strictly speaking all our APIs can still change, so this is more a question of which degree of changes to expect
- I am not sure if all programming users will read the code, in fact I doubt it (and honestly, I find that OK -- I read a lot of code but I cannot always read all of it!)
- for me the goal of
experimentalis indeed to lower the entrance barrier for code
One can indeed use @which to find out if a function comes from experimental, but one has to (a) know about this, and (b) even begin to guess that this might be relevant -- if I find galois_group, I then read its docstring, determine it does what I need, and happily use it. Why would I even consider to think it might be experimental??
Here is a suggestion what we could do to move forward:
- maybe the manual could have a section "Experimental" which explains what the meaning of experimental is, and how to determine code is experimental (the
@whichtrick) - in addition, maybe doc strings for code in experimental could mention prominently a sentence like "This function is part of the OSCAR experimental code. It is perfectly fine to use, but please read REF to fully understand what this means"
- we could even write our own drop-in variant of the
md_strmacro, which automatically detects when it is used insideexperimentaland then adds that verbiage to the end of the docstring - perhaps "experimental" is a name that still sounds too "risky" to some people. I.e., for some the connotation may indeed suggest that the code is unreliable. Perhaps we can find a better name. How about "staging"? Or "develop" / "development" ; or ...?
One minor point I forgot: perhaps each experiment could have an (optional!) README.md that says what the experiment is about and what its status is
I would like to open up this discussion again so we don't have to do it all on Friday.
From what I can see, most people in this thread agree that the code in experimental may have an unstable API (and I think this is basically the defining property of experimental since who would consciously write wrong code?). But having an unstable API breaks the semantic versioning business.
So, how do we proceed after 1.0? Let me throw the following options in the discussion:
-
experimentalhas to follow the Semver guidelines, so no API changes till 2.0. This would essentially mean to dropexperimentalcompletely from my point of view -- every contributor would have to start their own little package until the API is stable. -
experimentalis officially declared "unstable" and we somehow notify users by one means or another from the list above https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1485043175 - [NEW!] We start a separate package with OSCAR as a dependency in which we put all experimental code. This package could then keep a 0.* version number forever and whenever some code is stable enough it can move to "OSCAR stable". This would of course require that currently no code in
srcdepends onexperimental(but in the long run I would declare this a "feature" enforcing our guidelines).
Pinging @lgoettgens @HechtiDerLachs @HereAround @StevellM since they seem to be the biggest contributors to experimental from what I can see.
I think everyone is happy that experimental exists and that it does not have to follow semver. This answers https://github.com/oscar-system/Oscar.jl/issues/2011#issuecomment-1940901930.
But it is still not clear, if one should be made aware that this can break at any time. I am mainly thinking about the code of people we don't know about. They happily use galois_group or your other favourite experimental function in their scripts and suddenly it does not work anymore, because someone added or removed another return value (or we removed the function altogether, because why not?). And no, they did not read the online documentation or called @which. They were looking for Galois groups and found galois_group.
We are currently looking into possibilities to use Documenter to mark experimental functions in the generated docs. Stay tuned.
We are currently looking into possibilities to use Documenter to mark experimental functions in the generated docs. Stay tuned.
Triage sees this as the only reasonable way. Is there any update @micjoswig ?
The Documenter fix is unrelated. For this to happen we need to tap into the documentation which is collected by Julia itself. And there is no clean way to do it if we want it automatized, which we definitely want, since otherwise authors in experimental will just ignore this part.