libelektra
libelektra copied to clipboard
ElektraInitiative
[decisions] extra libraries
@markus2330 Is the only open question here, how many extra libraries we need? If so, I would propose leaving this open for now and creating an issue to decide before 1.0 That way we can start with the implementation of the new operations library.
We could start the new operations library but nevertheless we should continue with the work to architect how the libraries should look like. Important to consider is, however, that we should focus on cleanup first, and then create new things. Otherwise the new libs will get as messy as the old ones.
We could simply start by moving & merging keyIsBelow as proposed in #4346. That would be a sort of cleanup and would get the library started. From there we can see what else we find. I think it is pretty clear now that creating an operations library is a goal, so we have to start somewhere.
AFAICS several core functions (e.g. ksCut) depends on keyIsBelow*, so it is probably not a good starting point. We definitely do not want wrong library dependencies from the start and circular dependencies is a strong architectural smell.
AFAICS several core functions (e.g. ksCut) depends on keyIsBelow*, so it is probably not a good starting point. We definitely do not want wrong library dependencies from the start and circular dependencies is a strong architectural smell.
Yes, keyIsBelow* is used in some parts of libelektra-core and libelektra-kdb. IMO having a dependency from libelektra-kdb to the new libelektra-operations would be fine. As for libelektra-core, I only found references from these functions: ksCut, ksFindHierarchy, ksRename, keyReplacePrefix.
None of these, are used anywhere inside libelektra-core (except among themselves) and I don't really see why these would be core minimal API. So I would actually move them to libelektra-operations as well.
That said, ksIsBelow could actually be considered core minimal API and should probably stay in libelektra-core. It's is kinda the only thing that (publically) defines the hierarchy of a KeySet.
So my new suggestion would be:
- Start
libelektra-operationsby movingksCut,ksFindHierarchy,ksRename,keyReplacePrefixthere. - Rewrite
ksIsBelowinto a single function, but keep it inlibelektra-core.
IMO having a dependency from libelektra-kdb to the new libelektra-operations would be fine
Yes, I agree.
None of these, are used anywhere inside libelektra-core (except among themselves) and I don't really see why these would be core minimal API. So I would actually move them to libelektra-operations as well.
I fully agree!
So my new suggestion would be
Perfect, please update the decision (if not yet done).
Please always ask for review/write status reports in the decision PRs, so that I know what to review next.
@markus2330 Summary of open questions:
- Prefix(es): I proposed
elektraX*,elektraE*forlibelektra-operationsandlibelektra-ease. - Library stability: I proposed these rules:
- Everything in this repo will be released together with the same version numbers.
- Semantic versioning guarantees only apply between this repo and the outside, but not within this repo. An application built for libelektra-core v1.0.0 will work with libelektra-core v1.x.y. But any libelektra-ABC va.b.c only works with libelektra-XYZ va.b.c. In other words, all of Elektra's packages must be installed with the same version. Any proper package manager setup should already enforce this.
- Some libraries have headers
elektra/foo/unstable.hthat are exempt from semantic versioning.
EDIT: In the latest update, I added unique prefixes for all libraries. I also explicitly listed static libraries that will be linked into other libraries. Finally, I merged the plugin libraries into one library with two headers.
Thank you for continuing work on the decsision! :heart:
elektraE*
I am not a big fan of one-letter shortenings but if we do it, we should use it consistently for all libs.
In particular the capitalization is confusing, e.g. "elektraEKey" is misleading. elektra_e_ks would be better...
What I could imagine is that the prefix is elektrax, where x is a consonant (not a vowel, i.e., a, e, i, o, u) for each lib. E.g. elektraxKey would be imho easier to read.
What exactly was wrong with elektraEase as prefix?
elektraX*
Again, elektrax or elektra_x would imho look better. Furthermore, imho the library name should somehow match with the prefix. So if we use "x" for extension, the lib should be called libelektra-extension.
Everything in this repo will be released together with the same version numbers. Semantic versioning guarantees only apply between this repo and the outside, but not within this repo. An application built for libelektra-core v1.0.0 will work with libelektra-core v1.x.y. But any libelektra-ABC va.b.c only works with libelektra-XYZ va.b.c. In other words, all of Elektra's packages must be installed with the same version. Any proper package manager setup should already enforce this.
Yes, fully agree (the word "already" should be removed, though). Can you clarify this in doc/VERSION.md?
Some libraries have headers elektra/foo/unstable.h that are exempt from semantic versioning.
This is not how reliable software can be. You can have private/unstable libraries but not private/unstable header files which are part of a seemingly stable library. The library would stop being stable if it provides such an interface. The reason is: Maintainers only know which libraries (in which versions) link against which applications, as this is information you have without studying the source code. On library upgrades, they must be able to safely assume that all applications will continue to work. (Unless the SOVERSION changed, which we do not want to do, at least within Elektra 1.*.)
elektra_e_ks would be better
That would be a totally different naming scheme than we have right now...
What I could imagine is that the prefix is elektrax, where x is a consonant (not a vowel, i.e., a, e, i, o, u) for each lib.
That would 1) be very limiting 2) could become quite confusing.
If you like it better, we can use elektrax instead of elektraX for libelektra-operations, but capitalising the X would be consistent with the camel case rules.
What exactly was wrong with elektraEase as prefix?
I thought it could result in awkward and annoying names, depending on what functions end up in this library. It could happen that a function is not minimal (i.e. not in the core), but still very useful. With the rules currently described in the PR such a function would end up in ease, if it can be implemented with the public API. I could live with an additional E/X in the name, but adding always adding Ease sounds really awkward to me.
Maybe the solution is to change the rules of when something goes into libelektra-operations and when it goes into libelektra-ease (do we really need two libs?). Then the more common "extensions" could all use elektraX* and elektraEase* is only for rarely used stuff like SHA hashes or topological sorts.
This is not how reliable software can be. [...] Maintainers only know which libraries (in which versions) link against which applications, as this is information you have without studying the source code.
That's not true, the one crucial piece that (distro/package) maintainers do have is documentation from application developers. In that documentation application developers should state, if they access some private APIs and depend on specific versions of Elektra (not encouraged).
On library upgrades, they must be able to safely assume that all applications will continue to work.
You cannot assume that without also making some assumptions about what application developers do and don't do. Application developers could always go in and manipulate the contents of a struct _Key directly if they wanted to, even though we don't consider that public/stable API. There is nothing stopping them and things will break, if we change struct _Key.
If you want something that is super stable, you have to make everything (including the contents and layout of all structs) part of the public API. But then why even have a keyNew function? Why return allocated Key *? You could just make things easier for everybody and let the API caller allocate everything (it would definitely be more minimal).
The important part about the unstable.h headers, however, is that I never intended for them to be used in applications. At best they might be used in external libraries, which should then clearly state that they need specific versions of Elektra to work, or take some other precaution to still present a stable API themselves. But mostly these headers should be used for cross-dependencies between Elektra's libraries. They could also be useful to introduce new APIs which may need some more work, before the can be fixed forever and made stable. Maybe the solution is to ship these headers in a separate elektra-unstable package that need to be installed separately. That way it should be very clear that using these could cause problems.
I thought it could result in awkward and annoying names
I think you only need to get used to it. E.g. at the first look, the "elektra" prefixes were also surprising for me.
goes into libelektra-operations and when it goes into libelektra-ease (do we really need two libs?)
I think their goals are quite different, so two libs make sense. "operations" really extends performant functionality, "ease" only makes some tasks easier to implement. From library size alone, a split is probably not needed (ease is only 1/3 of core or kdb).
Of course everything depends on a nice architecture of these two libs. Ideally, there should be many use cases that need only either operations or ease. (If everyone needs none or both, two libs obviously don't make sense.)
Then the more common "extensions"
This doesn't work as you often don't know what is/will be common. What is common in some application/plugins might be esoteric in others.
This is not how reliable software can be. [...] Maintainers only know which libraries (in which versions) link against which applications, as this is information you have without studying the source code.
That's not true, the one crucial piece that (distro/package) maintainers do have is documentation from application developers. In that documentation application developers should state, if they access some private APIs and depend on specific versions of Elektra (not encouraged).
It would only work if all applications consistently document the use of private APIs (which is kind of shameful, as everyone knows you shouldn't use them) and all maintainers read this documentation. I personally never saw any application stating that it uses a private API of some lib.
What I already saw often (e.g. fontconfig, glibc, ...) is that libraries do incompatible changes without pumping SOVERSION. This is a recipe for total disaster because it is a serious bug in production code (usually segfaults etc.) that needs to be fixed in every application using the lib. We always need to remember that Elektra serves a purpose: it should make configuration easier. If our APIs are only slightly unstable, applications are much better off not using Elektra at all.
If you want something that is super stable, you have to make everything (including the contents and layout of all structs) part of the public API.
No, actually you have to do the contrary: We have to shift around the contents of our private structs from time to time. Then nobody can rely on it. This is what Linux does with its internal ABIs. Unfortunately, this would be bad for our cache (but only once after upgrade).
I think you only need to get used to it. E.g. at the first look, the "elektra" prefixes were also surprising for me.
(while writing the code) I think "elektra" is different. I can "ignore" that part pretty easily since I know I'm working with Elektra. But a secondary prefix for the individual library is different. IMO there are two case here: 1) It's obvious in which library a function lives so don't have to think about the prefix (e.g. "opts") 2) The function has no clear library it belongs to so I have to actively remember the correct prefix for a function (e.g. "ease" vs "operations")
Once the code is written, yes, "elektra" and another prefix are pretty much the same and I just ignore them when reading the code, unless the prefix actually contributes to the name of a function (e.g. "plugin").
Ideally, there should be many use cases that need only either operations or ease. (If everyone needs none or both, two libs obviously don't make sense.)
I think this will be pretty hard to achieve, if we just go by: "operations" is for performance stuff or needs private API. That's why I originally suggested the somewhat nebulous approach of:
"operations" is for common extensions beyond the minimal API, "ease" is for rarely used stuff, (and to complete the set: "lowlevel-c" is for C specific things)
This doesn't work as you often don't know what is/will be common. What is common in some application/plugins might be esoteric in others.
To some extent, we can estimate how common a use case will be. For example, ksCut is a much more common use case, than "sha256 hash of a KeySet". Even if we don't know the exact use cases. There will of course be cases where it is hard to decide, but in those cases I'd just go by function size. Simple, small functions go into "operations" since it is more likely that there is already a dependency on that one.
use of private APIs (which is kind of shameful, as everyone knows you shouldn't use them
This is why I never called these APIs "private". We do also have private APIs that should definitely not be used. But these "unstable" APIs are just that. APIs without stability guarantees. We know they are public, you are not doing something naughty if you use them, but you have to know the risk and won't get all the stability.
If our APIs are only slightly unstable, applications are much better off not using Elektra at all.
I think you are taking things out of proportion. The main API is still stable. The "unstable" headers would just be additional APIs. It must always be possible to use Elektra sensibly without relying on these unstable APIs.
The function has no clear library it belongs to so I have to actively remember the correct prefix for a function (e.g. "ease" vs "operations")
It is our responsibility to avoid this case. It should be clear to which lib some feature belongs. We need to improve the library design if it is not yet clear.
"operations" is for common extensions beyond the minimal API
This description is probably too generic (it calls for overlap with others). Maybe we should restrict it to KeySet (data structure) operations, like union, cut, append (a whole KeySet).
"ease" is for rarely used stuff
As already said we cannot use rare/often as distinguish feature as this is very dependent on use-case (see also below). Better criteria are topics (like cryptography), minimal/optional (minimal interface in core, other stuff outside) or technical criteria (like needing POSIX interfaces)
Btw. obviously, there should be use cases for everything in every library.
(and to complete the set: "lowlevel-c" is for C specific things)
ksCut is a much more common use case, than "sha256 hash of a KeySet"
I disagree. It can easily imagine some application or plugin where sha256 hash of KeySet(s) is very important and ksCut is not needed.
But these "unstable" APIs are just that. APIs without stability guarantees.
Realistically there always will be some unstable APIs. This is acceptable as long as they are in unstable libraries.
If our APIs are only slightly unstable, applications are much better off not using Elektra at all.
I think you are taking things out of proportion.
I mean it exactly as said. I wouldn't want to contribute to a library which doesn't have stability as important goal. If there is any weak link, there is no stability at all. It doesn't matter if an application crashes because one unstable function changed or the whole API was redesigned: the user experience for application developers is terrible. (Of course with constant redesigns even worse as you effect every application.) Elektra was one of these libraries being terrible to application developers (especially 0.1 -> 0.6) and as a result lost every user in that period.
AFAICT there are two issues we still disagree on. First the library split and the prefixes used and second the unstable APIs. I'll give answers for both, but I'd prefer if we could first focus on the prefix and the library split. Once we've agreed on that we can discuss what to do with unstable APIs. Discussing both at the same time will just lead to chaos. It should also keep future comments shorter, if we're just discussing one thing.
Unstable APIs (for future discussion, ignore for now)
Realistically there always will be some unstable APIs. This is acceptable as long as they are in unstable libraries.
That is just not how things work. To reiterate once again.
libelektra-coremust definestruct _Keyandstruct _KeySet, otherwise it's not standalone- We want
libelektra-coreto be minimal, but (some) functions outside oflibelektra-coreshould still have access to the contents ofstruct _Keyandstruct _KeySet; for various reasons like performance and mmap - We don't want the contents of
struct _Keyandstruct _KeySetto fall under stable public API guarantees, i.e. changing the contents should not require increasing the major version
Based on these constraints I only see two basic options A) break a constraint (1-3 below) or B) do what I proposed all along (4 below).
- Put
struct _Keyandstruct _KeySetinto a separate unstable library. That would meanlibelektra-coreis no longer standalone and has a dependency on this new library. - Put everything that has access to
struct _Keyandstruct _KeySetintolibelektra-core. That would meanlibelektra-coreis not minimal anymore. We'd also have to do the same thing for any similar situation andmmapstoragewouldn't be allowed at all, unlesslibelektra-corealso acts as a plugin itself. - Make
struct _Keyandstruct _KeySetpublic stable API and increase the major version every time we change something. However, at this point we should really change the entire API to allow keys and keysets to be allocated by user code and not just viakeyNew/ksNew. - Accept that some libraries have both a stable API (defined in one set of headers) and an unstable API (defined in another set of headers).
Note: If it makes you more comfortable replace "unstable" with "internal". I just used the term "unstable" as "not stable but public". I'm also only using the term "stable" in terms of API guarantees here. If a function is not "stable" (i.e. "unstable") it doesn't mean that it is more likely to have bugs.
Another benefit of my proposed approach is that you can have a sort of experimental alpha/beta API. If we in introduce a new function foo, but we're not sure about the API yet, we can put it into the unstable header of a library. We can then start using it internally and maybe in some test apps. External people could use it too, but of course with the risk that they might need to change their code. When we like the API, we can release the stable foo simply by moving it from the unstable header to the stable one. It's not a breaking change, because we only break the unstable header, which we don't care about. Of course it could have been in a separate library and the same would be true, but then releasing foo means more work for users. If it just moves from one header to another, any thing that's complied already keeps working. In the source code, you might need to change the include, but it's likely you've already included the public header of foos library anyway. You could also always include both headers. When making foo stable means moving it to a different library, you definitely need to recompile/re-link.
Elektra was one of these libraries being terrible to application developers (especially 0.1 -> 0.6) and as a result lost every user in that period.
Arguable a library that is still in version 0.1 to 0.6 (i.e. pre 1.0) shouldn't have many users. Such a library is clearly still in under development and it's developers weren't comfortable enough to say you can now rely on this and we won't change things out from under you.
Prefixes/Library split
It is our responsibility to avoid this case. It should be clear to which lib some feature belongs. We need to improve the library design if it is not yet clear.
It's not so much about it being clear where a function belongs. For elektraKeysetCut it is kinda obvious that it can only in libelektra-core or libelektra-operations depending on whether it is minimal API. But that's also kind of the point. We are adding an extra unnecessary prefix to make it elektraOperationsKeysetCut just because C doesn't have proper namespaces or modules.
That's why I like elektraXKeysetCut. elektraKeyset* is minimal API, elektraXKeyset is extra stuff. You can ignore the X really easily compared to "Operations". And if you want to be specific saying "Elektra X Keyset Cut" is much easier than "Elektra Operations Keyset Cut".
I disagree. It can easily imagine some application or plugin where sha256 hash of KeySet(s) is very important and ksCut is not needed.
Of course you can imagine a plugin that does this. But is it really something that many plugins would do? However, I agree that discussing about how often something will be used is pointless.
My point was more that to use calculateSpecificationToken (SHA256 hash of keyset) you need a much more specific situation than for ksCut. That's simply because SHA256 is defined for byte sequences, so calculateSpecificationToken needs to make various assumptions about turning a KeySet into the byte sequence that will be hashed.
Maybe we should restrict it to KeySet (data structure) operations, like union, cut, append (a whole KeySet).
I think I get the idea, but I don't really like the definition. It is very hard to say that is and what isn't a "data structure operation".
Also we shouldn't exclude stuff that just operates on Key(s), like keyIsBelow.
This is probably still too general, but maybe we should aim for this:
libelektra-operationscontains extensions the theKeyandKeySetAPI. These functions could be part of a largerlibelektra-core, but are not, because we do not consider them minimal API.
I think this is similar to what yo mean by "data structure operation", I just don't know how really define this notion without being vague or relying on arbitrary examples.
As for libelektra-ease: Honestly, I have no idea how to define that. This lib has always been a bit of random pile of stuff. It's kind of our catch-all for things that don't fit elsewhere and aren't big or important enough for their own lib. However, one distinction I would definitely make is that things like "SHA hash of a KeySet", which don't really make sense in the world just Key and KeySet should never be in libelektra-operations. I think this also falls under your notion of "data structure operation".
Nice trick to reiterate the same argument again and say that nothing should be replied to it :wink: Discussions get much easier if we stop making long treatises outside of the decision text. If you want separate discussion (and I would also would like it as obviously stating a main goal is not enough if there is no decision backing it up), please put it to different decisions (new PRs) or at least separate discussion threads.
There is a fundamental misunderstanding at the beginning of your argumentation: It is not a library that defines how structs look like but header files define the structs. Header files that do not get installed, are not part of the API. So we simply do not install these header files and use them only internally.
(Btw. if there is really a need, and I doubt it, we could provide some serialize functions for Key and KeySet so that people could write something like mmap-storages. But nobody will do it, as writing a cache probably is, together with deciding on the API, the most difficult piece of Elektra.)
It's not so much about it being clear where a function belongs.
The purpose of this decision exactly is to find an architecture where it becomes clear! Long time ago it was also not so clear what belongs to "Key" and "KeySet". E.g. the iteration was part of "Key", which was obviously non-sense. See keyNext: https://manpages.debian.org/wheezy/elektra-doc/keyset.3.en.html
So our goal here is simply to avoid making non-nonsensical decisions, easy, isn't it? :wink:
Btw. maybe we should split "core" to libelektra-key and libelektra-keyset, and maybe we simply should not have libelektra-operations, as even with all the proposed keyset operations, libelektra-keyset is probably still smaller than libelektra-key. And maybe we should also not have libelektra-ease but libraries of concrete topics where it is clear what belongs there.
With core/extensions there is easily some tension and confusion what should be where. I absolutely agree with your concerns on that.
Nice trick to reiterate the same argument again and say that nothing should be replied to it
To clarify: I had written the response already when I noticed how long it was. I didn't want my thoughts to get lost, but also wanted to "simplify" the discussion process. Yes, I could have created a separate PR, but that would just create even more parallel discussions. I wanted to tackle one thing after the other...
I also want to mention, the structs were just an example. I used them because this is a known issue. There are lots of other things currently in kdbprivate.h. Many of the functions are just "the API we needed for X", making those things public and stable with all the semver consequences could easily lead to problems. I won't go into the unstable stuff any further to avoid opening the discussion again.
It's not so much about it being clear where a function belongs.
The purpose of this decision exactly is to find an architecture where it becomes clear!
With "it" I referred to my statement "The function has no clear library it belongs to so I have to actively remember the correct prefix for a function", which I wanted to clarify.
Yes, the purpose of the decision is exactly to clarify these things. I know that, we agree on that. That's not the issue. The issue was that you seemingly misunderstood what wanted to say and now seemingly misunderstood my attempt at clarification.
Btw. maybe we should split "core" to libelektra-key and libelektra-keyset,
Would that even be possible? Every key contains a metadata keyset and a keyset obviously contains keys. I think this would create a circular dependency and even if it doesn't right now, it certainly could affect future API.
Also, I don't see any use case where I'd want to use a Key without using a KeySet, so why split it into two libraries that are always used together.
with all the proposed keyset operations, libelektra-keyset is probably still smaller than libelektra-key
I'm not sure about that keysets are complex. The hashmap stuff alone is quite a bit of code.
Also if I understand you correctly, this would essentially mean abandoning the idea of a minimal API. I'm not against that, but just to be sure I understand you.
And maybe we should also not have libelektra-ease but libraries of concrete topics where it is clear what belongs there.
That part I agree with in theory, but that would probably lead to quite a few libraries with just one or two very basic functions, because they don't fit anywhere else.
I wanted to tackle one thing after the other...
I welcome this approach. If you write in a PR/issue that it should be handled after some other PR/issue, I'll usually be able to follow that approach, as there are no worries that it will be forgotten.
Would that even be possible?
Yes possible but as you mentioned we would need to revert the decision with meta data, as we would not be allowed to expose the KeySet.
A different mindset would be to break the 1:1 relation "class" <-> library, like it already happened in libelektra-core. And each class has it own name prefixes.
E.g.:
- core has key and keyset, methods are named as elektraKey and elektraKeyset
- operations has keyset eXtensions elektraKsXCut (so then you finally get your X)
- ease has the "class" array (calculateSpecificationToken would be in a class related to cryptography)
Then
- everything would be consistent.
- people only need to remember which class is in which library (and do not need to remember every individual method)
If we manage that there are not more than a handful classes per library it should work out quite fine.
And each class has it own name prefixes. [...] If we manage that there are not more than a handful classes per library it should work out quite fine.
I like that approach. I'd use the term "(API) group" instead of "class" to avoid confusion. Each such group should also be documented as a separate group in doxygen (@defgroup/@ingroup, see here).
so then you finally get your X
I'm completely open to better suggestions. I just really really dislike the super unnecessarily long "operations".
The X actually comes from AndroidX which is the (technical) name for all the development libraries that Google provides outside the base Android SDK that is part of OS.
elektraKsXCut
That should probably be either elektraXKeysetCut or elektraKeysetXCut to match the elektraKeyset functions from libelektra-core.
operations has keyset eXtensions
Maybe we should also call it libelektra-extensions or libelektra-extra.
I like that approach. I'd use the term "(API) group" instead of "class" to avoid confusion.
"class" is already used for "Key", "KeySet" and "KDB". Do you want to drop the terminology there, too?
Each such group should also be documented as a separate group in doxygen (@defgroup/@ingroup, see here).
Exactly, full ACK. :+1:
I just really really dislike the super unnecessarily long "operations".
Maybe we simply pick another word. E.g. for ksCut: ksOpCut ksDoCut ksRoutineCut ksActionCut
The X actually comes from AndroidX which is the (technical) name for all the development libraries that Google provides outside the base Android SDK that is part of OS.
Which is actually confusing as our "ksop" library is actually included in the "base" Elektra.
X was very trendy more than 30 years ago (X-Window system).
Maybe we should also call it libelektra-extensions or libelektra-extra.
Extra is short.
Maybe we simply pick another word. E.g. for ksCut: ksOpCut ksDoCut ksRoutineCut ksActionCut
IMO only ksOpCut is okay here. "routine" and "action" both have to much specific meaning in various software engineering contexts. "do" could make some very awkward sounding names.
Also IMO the names should be: elektraOpKeysetCut, elektraDoKeysetCut, elektraRoutineKeysetCut, elektraActionKeysetCut
In other words, the prefix for the library (elektraOp, etc.) should come first, then a prefix for a specific group like Keyset and then the actual name of the function. Think of it in terms of namespaces/classes in C++: elektra::op::Keyset::cut
Extra is short.
elektraExtraKeysetCut is not that short anymore, but I'd be okay with it, since "extra" is a pretty nice name and very clearly tells the user that this additional stuff.
So do we go with libelektra-extra instead of libelektra-operations with prefix elektraExtra*? If so, I'll update the decision and we can hopefully merge this
@markus2330 I've moved the library list into the library_split.md decision and remove the two extra decisions for the new libraries. I hope this is now satisfactory for you.
Let's not merge this until after #4187 is merged. The changes in doc/decisions/README.md will cause another conflict and it's easier and cleaner to rebase this PR after #4187 than to deal with it in the huge #4187 again (especially since there is already a conflict here)