ast
ast copied to clipboard
att
incomplete KSH_VERSION string reported
Description of problem: commit c99e9ff13e8fc1bf46127e6f28d0f0bde16ffda0 is currently the basis for a brand new "ksh-devel" package thankfully provided by MacPorts for OSX users. when installed ksh reports its version as
2020.0.0-beta1
w/o the invariant leading "Version A" part that seems the only way to reliably tell that the current shell is some ksh93 variant (e.g. ksh93u+ says : "Version AJM 93u+ 2012-08-01" which also fits the "^Version A" regex pattern...)
Question 1: is the omission of the leading "Version A" part a bug and could it please be fixed?
Question 2: I had to exchange several mails with the macports package maintainer to collect the relevant bit that we are talking about commit c99e9ff13e8f. would it be possible to include the commit HASH into the version string dynamically during the build process like so:
Version A 2020.0.0 (c99e9ff13e8f)
12 digits should suffice for a couple of hundred years of further active development ;)... I believe this would greatly help generally.
Question 3: is it an error that c99e9ff13e8f states the version as 2020.0.0-beta1 despite the release 2020.0.0 being 2 months older? .
in any case the main issue is the current omission of "Version A" in the KSH_VERSION value since I need that for identifying that a ksh shell is running in my scripts...
Ksh version:
commit c99e9ff13e8fc1bf46127e6f28d0f0bde16ffda0
How reproducible:
always
Answer for question 1
On my system:
$ ksh --version
version sh (AT&T Research) 2020.0.0
The "AJM" portion is no longer present because we no longer support optionally including features when building ksh. You can expect a ksh 2020.0.0 version to have the same features on every platform. Also, you cannot, and should not, expect the version string to begin with "Version A". On macOS:
$ /bin/ksh --version
version sh (AT&T Research) 93u+ 2012-08-01
It's probably safer to see if $KSH_VERSION exists.
Answer for question 2
would it be possible to include the commit HASH into the version string....
Maybe. At present we use the output of git --always --dirty --tags which does not include the commit hash if it corresponds to a tag. If git, or git meta data about the source, isn't available there is a hardcoded fallback. So if we changed this that too would have to include the commit hash for distro builds. Not sure how I feel about that.
Answer for question 3
is it an error that c99e9ff states the version as 2020.0.0-beta1 despite the release 2020.0.0...
No, it is not an error in the sense I think you mean the term. However it is not optimal. We are still learning how to best manage changes to this project.
reply to answer 1: it is not about the output of `ksh --version' but about KSH_VERSION and what the 'ESC ^V' shortcut tells you at the ksh prompt. and it definitely (unfortunately, for sure) is insufficient to check for existence of KSH_VERSION. its existence is necessary but not sufficient to ensure it is ksh93: pdksh descendants (oksh, mksh) both have KSH_VERSION set (as probably the original ksh88, too, but I am guessing here). so there is demand for a means of unambiguously determining what shell exactly is running.
and ksh93 so far always has reported a string starting with said "Version A" at it's prompt. this was invariant. it even is in the 2020 release. only now I see something different. I now, in addition realise that c99e9ff actually does still do the same with 'ESC ^V' (which is good in my view) but no longer -- as reported yesterday -- when querying KSH_VERSION. at least this discrepancy is a bug I presume? and while I understand that "Version A" is historical cruft it still seems formally to be correct, even if redundant, so no harm done keeping it? right now it seems the easiest route would be to just continue to include "Version A": this would up to now suffice to determine that I am dealing with a fully ksh93u+ syntax compatible shell. but if you decide upon another scheme, sure. it only needs to be such that one can extract the "I am a ksh93 shell" info.
reply to answer 2: I would say, a completely consistent versioning, i.e. always including the hash, would be preferable. even the releases can be identified by the respective hash, of course. they are just additionally tagged. and I would argue that all descendants of 2020.0.0 should keep that denomination as part of the version string. "2020.0.0 (c99e9ff)" seems more correct than "2020.0.0-beta1 (c99e9ff)" quite obviously given the actual naming history (beta1 was the beta before 2020.0.0 etc.). in any case, it definitely would be helpful to include the hash always in my view
reply to answer 3: I retract "error" than. but it sure is not desirable that a ksh compiled from commit c99e9ff identifies itself as 2020.0.0.-beta1 although it is newer than the release that was coming after the original true beta1 version. and the current absence of the hash makes it definitely ambiguous and confusing: "am I running the beta1 from september or am I running yesterday's comitted version?"
as just one example of a working unique version identification: my DVCS of choice is https://fossil-scm.org/home/doc/trunk/www/index.wiki (much more lovable than git but that's a different story -- no flame war intended...) and it consistently identifies itself like so:
This is fossil version 2.10 [9d9ef82234] 2019-10-04 21:41:13 UTC
2.10 is the last released version but 9d9ef82234 is (or can be) some much younger commit. if bugs are to be reported I just can report the hash to the fossil maintainers and everybody knows about what exact version in between releases one is talking about. seems bullet-proof to me. adding the date/time is not bad either, just to tell the user how old that version is..
Note that the 2010.0.0-beta1 version is a consequence of how we currently handle the case where either the git command isn't available or the .git meta data directory isn't available (e.g., in a distro source archive). In that case we fallback to the hardcoded value in the meson_options.txt file. A value that is generally only updated when a new release is being made. It is not going to be updated every single time a commit is made. And that file has the correct value in the 2020.0.0 branch that MacPorts should be using:
option('fallback-version-number', type : 'string', value : '2020.0.0')
So the question is what version is MacPorts actually using?
Regarding the ksh version issue. This is what I see using ksh built from the master branch:
$ ksh
KSH PROMPT:1: set -o vi
KSH PROMPT:2: Version A 2020.0.0-beta1-170-gc99e9ff1
Nonetheless, you should not expect the A to be present. Historically that identified an optional feature (command auditing) was included and therefore is not guaranteed to be present. Were there any ksh versions in widespread use built without that feature? I have no idea but that doesn't change the fact you cannot simply assume it will be present. The fact we still include that option letter in version string is a mistake in my opinion since we've decided to include the feature unconditionally.
If you need to detect a "real" ksh rather than a clone it should be sufficient to check for AT&T Research in the output of ksh --version. It would be wrong for a clone to include that string in its version string. We're also very unlikely to remove that string before the heat death of the universe. In fact, you could argue it should be present in the vi mode [ctrl-V] output. Why the two strings are different is unclear. I think they should be identical.
On 18.11.19 03:43, Kurtis Rader wrote:
Note that the
2010.0.0-beta1version is a consequence of how we currently handle the case where either thegitcommand isn't available or the .git meta data directory isn't available (e.g., in a distro source archive). In that case we fallback to the hardcoded value in the meson_options.txt file. A value that is generally only updated when a new release is being made. It is not going to be updated every single time a commit is made. And that file has the correct value in the 2020.0.0 branch that MacPorts should be using:option('fallback-version-number', type : 'string', value : '2020.0.0')So the question is what version is MacPorts actually using?
as said, the precise information is not derivable from the installed versions, since the hash is not part of the version string. my understanding regarding the new 'ksh' package is that it is the last official release from your side, while 'ksh-devel' is -- according to eamil from the package maintainer -- derived from commit c99e9ff.
which makes me wonder why at least in the latter case the hash is not in the version string since it should if I understand you correctly.
I understand that it is not necessarily straigh-forward in the current setup to always include the hash in the version string, but this should be doable and would be prererable. at least the hash really should always be present for everything expect release. in the present example: the ksh-devel package version. if I understand correctly and the hash should be there that would mean the local setup of the package maintainer differs somehow and the hash does not make it to the version string, no?
On 18.11.19 03:54, Kurtis Rader wrote:
If you need to detect a "real" ksh rather than a clone it should be sufficient to check for
AT&T Researchin the output ofksh --version. It would be wrong for a clone to include that string in its version string. We're also very unlikely to remove that string before the heat death of the universe. In fact, you could argue it should be present in the vi mode [ctrl-V] output. Why the two strings are different is unclear. I think they should be identical. as mentioned, what I need is a way that an already running shell can determine its own identity (in my case: when sourcing function definitions whose definition partly depends on which shell (possibly bash, possibly mksh, possibly ksh93 or ksh2020) is used.
if the "Version A"part simply is still true in your case I don't see that you need to delete it, even if redundant. on the other hand, this is up to you. but some sort of an invariant and sufficiently specific substring in the version string would be really helpful. "ksh20" would do, e.g.
I also repeat that the c99e9ff version (ksh-devel macports: you might check yourself...) yields different output with
ESC ^V and "echo $KSH_VERSION"
how come?
On 18.11.19 03:54, Kurtis Rader wrote:
If you need to detect a "real" ksh rather than a clone it should be sufficient to check for
AT&T Researchin the output ofksh --version. It would be wrong for a clone to include that string in its version string. We're also very unlikely to remove that string before the heat death of the universe. In fact, you could argue it should be present in the vi mode [ctrl-V] output. Why the two strings are different is unclear. I think they should be identical.
I am sorry to be repeatitive: 'ksh --version' is of no help. I need $KSH_VERSION.
I am sorry to be repeatitive: 'ksh --version' is of no help.
Understood, but there is no simple solution since the legacy ksh93u+ release emits two different strings for what are logically identical use cases. For interactive detection I still do not understand why the Version A detection is failing for you. That string, as I've already stated, is present in the [ctrl-v] output from ksh built from git master branch head. On my system:
$ /usr/local/bin/ksh
KSH PROMPT:1: set -o vi
KSH PROMPT:2: Version A 2020.0.0-beta1-170-gc99e9ff1
Given that people are, albeit incorrectly, depending on this I'm happy to add a comment to the code to help ensure it is never removed.
I can't speak to what is going on with the MacPorts version as I don't use MacPorts. I appears from what you have said that they are building an experimental ksh from the master branch head without having either a git command or they have excluded the ./.git directory from the build source tree. In which case, as I've already noted, we fallback to a hardcoded string representing the most recent "version" for the current branch.
We currently allow people to specify an arbitrary string (assuming the prefix is a legitimate semantic version) when configuring the build. For example, meson -Dfallback-version-number='2020.0.0-gc99e9ff. It seems to me the MacPorts build recipe should probably leverage that mechanism.
That string, as I've already stated, is present in the [ctrl-v] output from ksh built from git master branch head.
as explained, with the c99e9ff derived macports ksh-devel package,
ESC ^V --> Version A 2020.0.0-beta1 but print $KSH_VERSION -> 2020.0.0-beta1
that's the situation...
Given that people are, albeit incorrectly, depending on this I'm happy to add a comment to the code to help ensure it is never removed.
I would find this useful for the time being in order to have an easy means to test for whether a ksh93 is running. in any case ESC ^V and print $KSH_VERSION should definitely always have the same value/output. do you agree? and independent of how the macports people build it, exactly.
I appears from what you have said that they are building an experimental ksh from the master branch head without having either a git command or they have excluded the ./.git directory from the build source tree. In which case, as I've already noted, we fallback to a hardcoded string representing the most recent "version" for the current branch.
I can ask them. I am quite sure they choose the path of least resistence (clone/build). as mentioned, the package is not head (of course -- they are not building it every hour ;)) but derived from commit c99e9ff. to me it simply shows that the build procedure seems too fragile in this respect. how version strings are generated and consistency between ESC ^V and KSH_VERSION is ensured and how to get the commit hash into the version string to avoid any ambiguities. this should be ensured upstream not by the package maintainers of the different distros....
It seems to me the MacPorts build recipe should probably leverage that mechanism.
I can ask. but see above: ensuring deterministic version string(s) upstream is sure preferable. I do not see any benefit in making the version string user settable, really.
I can ask. but see above: ensuring deterministic version string(s) upstream is sure preferable.
Agreed, but not possible. I'm thinking we should do the same thing, more or less, that the Meson project does. The master branch defaults to the last major/minor version plus a magic value that means "not a stable release". In the case of Meson that is currently "0.52.999". We would use "2020.99.99". That way we only have to update the master branch fallback version once after forking a beta or stable branch, and even then only if a new major release has been created.
I do not see any benefit in making the version string user settable, really.
Agreed, again. That's why I've got an experimental branch that removes the ability for distros to set the version via a build time flag.
in any case ESC ^V and print $KSH_VERSION should definitely always have the same value/output. do you agree?
Yes. This type of pointless inconsistency drives me nuts. But there are no doubt individuals who will claim that harmonizing these two cases cannot be done because it breaks backward compatibility with ksh93u+.
On 20.11.19 04:51, Kurtis Rader wrote:
in any case ESC ^V and print $KSH_VERSION should definitely always have the same value/output. do you agree?
Yes. This sort of pointless inconsistency drives me nuts. But there are no doubt individuals who will claim that harmonizing these two cases cannot be done because it breaks backward compatibility with ksh93u+.
quite to the contrary: what you say here is competely untrue. why do you not start ksh93u+ and see yourself? of course do ESC ^V and print $KSH_VERSION yield the same result (as does print ${.sh.version}). as it should be. as it is in ksh93u+. as it is not in the reported case causing opening this issue....
On 20.11.19 04:46, Kurtis Rader wrote:
I can ask. but see above: ensuring deterministic version string(s) upstream is sure preferable.
Agreed, but not possible. I'm thinking we should do the same thing, more or less, that the Meson project does. The
not worth the trouble? maybe. not possible? how's that? if build from a git clone of the repo there should be there problem to make the build do that. and the tarballs could be modified to contain a file "version.txt" (or whatever) holding the hash/version used for creating the tarball. and the build could test for presence of that file and use it
master branch defaults to the last major/minor version plus a magic value that means "not a stable release". In the case of Meson that is currently "0.52.999". We would use "2020.99.99". That way we only have to update the master branch fallback version once after forking a beta or stable branch, and even then only if a new major release has been created.
the only clean solution for me seems to always include the commit hash. everything else is marginal. so 2020.{hash} is much better than this 'magic number" that holds no other information than that it is 2020+x (somewere beyond 2020). we are talking about a situation where distros can/will decide to provide "devel" packages quite close to head of master and one needs to be able to unambiguously identify what they actually provide. hence necessity for the hahs. 99.99 does not help any.
I do not see any benefit in making the version string user settable, really.
Agreed, again. That's why I've got an experimental branch that removes the ability for distros to set the version via a build time flag.
see above: hash included? always? even for users downloading tarballs or cloning?
the only clean solution for me seems to always include the commit hash everything else is marginal.
That is not possible if there is no git metadata; i.e., the .git directory isn't present. This is typically the case when distros package the source for building. If that directory is present but there isn't a git command available we could parse the git metadata ourselves using a Python or custom C program. If you have git metadata and a git command, which is normally true if you're building from a clone of this project, you'll always get the most recent commit hash if it does not have an associated release tag.
ESC ^V and print $KSH_VERSION yield the same result
That they don't, now, was a conscious decision. Using $KSH_VERSION in a string or numeric context should act on the same value. I was actually thinking about ksh --version and [ctrl-V] emitting different strings being unexpected and wrong.
On 21.11.19 03:35, Kurtis Rader wrote:
the only clean solution for me seems to always include the commit hash everything else is marginal.
That is not possible if there is no git metadata; i.e., the .git directory isn't present. This is typically the case when distros package the source for building. If that directory is present but there isn't a
gitcommand available we could parse the git metadata ourselves using a Python or custom C program. If you have git metadata and agitcommand, which is normally true if you're building from a clone of this project, you'll always get the most recent commit hash if it does not have an associated release tag.
I understand what you say: the current machinery does not do it. impossible to do? no. look at so many other projects. whether you deem it worth the work to adjust the build process: up to you. but it seems to boil down to what I've tried to explain previously:
building from a git clone: hash information available. could always appear in the version string if you were adjusting the build to do that (even for tagged releases). what argument is there against doing this???
building from tarball w/o git: more "tricky" in that you would have to add the commit hash info somehow to that tarball (most simple mindedly via a standalone text file "thisismyhash.txt" which is only added to the tarballs. something like that. have used that approach myself at times.)
what is not possible about this? if you don't want it, don't see the value, you should state it thus. claiming that it is impossible to tell a software bundle (tarball) exactly how it came about? well....
ESC ^V and print $KSH_VERSION yield the same result
That they don't, now, was a conscious decision. Using
$KSH_VERSIONin a string or numeric context should act on
the same value. I was actually thinking about ksh --version and
[ctrl-V] emitting different strings being unexpected and wrong.
I would say: all three(!) ways of querying the version should yield the same string. but the only string that matters for scripting is ${.sh.version}. and that one needs to be deterministic and should contain a unique and unambiguous identifier for that exact version (a.k.a. the commit hash).
We provide a means, via the -Dfallback_version=1.2.3-whatever build option, to override the hardcoded default if git metadata isn't available. Any distro source archive is free to use that. What I'm trying, unsuccessfully, to communicate is that if git describe --always --dirty --tags doesn't produce a version string we have no choice but to use the hardcoded default or the -Dfallback_version value. We've now made the hardcoded default clearer about the situation. There isn't anything we can do, as a project, about things like the Macports package for the, experimental, master ksh branch not having the version string you expect other than to improve the documentation about how to supply a custom version string when doing the build.
the problem is two-fold:
-
your release(s), seemingly, do not put the hash into the version string even if they could (building from git repo). correct? if so, this should be changed anyway: please put always hash into version here.
-
the hash, you tell me, can't be inserted if 'git describe --always --dirty --tags' is not available at time of building, i.e. if build very probably uses the downloadable tarballs you provide. what about changing the fallback strategy that the build procedure first tries 'git describe --always --dirty --tags' and if that returns error (no. 128, it seems) does next look for a certain file in some place within the projects tree that contains that info? the tarball could contain that file if you were putting it there: it is known at time of building the tarball what state of the repo (last commit hash) we are talking about.
only if that fails, too, revert to what you are doing now immediately: use a hard-coded string.
since I trust that in fact people will in 99.9% build either from cloned git repo or from your tarballs the hash info would be there 99.9% of the time. problem solved, no?
@jghub, Have you ever used a revision control system like Git? In the case of Git the commit hash is derived from the files being changed and the commit message. Other RCS like Perforce use a different approach based on a monotonically increasing change number but that doesn't alter the problem. There is no way to store the commit hash (or change number) in "a certain file in some place" until the commit has been made. But once you change that file you have to commit the change of that file. Which changes the commit hash. That would work if we were willing to always make changes in a two step process:
-
Commit the real change.
-
Write the commit hash from step 1 in "a certain file in some place" and commit that change.
That is not going to happen. There is literally no project on earth which does what you are proposing.
your release(s), seemingly, do not put the hash into the version string even if they could (building from git repo). correct?
That is not correct. The commit hash is included if the current branch head does not correspond to a release tag. For example, as I write this my ksh version is 2020.0.0-beta1-173-g42a580c5-dirty. It's not prefixed by 2020.99.99 despite that change being included in the build for reasons that aren't really relevant to this discussion. The key is that the version tells me my build includes 173 commits since the 2020.0.0-beta1 tag, the most recent commit is 42a580c5 and my git environment is "dirty"; i.e., has uncommitted changes.
If the build was from a commit with an associated release tag there is no need to include the commit hash. Although we could do so if someone can make a good argument for doing so. Note, however, we can only do this if the git command and .git directory are available.
you are not listening/reading carefully and you attitude is presumptuous. your working hypothesis should be that the other person is neither ignorant nor dumb. tit for tat remark at this point would be "@krader1961 Have you ever used a scripting language like ksh?" I do hope you get it why such questions are annoying.
regarding the problem at hand: you are wrong, logically and practically, as far as I can see.
for one, of course the suggested "version file" should never be under revision control. I never said it should... it could be in .gitignore for good measure but basically it is only needed during tarball generation and can be deleted afterwards. in semi-pseudo code like this:
`
first cd to root_of_project ...
hash=$(print -f%.10s $(git rev-parse HEAD)) # or similar print $hash >| .headhash grep -q '^.headhash$' .gitignore || print .headhash >> .gitignore cd .. # or cd to somewhere else, wherever tarfile should reside base=${OLDPWD##*/} tar czf ${base}_$hash.tgz $base || print some problem rm $OLDPWD/.headhash `
then the hash info is available in the tarball. the build procedure would have to check first for available git info and in the second step (if no git clone) for existence of `.headhash' and, if available, use its content for augmenting the version string and only fall back to some generic default if the file does not exist. you are just omitting step 2 right now and use "either git (hash or tag) info or hard-coded generic". that is the whole problem it seems.
whether the hash should be added always, even for unambiguously tagged releases is a matter of taste. I don't care, really. but it should be obvious that each build (from git clone or tarball) should provide a unique identifier in the version string which unambiguously corresponds to certain commit/state of the project. your insistence that this can't be done outside of a git clone is really astonishing.
and the idea to provide some hardcoded 2020.99.99 version string for all non-tagged/released versions which does not allow to tell which version/commit I am actually running is bizarre in my view.
@jghub I don't think it's worth having a version number for each commit of -current (i.e. non-release) versions of a program. It can get in the way of reproducible builds, make the build process more complex, and the -current/-master of any program isn't something that's meant to be supported or packaged anyway.
In a project I used to work on, we'd have our version numbers, perhaps 2.3.4, and then the very first commit after we made a release would be to change the version number to 2.3.4+master And if there was a +master in the version number, you knew you weren't on a release. What do you think of this method?
We used to include a build date (or even a git hash if you had git installed) into the build (of that program I used to work on) and it made the Makefile a pain in the butt and caused programs like iwyu to no longer work on the code. We got rid of that and it made things easier, with an acceptable loss of information. People running -master were expected to be running the latest commit. Otherwise, people should be running a tagged release...
@kshfan
I don't think it's worth having a version number for each commit of -current (i.e. non-release) versions of a program. It can get in the way of reproducible builds, make the build process more complex, and the -current/-master of any program isn't something that's meant to be supported or packaged anyway.
well, it seems the hash is part of the version string for non-tagged versions anyway if you build from a git clone. I like that. to the extent that it does complicate the build process (minimally I would say...) that is the state of affairs anyway. I am just asking for what lot's of other projects do: include the hash info also into the tarballs and use it if build is done from them. I believe this would be valuable for distros also providing "devel" packages -- as macports is doing right now for OSX.
In a project I used to work on, we'd have our version numbers, perhaps 2.3.4, and then the very first commit after we made a release would be to change the version number to 2.3.4+master And if there was a +master in the version number, you knew you weren't on a release. What do you think of this method?
that sounds similar to what krader1961 is proposing (a derivative string of the last release but kept invariant until next release). I do not like this idea for the reasons explained: if I run the devel-package I could never report a bug with reference to the actual binary causing the bug. I only could state the too generic thing "I am somewhere beyond release N"...
We used to include a build date (or even a git hash if you had git installed) into the build (of that program I used to work on) and it made the Makefile a pain in the butt and caused programs like iwyu to no longer work on the code. We got rid of that and it made things easier, with an acceptable loss of information. People running -master were expected to be running the latest commit. Otherwise, people should be running a tagged release...
mostly yes. but if the project provides tarballs for download, those should contain unique version identifiers but whatever means -- either by including the hash (I do not completely get, why this messed up or complicated your build process so heavily) or by consistently tagging all comits from which tarballs are derived.
MacPorts [...] are building an experimental ksh from the master branch head
To make sure it's understood: we are offering that in our ksh-devel port for users who wish to track your development more closely. Normal users should use the ksh port which contains your latest stable release.
without having either a
gitcommand or they have excluded the ./.git directory from the build source tree.
For several reasons, MacPorts does not clone a git repository; it downloads and extracts a tarball that was automatically generated by GitHub. Hence the git command cannot be used to obtain any of the git metadata. We do of course know what git commit we are fetching, so we can supply that information to the build system:
We currently allow people to specify an arbitrary string (assuming the prefix is a legitimate semantic version) when configuring the build. For example,
meson -Dfallback-version-number='2020.0.0-gc99e9ff. It seems to me the MacPorts build recipe should probably leverage that mechanism.
Thanks, I didn't know that existed. I'm using it in MacPorts now. I am not at present using your semver as the prefix, but rather the YYYYMMDD date corresponding to the commit, as is MacPorts custom for identifying development versions. If I should switch this to a semver, is there a specific file in the source code that I should consult to determine which semver to use?
am not at present using your semver as the prefix, but rather the YYYYMMDD date corresponding to the commit, as is MacPorts custom for identifying development versions.
That's fine. In fact, doing that was part of our thinking about switching from the legacy version strings (e.g., 93u+) to a semantic version. In the past ksh had two different versions. Which one was used depended on the context. In a string context you got the 93u+ string. In a numeric context you got the date, in YYYYMMDD format, associated with that version. Now we use the same value for both. The only difference is the presence or absence of the periods and padding: 2020.1.13 is 20200113 in numeric context. You can, but really shouldn't, zero pad the minor and patch components of the string; e.g., 2020.01.13 is also legal.