Convert scripts (aside from the ones used for testing) to POSIX sh

[E5ten]

This partially addresses #77, because while tests still require bash, building can now be done without it.

[landley]

I'm doing bash intentionally, and writing a bash replacement shell.

[E5ten]

But then, even once toysh is a working bash replacement, you'd still already need either bash or toybox to build itself, wouldn't it be better for toybox to be able to build on any system with a functioning POSIX-compatible shell?

[landley]

I have a similar problem with sed/gsed on macos.

What I'd like to do is have tiny "make sed" and "make toysh" scripts that build those tools with no dependencies on platforms where the right tool is absent. (A bit like generated/build.sh but without needing to generate headers either, probably tiny ones as HERE documents.)

THOSE scripts should be portable.

[landley]

This is post-1.0 todo items, though.

What AOSP does is ship prebuilt binaries, which sadly limits the targets it can build on. Part of what I'm trying to do with toyroot is make a known build environment, which can be reproduced (build regression testing!) but doesn't NEED to be. (So building up/under it and reproducing it are orthogonal problems. It's sort of a build checkpoint.)

[E5ten]

I just don't understand the reason for opposing making as much of the build as possible portable, when someone has already done the work for it and it removes the need for a special make target to bootstrap the scripts, especially considering the target doesn't exist yet, and it would allow anyone with a working sh and GNU make to build toybox right now.

[landley]

I plan to implement a make in toybox that handles gmake files. Do you want me to convert the Makefiles to work with BSD make?

The standard I'm trying to adhere to isn't always pure posix.

[E5ten]

I was intending to try to make the Makefiles POSIX compatible at some point, but that isn't really related? In this case the work to adhere to POSIX has been done, what reason is there to not make the build more portable in this case?

[landley]

I'm explaining to you that this is not a design direction I want to go in, and you're telling me that you already did the work. Because that should change the design decision?

The shell standard I've chosen to implement (and require) is bash compatible, not posix sh. I've done some transitional patches in the test suite to make the android guys' life easier, but now that I'm actively writing toysh I'm less interested in nerfing my real-life test data and more interested in making toysh work and making it easier to use to solve these problems.

One of my project goalposts is getting toybox to build itself under itself, zeroing out the airlock binary list from https://github.com/landley/toybox/blob/master/scripts/install.sh#L109 . I could instead change my use of sed to be posix only, elimintate all use of awk, patch bison out of my kernel locally, and so on... in which case why write toybox at all? Toybox needs to eat its own dogfood, but what we provide isn't posix.

I want this project to provide a Linux build environment within which you can build and run traditional Linux software. I'm aware MacOS is not Linux, and diverging farther from it (removing bash and replacing it with zsh). I've chosen to address that with the same "create airlock" strategy I've used for years, in this case statically building the tools some systems haven't got so toybox can build under toybox with a mini-airlock on such systems, by using a shell script that builds JUST those tools with no external dependencies on things like "sed". That is how I prefer to address that problem, and yes that script would need to run under ksh and powershell and fish and possibly OS/2 cmd.exe. (It should be just a list of command lines, possibly without even using environment variables, dunno yet, haven't got there yet.)

[E5ten]

I just don't really understand how the tradeoff makes sense, almost all that the scripts test in terms of bash capabilities are that echo has -e and -n, source instead of . exists, and [ can take == instead of just =, and in exchange toybox has a much more specific requirement to build, needing a specific shell (one which is GPL licensed) to build, instead of POSIX sh, to me it doesn't make sense to have a stricter requirement on things needed to build toybox just because toybox eventually plans to fill those needs (and would still be needed to build itself regardless).

[illiliti]

Please don't force people to use bash. Non-portable requirements is always bad idea, especially if it's unnecessary

[tpimh]

As I mentioned in the issue I opened a while ago, not all systems have bash installed. And with a recent trend of shifting away from GNU, it's even more important now. Would be great to be able to build toybox on a system with only dash or busybox's ash available. Right now I only install bash on my Alpine system, because I need to build toybox, all other packages I build only require POSIX shell. Also it would be much less work to make toybox self-hosting without the need to implement these bashisms support in toysh.

[landley]

That's why I'm making toysh handle bash syntax, and why I have a plan to teach the build to create toysh and sed binaries to run the rest of the build under when the host hasn't got bash or gsed.

"Here's a problem." "Here's how I plan to fix that." "I repeat that the problem exists." "As I said, I want to do this to fix it." "It sure is a problem."

[E5ten]

But to me, even once your fix is in place (which isn't the case yet) it would still make building toybox less convenient without bash, because one would have to know about the toysh make target, build that, and then figure out how to tell the build system to use that as sh (I guess by installing it as sh) which they might not even want to do. But even if they didn't have to do that, having to build the toysh target just to be able to build toybox is more of a hassle, for no apparent reason, because the build scripts could just be portable in the first place.

[E5ten]

It's not like being able to build toybox with POSIX sh in any way precludes toysh from being a bash replacement.

[tpimh]

"Here's a problem." "Here's how I plan to fix that."

Now this pull request is a solution. Maybe this proposed solution is not how you plan to fix it, but it can fix it for many others while toysh is not ready and make it easier to fix it later according to your plan.

[konimex]

That's why I'm making toysh handle bash syntax, and why I have a plan to teach the build to create toysh and sed binaries to run the rest of the build under when the host hasn't got bash or gsed.

Honestly, what does this supposed to mean? Since my use case is pretty much Linux without GNU when possible, using bash is to be minimised.

So, if I'm correct, in case of bash and gsed not existing, you're thinking of making the build system build toysh and toybox sed first, and make use of the built toysh and toybox sed to build the rest of the toybox suite? But how exactly can one use that approach if configure has a hard #!/bin/bash requirement and it doesn't even exist?

[landley]

Fun fact: a script can be run with a different shell from the command line, with the #!/ functioning as a # comment. A portability wrapper going "toysh scripts/make.sh" has been on the todo heap ever since http://lists.landley.net/pipermail/toybox-landley.net/2019-January/010031.html (or maybe just changing how the Makefile calls stuff, perhaps a new toybox_portabuild target that does the simple sed and toysh builds and airlock pathing and such) but I just haven't had toysh in defconfig yet so haven't gone there.

I'm going to unsubscribe from this thread. Have fun bikeshedding, I find this hugely demotivational and might step away from the project for a while and go focus on $DAYJOB instead after this release, because this is a hobby not a job (the patreon is less than 2% of my income) and being repeatedly accused of incompetence and having explicit "that is not the approach I want to take" ignored FIVE TIMES has kind of taken the fun out of it.

[nykula]

Existing scripts say bash but they are mostly compatible with pdksh and mksh, except <() in make.sh which I dummy out locally. Where I don't have gmake, e.g. on my phone, I build like this:

$SH scripts/genconfig.sh; (cd kconfig
  for i in zconf.tab.c lex.zconf.c zconf.hash.c; do ln -s ${i}_shipped $i; done
  cc $CFLAGS -static -o conf conf.c zconf.tab.c \
  -DKBUILD_NO_NLS=1 -DPROJECT_NAME='"ToyBox"')
kconfig/conf -D /dev/null Config.in >/dev/null
sed -Ei 's/.*_(BC|DIFF|EXPR|GZIP|TCPSVD|TR|VI|WGET)[= ].*/CONFIG_\1=y/' .config
sed -i 's/<(.*)/:/' scripts/make.sh; CFLAGS="$CFLAGS -static" \
  HOSTCC="cc $CFLAGS -static" CPUS=8 $SH scripts/make.sh

Might be of use to somebody in this thread. In fact, many software build systems are fine with /bin/bash linked to mksh, I build this way lots of packages not only toybox.