`source` and `alias` interact in an unexpected manner (aliases in sourced files don't work)

[krader1961]

While working on rewriting how the ksh unit tests are run (so that they use a consistent framework) my original implementation was to write a test driver script that looked like this:

# Setup the test environment. Create env vars, a temp directory, etc.

source $the_actual_unit_test

# Check for errors and exit with an appropriate status

The problem is that if $the_actual_unit_test script contains any alias commands they only sort of work. Here's an example (from the glob.sh unit test):

function test_glob {
    # do stuff
}
alias test_glob='test_glob $LINENO'
alias tg="test_glob $LINENO"

# This shows that test_glob and tg is in fact an alias with the expected definition.
type test_glob
type tg
# This also shows that the aliases exist.
alias

# This does not work. It runs the unaliased function which means the first argument
# is not `$LINENO` it's `data`.
test_glob data more_data
# This results in a `tg: command not found` error.
tg data more_data

# This does work; i.e., it uses the alias which means the first arg is `$LINENO` as expected.
x=$(test_glob abc def)

Something about the context created by source keeps the alias from being expanded unless used in a subshell. This might be expected behavior but I can't see any reason why this shouldn't work and the current behavior is not documented.

[kernigh]

ksh(1) says,

       Aliasing is performed when scripts are read, not while they are
       executed.  Therefore, for an alias to take effect, the alias definition
       command has to be executed before the command which references the
       alias is read.

This doesn't tell me when commands are read. I use a syntax error, like a mismatched esac, to observe when commands are read.

# hi.sh
print hi
esac

$ type nksh
nksh is an alias for /home/kernigh/park/ast/build/src/cmd/ksh93/ksh
$ nksh hi.sh
hi
hi.sh: line 2: syntax error at line 3: `esac' unexpected
$ nksh -c 'source ./hi.sh'
/home/kernigh/park/ast/build/src/cmd/ksh93/ksh: source: syntax error at line 3: `esac' unexpected

So if I run hi.sh, then ksh prints hi before it reads the esac; but if I source hi.sh, then ksh reads the esac before it would print hi. After I change esac to x=$(esac), both ways print hi before they read the esac.

# hi.sh
print hi
x=$(esac)

$ nksh hi.sh
hi
hi.sh: line 3: syntax error at line 3: `esac' unexpected
$ nksh -c 'source ./hi.sh'
hi
/home/kernigh/park/ast/build/src/cmd/ksh93/ksh: source: line 3: syntax error at line 3: `esac' unexpected

I observe that ksh expands aliases at the same time as it checks for syntax errors. This is what the manual means when it says, "Aliasing is performed when scripts are read." In my opinion, this is a bug. The shell may make early syntax checks -- as it does for esac but not $(esac) -- but it shouldn't make early alias expansions. Among other shells, at least bash, csh, and zsh don't make early alias expansions.

[kernigh]

My next example shows that functions don't have the same problem as aliases in ksh. The example runs in ksh, bash, dash, and zsh.

[ -n "$BASH_VERSION" ] && shopt -s expand_aliases
alias call='echo OLD'
# call() { echo OLD "$@"; }
call 11
. ./part.sh

alias call='echo NEW'
# call() { echo NEW "$@"; }
call 22
echo $(call 33)

The output from bash begin.sh or dash or zsh is:

OLD 11
NEW 22
NEW 33

The output from ksh is:

OLD 11
OLD 22
NEW 33

The OLD 22 is wrong. If I change call from an alias to a function (comment the alias ... lines, uncomment call() ...), then ksh outputs the same NEW 22 as bash, dash, zsh. The early alias expansion by ksh might be a bad optimization. Functions don't have the same bad optimization. I argue that aliases don't need more optimization than functions.

[kernigh]

This looks broken to me:

$ printf 'alias a="print ok"; a' | nksh
/home/kernigh/park/ast/build/src/cmd/ksh93/ksh: line 1: a: not found
$ printf 'alias a="print ok"\na' | nksh 
ok

[kernigh]

I wrote:

Among other shells, at least bash, csh, and zsh don't make early alias expansions.

I was wrong; it turns out that bash, dash, yash, and zsh make early alias expansions, just like ksh. These shells expand aliases whey they parse a command, not when they execute the command. This is why a isn't found in

$ dash -c 'alias a="echo ok"; a'
dash: 1: a: not found

When sourcing a file, ksh parses the whole file before executing it, but dash and yash don't. (The ksh(1) manual says for . name, "Otherwise if name refers to a file, the file is read in its entirety and the commands are executed in the current shell environment.") In the next example, other shells find the alias, but ksh doesn't.

$ cat exam
alias a="echo ok"
a
$ nksh -c '. ./exam'
/home/kernigh/park/ast/build/src/cmd/ksh93/ksh: .: line 2: a: not found
$ dash -c '. ./exam'
ok
$ yash -c '. ./exam'
ok
$ zsh -c '. ./exam'  
ok
$ bash -c 'shopt -s expand_aliases && . ./exam'
ok

[saper]

The Korn Shell book says explictly on page 152 (2nd Edition):

ksh reads a complete dot script and splits it into tokens before it executes any of the commands in the file. The dot script is the file specified as the first argument to the . (dot) command. Therefore, if you use aliases or set -k in the dot script, they do not affect the commands in the dot script. They affect only the reading of subsequent commands.

Few paragraphs above there is an explanations how alias definitions do not work within a compound command, including a function definition with an example:

for  i in 0 1
do   if   ((i==0))
     then alias print=date
     fi
     print +%H:%M:%S
done
print +%H:%M:%S

The output is

+%H:%M:%S
+%H:%M:%S
18:54:26

There is also an example how alias foo=bar; foo will not expand foo to bar. I think understanding the concept of compound command is crucial and ksh is pretty consistent here.

[krader1961]

Note that the explanation by @saper doesn't appear to be entirely true in as much as defining and using an alias in a file sourced from ~/.kshrc does work. Put the following in /tmp/s and add source /tmp/s to the bottom of your ~/.kshrc:

alias wtf2='echo alias wtf2'
wtf2

Now start a new ksh process and you'll see that the wtf2 is in fact expanded. Now remove the source /tmp/s from your ~/.kshrc, start a ksh process, and type source /tmp/s. The alias isn't recognized until the source completes. So what makes using source inside ~/.kshrc behave differently and as someone would naively expect?

[krader1961]

@kernigh or @saper I don't understand why the example involving ~/.kshrc in my previous comment works as I would expect while putting the alias in an explicitly sourced file does not. If the former works it seems to me the latter should also work since both are "source"d.

[saper]

I don't have a specification for source in the book. I think it was added later. It might be the same as the dot command. In this case it behaves the same as way as the dot command.

ENV file is a different feature. It is not equivalent to the dot command.

I won't quote the documentation again (pages 88 and 199) but the description does not contain the information that the file pointed to by ENV is parsed at once and does not contain caveat I have referred to above. So those are two different features.

[krader1961]

ENV file is a different feature. It is not equivalent to the dot command.

Yes, but therein lies the problem. I'd be willing to bet that if you polled 1000 people who have used ksh, or a closely related shell like bash, that 998 of them would model the ENV file behavior as equivalent to source'ing that file in a running shell. Too, internally both cases are implemented by the same function that reads a file in the current context; that is, does the equivalent of source. What I don't know at this time is why the logic handles the ENV file case differently. I can't see any reason for the difference other than someone implemented source differently and the AST team was subsequently afraid of correcting the behavior so it was consistent with the ENV file case.

Also, see #1408 where . and source are not defined the same. Again, 99.8 percent of ksh users will undoubtedly tell you they expect them to behave the same.

[saper]

I think this is just different expectations.

I study ksh as an exercise to learn shell programming better. And also a bit about learning about how things have been developed in the past.

Bash has no specification, therefore it is all about expectations. What I like about ksh is that it is defined as a language (speaking of ksh93 plain).

I only wish we've had a better definition of what ksh93u or ksh93v are.

I do not agree on the consistency argument though. To some people dot command and ENV do similar things. But other people may consider dot command execution to be an early form of what became later known as functions. (I think the behaviour here changed between ksh88 and ksh93 in an interesting way). ENV file is not meant to be a function and we want its side-effects to last in a session.