ast icon indicating copy to clipboard operation
ast copied to clipboard

Published 20 hours ago • verified publisher icon att

ksh dumps function body to stdout and freezes when eval'ing a function definition

Open McDutchie opened this issue 5 years ago • 4 comments

Description of problem: Under certain conditions, when eval'ing a function definition, ksh (both beta and current development) erroneously dumps the function body to standard output and becomes prone to freezing. These conditions include file descriptor 3 (and no other) being exec'ed, and lots of function definitions having been previously eval'ed.

Ksh version: Version ABIJM 93v- 2014-12-24 Version A 2017.0.0-devel-2343-g09033f5

Steps to reproduce: Run the crazy test script below, and/or:

  1. git clone -b 0.14 https://github.com/modernish/modernish
  2. cd modernish
  3. /path/to/ksh bin/modernish --test

Running the modernish regression test suite gives you lots of dumped function definitions to standard output and then a freeze.

Test script output (see crazy test script below)

$ /path/to/ksh test.sh
 { : This function body is dumped to standard output; }$

Expected output

$ /path/to/ksh test.sh
$

Additional info: The test script below is crazy, and I apologise for that, but after about a year of intermittently trying to track down a specific trigger, it's the best I've been able to come up with. I came up with this by systematically deleting everything from modernish until it stopped triggering the bug. The test script below is at the point where if you remove even one line, or even change indentation, the bug is no longer triggered. Obviously all the code before # ------- MAIN ------- now does not make sense, as it was stripped down to the minimum needed to trigger the bug, so don't bother trying to understand it (look at the modernish code repo instead if you're curious).

#! /bin/ksh
eval '
function _Msh_testFn {
	_Msh_test2=${_Msh_test}
}
function _Msh_testFn2 {
	typeset _Msh_test=local || return
	_Msh_testFn
}'
	_Msh_tSH_testBI='case $CCn${_Msh_biCache}$CCn in
			( *"$CCn${1#--bi=}$CCn"* ) ;;
			( *"/${1#--bi=}$CCn"* )
				# Found builtin with path name. Isolate the "directory" and check $PATH.
				_Msh_tSH_D=${_Msh_biCache}${CCn}
				_Msh_tSH_D=${_Msh_tSH_D%%/${1#--bi=}${CCn}*}
				_Msh_tSH_D=${_Msh_tSH_D##*${CCn}}
				case :$PATH: in
				( *":${_Msh_tSH_D}:"* | *":${_Msh_tSH_D}/:"* )
					unset -v _Msh_tSH_D ;;
				( * )	unset -v _Msh_tSH_D; return 1 ;;
				esac ;;
			( * )	return 1 ;;
			esac'
	_Msh_tSH_testKW='case "${_Msh_kwCache} " in
			( *" ${1#--[rk]w=} "* ) ;;
			( *" !${1#--[rk]w=} "* ) return 1 ;;
			( * )	case $(command -V "${1#--[rk]w=}" 2>/dev/null) in
				( *"${_Msh_kwOutput}" )
					_Msh_kwCache="${_Msh_kwCache} ${1#--[rk]w=}" ;;
				( * )	_Msh_kwCache="${_Msh_kwCache} !${1#--[rk]w=}"
					return 1 ;;
				esac
			esac'
eval 'thisshellhas() {
	case ${#},${-} in
	( 0,* )	_Msh_dieArgs thisshellhas "$#" "at least 1" || return ;;
	( *a* )	set +a; thisshellhas "$@"; eval "set -a; return $?" ;;
	esac
	while :; do
		case $1 in
		( --cache )
			_Msh_cacheCap
			;;
		( --show )
			_Msh_cacheCap --show
			;;
		( "" | --bi= | --[rk]w= | --bi=*/* | --[rk]w=*/* \
		| --bi=*[!\[\]\!{}"$SHELLSAFECHARS"]* \
		| --[rk]w=*[!\[\]\!{}"$SHELLSAFECHARS"]* \
		| --sig=*[!"$SHELLSAFECHARS"]* )
			return 2  # invalid identifier
			;;
		( --bi=* )
			'"${_Msh_tSH_testBI}"'
			;;
		( --[rk]w=* )
			'"${_Msh_tSH_testKW}"'
			;;
		( --sig=* )
			use -q var/stack/trap || die "thisshellhas: --sig: requires var/stack/trap" || return
			if _Msh_arg2sig "${1#--sig=}" -nv; then
				REPLY=${_Msh_sig}
				unset -v _Msh_sig
			else
				unset -v _Msh_sig REPLY
				return 1
			fi ;;
		( --* )	die "thisshellhas: invalid option: ${1%%=*}" || return
			;;
		( -o )	case ${2-} in
			( allexport | errexit | ignoreeof | noclobber \
			| noglob | noexec | nounset | verbose | xtrace )
				;;
			( "" | *[!"$ASCIIALNUM"_-]* | -* )
				return 2 ;;
			( * )	case " ${_Msh_optCache} " in
				( *" $2 "* )	;;
				( *" !$2 "* )	return 1 ;;
				( * )	if (set +o "$2") 2>/dev/null; then
						_Msh_optCache=${_Msh_optCache:+${_Msh_optCache} }$2
					else
						_Msh_optCache=${_Msh_optCache:+${_Msh_optCache} }!$2
						return 1
					fi ;;
				esac ;;
			esac
			shift ;;
		( -[aCefmnuvx] )
			;;
		( -["$ASCIIALNUM"] )
			case " ${_Msh_optCache} " in
			( *" $1 "* )	;;
			( *" !$1 "* )	return 1 ;;
			( * )	if isset "$1" || (set "+${1#-}") 2>/dev/null; then
					_Msh_optCache=${_Msh_optCache:+${_Msh_optCache} }$1
				else
					_Msh_optCache=${_Msh_optCache:+${_Msh_optCache} }!$1
					return 1
				fi ;;
			esac ;;
		( -* )	return 2 ;;
		( *[!ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_]* )
			thisshellhas "--bi=$1" || thisshellhas "--rw=$1" || return
			;;
		( * )	'"${_Msh_tSH_bashLEPIPEMAIN}"'
			case " ${_Msh_cap} " in
			( *" $1 "* )	;;
			( *" !$1 "* | *" #ALLCACHED "* ) return 1 ;;
			( * )	_Msh_doCapTest "$1"
				case $? in
				( 0 )	_Msh_cap=${_Msh_cap:+${_Msh_cap} }$1
					unset -v _Msh_test ;;
				( 1 | 127 )	# 127 = *.t file not found
					_Msh_cap=${_Msh_cap:+${_Msh_cap} }!$1
					unset -v _Msh_test
					return 1 ;;
				( * )	die "thisshellhas(): failure while testing for $1" || return ;;
				esac ;;
			esac
			;;
		esac
		shift
		case $# in
		( 0 )	break ;;
		esac
	done
}'
_Msh_inSbSh_generic_part1='_Msh_inSbSh_P=$(exec "$MSH_SHELL" -c "set -o nounset && echo \$PPID" 2>/dev/null) \
	&& case ${_Msh_inSbSh_P} in ("" | *[!0123456789]*) ! : ;; esac \
	|| _Msh_inSbSh_P=$(PATH=$DEFPATH exec /bin/sh -c '\''ppid=`ps -o ppid= -p $$` && echo $ppid'\'')
	case ${1-},${_Msh_inSbSh_P} in
	( *, | *,*[!0123456789]* )
		putln "${ME##*/}: insubshell: internal error: cannot determine parent PID" 1>&2
		isset _Msh_die_isrunning && return 0 || die ;;'
_Msh_inSbSh_generic_part2='
	(,$$)	unset -v _Msh_inSbSh_P; return 1 ;;
	(*,$$)	REPLY=$$; unset -v _Msh_inSbSh_P; return 1 ;;
	(,*)	unset -v _Msh_inSbSh_P; return 0 ;;
	(*,*)	REPLY=${_Msh_inSbSh_P}; unset -v _Msh_inSbSh_P; return 0 ;;
	esac'
_Msh_inSbSh_method='
	if ((.sh.subshell > 0)); then
		command ulimit -t unlimited 2>/dev/null
		(($# == 0)) && return 0
	fi
	'"${_Msh_inSbSh_generic_part1}${_Msh_inSbSh_generic_part2}"
eval 'insubshell() {
	case ${#},${1-} in
	( 1,-p | 1,-u )  ;;
	( [!0]* ) die "insubshell: invalid arguments: $@" || return ;;
	esac
	'"${_Msh_inSbSh_method}"'
}'
_Msh_isset='[[ -v $1 ]]'
_Msh_isset_v='[[ -v $2 ]]'
_Msh_isset_x='unset -v _Msh_issetEx_WasRun _Msh_issetEx_FoundIt
		export "_Msh_issetExV=$2"	# guarantee one exported variable to check if this method works
		alias export=_Msh_issetExHandleExport
		eval "$(command export -p)"
		unalias export
		unset -v _Msh_issetExV
		isset _Msh_issetEx_WasRun && unset -v _Msh_issetEx_WasRun ||
			die "isset -x: internal error: '\''export -p'\'' not parseable or '\''export'\'' not aliasable" || return
		isset _Msh_issetEx_FoundIt && unset -v _Msh_issetEx_FoundIt'
	_Msh_isset_r='! ( command unset -v "$2" || \exit 1 ) 2>/dev/null'
	_Msh_isset_f='command typeset -f "$2" >/dev/null 2>&1'
eval 'isset() {
	case ${#},${1-},${2-} in
	( 1,-o, )  die "isset -o: long-form option name expected" ;;
	( 1,-["$ASCIIALNUM"], )
		   case $- in ( *"${1#-}"* ) ;; ( * ) return 1 ;; esac ;;
	( 1,, | 1,[0123456789]* | 1,*[!"$ASCIIALNUM"_]*, \
	| 2,-[vxrf], | 2,-[vxrf],[0123456789]* | 2,-[vxrf],*[!"$ASCIIALNUM"_]* \
	| 2,-o, | 2,-o,*[!"$ASCIIALNUM"_-]* )
		   return 2 ;;  # invalid identifier
	( 1,* )    '"${_Msh_isset}"' ;;
	( 2,-v,* ) '"${_Msh_isset_v}"' ;;
	( 2,-x,* ) '"${_Msh_isset_x}"' ;;
	( 2,-r,* ) '"${_Msh_isset_r}"' ;;
	( 2,-f,* ) '"${_Msh_isset_f}"' ;;
	( 2,-* )   die "isset: invalid option: $2" ;;
	( * )	   die "isset: invalid arguments" ;;
	esac
}'
	eval '_Msh_qV_dblQuote() {
		_Msh_qV=${_Msh_qV_VAL//\\/\\\\}
		_Msh_qV=${_Msh_qV//\$/\\\$}
		_Msh_qV=${_Msh_qV//\`/\\\`}
		case ${_Msh_qV_VAL} in
		( *[$CONTROLCHARS]* )
			_Msh_qV=${_Msh_qV//$CC01/'\''${CC01}'\''}
			_Msh_qV=${_Msh_qV//$CC02/'\''${CC02}'\''}
			_Msh_qV=${_Msh_qV//$CC03/'\''${CC03}'\''}
			_Msh_qV=${_Msh_qV//$CC04/'\''${CC04}'\''}
			_Msh_qV=${_Msh_qV//$CC05/'\''${CC05}'\''}
			_Msh_qV=${_Msh_qV//$CC06/'\''${CC06}'\''}
			_Msh_qV=${_Msh_qV//$CC07/'\''${CCa}'\''}
			_Msh_qV=${_Msh_qV//$CC08/'\''${CCb}'\''}
			_Msh_qV=${_Msh_qV//$CC09/'\''${CCt}'\''}
			_Msh_qV=${_Msh_qV//$CC0A/'\''${CCn}'\''}
			_Msh_qV=${_Msh_qV//$CC0B/'\''${CCv}'\''}
			_Msh_qV=${_Msh_qV//$CC0C/'\''${CCf}'\''}
			_Msh_qV=${_Msh_qV//$CC0D/'\''${CCr}'\''}
			_Msh_qV=${_Msh_qV//$CC0E/'\''${CC0E}'\''}
			_Msh_qV=${_Msh_qV//$CC0F/'\''${CC0F}'\''}
			_Msh_qV=${_Msh_qV//$CC10/'\''${CC10}'\''}
			_Msh_qV=${_Msh_qV//$CC11/'\''${CC11}'\''}
			_Msh_qV=${_Msh_qV//$CC12/'\''${CC12}'\''}
			_Msh_qV=${_Msh_qV//$CC13/'\''${CC13}'\''}
			_Msh_qV=${_Msh_qV//$CC14/'\''${CC14}'\''}
			_Msh_qV=${_Msh_qV//$CC15/'\''${CC15}'\''}
			_Msh_qV=${_Msh_qV//$CC16/'\''${CC16}'\''}
			_Msh_qV=${_Msh_qV//$CC17/'\''${CC17}'\''}
			_Msh_qV=${_Msh_qV//$CC18/'\''${CC18}'\''}
			_Msh_qV=${_Msh_qV//$CC19/'\''${CC19}'\''}
			_Msh_qV=${_Msh_qV//$CC1A/'\''${CC1A}'\''}
			_Msh_qV=${_Msh_qV//$CC1B/'\''${CCe}'\''}
			_Msh_qV=${_Msh_qV//$CC1C/'\''${CC1C}'\''}
			_Msh_qV=${_Msh_qV//$CC1D/'\''${CC1D}'\''}
			_Msh_qV=${_Msh_qV//$CC1E/'\''${CC1E}'\''}
			_Msh_qV=${_Msh_qV//$CC1F/'\''${CC1F}'\''}
			_Msh_qV=${_Msh_qV//$CC7F/'\''${CC7F}'\''} ;;
		esac
		_Msh_qV_VAL=\"${_Msh_qV//\"/\\\"}\"
	}'
eval 'str() {
	case ${#},${1-} in
	( 1,empty | 1,eq )
			;;
	( 1,isint | 1,isvarname | 1,ne )
			return 1 ;;
	( 2,eq | 2,in | 2,begin | 2,end | 2,match | 2,ematch )
			return 1 ;;
	( 2,ne )	;;
	( 2,empty )	case ${2:+n} in ( n ) return 1 ;; esac ;;
	( 2,isint )	case ${2#"${2%%[!" $CCt$CCn"]*}"} in
			( 0[xX]*[!0123456789abcdefABCDEF]* | [+-]0[xX]*[!0123456789abcdefABCDEF]* )
				return 1 ;;
			( 0[xX]?* | [+-]0[xX]?* )
				;;
			( "" | [+-] | ?*[+-]* | *[!0123456789+-]* | 0*[!01234567]* | [+-]0*[!01234567]* )
				return 1 ;;
			esac ;;
	( 2,isvarname )	case $2 in
			( [0123456789]* | *[!"$ASCIIALNUM"_]* )
				return 1 ;;
			esac ;;
	( 3,eq )	case $2 in (  "$3"  ) ;; ( * ) return 1 ;; esac ;;
	( 3,ne )	case $2 in (  "$3"  ) return 1 ;; ( * ) ;; esac ;;
	( 3,in )	case $2 in ( *"$3"* ) ;; ( * ) return 1 ;; esac ;;
	( 3,begin )	case $2 in (  "$3"* ) ;; ( * ) return 1 ;; esac ;;
	( 3,end )	case $2 in ( *"$3"  ) ;; ( * ) return 1 ;; esac ;;
	( 3,match )	'"${_Msh_doMatch}"' ;;
	( 3,ematch )	'"${_Msh_doEMatch}"' ;;
	( 3,lt )	'"${_Msh_doSortsBefore}"' ;;
	( 3,gt )	'"${_Msh_doSortsAfter}"' ;;
	( 3,le )	str lt "$2" "$3" || str eq "$2" "$3" ;;
	( 3,ge )	str gt "$2" "$3" || str eq "$2" "$3" ;;
	( *,empty | *,isint | *,isvarname )
			_Msh_dieArgs "str $1" "$((${#}-1))" "max. 1" ;;
	( *,eq | *,ne )	_Msh_dieArgs "str $1" "$((${#}-1))" "max. 2" ;;
	( *,in | *,begin | *,end | *,match | *,ematch )
			_Msh_dieArgs "str $1" "$((${#}-1))" "1 or 2" ;;
	( *,lt | *,gt | *,le | *,ne )
			_Msh_dieArgs "str $1" "$((${#}-1))" "2" ;;
	( 0, )		die "str: operator expected" ;;
	( * )		die "str: invalid operator: $1" ;;
	esac
}'

# --------------------
# ------- MAIN -------
# --------------------

exec 3>&1
eval "fn() { : This function body is dumped to standard output; }"

McDutchie avatar Feb 09 '19 16:02 McDutchie

There are intermittent test failures that are similar to the description above. It is likely to be yet another instance of using heap memory after it has been freed or something similar.

@McDutchie I take it you can't reproduce this problem with ksh93u+?

krader1961 avatar Feb 10 '19 02:02 krader1961

Op 10-02-19 om 02:24 schreef Kurtis Rader:

@McDutchie I take it you can't reproduce this problem with ksh93u+? Correct, it's only with the beta and the development code.

McDutchie avatar Feb 10 '19 10:02 McDutchie

Perhaps this is the code to reproduce this Issue. The problem occurs when the total size of the function defined by eval exceeds 8KB. This is reproduced even when two 4KB functions are defined. Confirmed with ksh 2020 on Ubuntu 20.04.

#!/bin/sh

str=''
for i in $(seq 1024); do
  str="${str}12345678"
done
echo "${#str}"
eval "foo() { $str; }"

ls -al /proc/$(exec sh -c 'echo "$PPID"')/fd/ > /dev/tty

baz() { eval "bar() { This string will be output to STDOUT; }"; }
( baz >&3 ) 3>&1

output

8192
total 0
dr-x------ 2 user user  0 Jan 17 10:00 .
dr-xr-xr-x 9 user user  0 Jan 17 10:00 ..
lrwx------ 1 user user 64 Jan 17 10:00 0 -> /dev/pts/0
lrwx------ 1 user user 64 Jan 17 10:00 1 -> /dev/pts/0
lr-x------ 1 user user 64 Jan 17 10:00 10 -> /tmp
lr-x------ 1 user user 64 Jan 17 10:00 11 -> /tmp/script.sh
lrwx------ 1 user user 64 Jan 17 10:00 2 -> /dev/pts/0
lrwx------ 1 user user 64 Jan 17 10:00 3 -> '/tmp/sf.XXa7Xyug (deleted)'   <=== Something's wrong.
 { This string will be output to STDOUT; }

ko1nksm avatar Jan 17 '21 10:01 ko1nksm

Workaround

#!/bin/sh

if [ "${KSH_VERSION:-}" ]; then
  ( exec 3>/dev/null 4>&3 5>&3 6>&3 7>&3 8>&3 9>&3
    eval "dummy() { $(printf '%8192s' ':'); }"
  ) 3>&- 4>&- 5>&- 6>&- 7>&- 8>&- 9>&-
fi

str=''
for i in $(seq 1024); do
  str="${str}12345678"
done
echo "${#str}"
eval "foo() { $str; }"

ls -al /proc/$(exec sh -c 'echo "$PPID"')/fd/ > /dev/tty

baz() { eval "bar() { This string will be output to STDOUT; }"; }
( baz >&3 ) 3>&1

output

8192
total 0
dr-x------ 2 user user  0 Jan 17 14:18 .
dr-xr-xr-x 9 user user  0 Jan 17 14:18 ..
lrwx------ 1 user user 64 Jan 17 14:18 0 -> /dev/pts/0
lrwx------ 1 user user 64 Jan 17 14:18 1 -> /dev/pts/0
lr-x------ 1 user user 64 Jan 17 14:18 10 -> /tmp
lr-x------ 1 user user 64 Jan 17 14:18 11 -> /tmp/script.sh
lrwx------ 1 user user 64 Jan 17 14:18 2 -> /dev/pts/0
lrwx------ 1 user user 64 Jan 17 14:18 20 -> '/tmp/sf.XXaLaLlg (deleted)'

ko1nksm avatar Jan 17 '21 14:01 ko1nksm