doskey macros broken for inputs with >1 consecutive whitespace

[lhecker]

trafficstars

Steps to reproduce

Run this:

doskey foo=echo start $3 $2 $1 end
foo a b c
foo a  b c

Expected Behavior

start c b a end
start c b a end

Actual Behavior

start c b a end
start b  a end

The issue is caused by this function: https://github.com/microsoft/terminal/blob/42e9ddcc78710c101315a58b87c1c9db893fe3a7/src/host/alias.cpp#L833-L861

For an input like "a b c" it'll return

"a"
""
"b"
"c"

The issue can either be fixed by fixing s_Tokenize itself or by reverting the code to the previous conhost v1 code. Either would likely be fine.

[237dmitry]

In conhost the same behavior

[lhecker]

For sure! The bug was introduced in 2018. 🙂

[zadjii-msft]

Note to self: not fixed in #15783