String interpolation clash causing named capture group fail in regexp_replace()

[philrz]

While verifying the changes for #4296, I spotted what I think may be a bug. Repro is with Zed commit 5b40e33.

I saw in the reviews of #4435 that we're using functionality described at https://pkg.go.dev/regexp#Regexp.Expand. Therefore I tried to use the example shown there in my verification. The following baseline shows it working much as it's shown in the Go docs.

$ zq -version
Version: v1.6.0-9-g5b40e33b

$ cat lines.txt 
# comment line
option1: value1
option2: value2

# another comment line
option3: value3

$ zq -i line lines.txt 
"# comment line"
"option1: value1"
"option2: value2"
""
"# another comment line"
"option3: value3"

$ zq -i line lines.txt | zq -z 'regexp_replace(this,/(?m)(?P<key>\w+):\s+(?P<value>\w+)$/,"$key=$value")' -
"# comment line"
"option1=value1"
"option2=value2"
null(string)
"# another comment line"
"option3=value3"

However, the docs also indicate the syntax ${key} should work, and I find it doesn't.

$ zq -i line lines.txt | zq -z 'regexp_replace(this,/(?m)(?P<key>\w+):\s+(?P<value>\w+)$/,"${key}=${value}")' -
error("regexp_replace: string values required for all args")
error("regexp_replace: string values required for all args")
error("regexp_replace: string values required for all args")
error("regexp_replace: string values required for all args")
error("regexp_replace: string values required for all args")
error("regexp_replace: string values required for all args")

I think it's a clash with Zed's string interpolation. 🙁

I sense it's needed in cases where the variable reference could be ambiguous. For instance, if I wanted the key/value output to be bookended by FOO and BAR and tried to write it without the {}, I don't get the BAR.

$ zq -i line lines.txt | zq -z 'regexp_replace(this,/(?m)(?P<key>\w+):\s+(?P<value>\w+)$/,"FOO$key=$valueBAR")' -
"# comment line"
"FOOoption1="
"FOOoption2="
null(string)
"# another comment line"
"FOOoption3="

As we're fixing this, it seems it would be good to add a test of named capture groups and an example in the docs that shows them.

[philrz]

@nwt pointed out that a good way to resolve the clash would be to have our string interpolation behave more like Python's f-strings.

[philrz]

Verified in Zed commit e40a039.

Now that string interpolation uses F-strings and hence there's no longer a clash, the examples above that use ${} now work as expected.

$ zq -version
Version: v1.15.0-26-ge40a0394

$ zq -i line lines.txt | zq -z 'regexp_replace(this,/(?m)(?P<key>\w+):\s+(?P<value>\w+)$/,"${key}=${value}")' -
"# comment line"
"option1=value1"
"option2=value2"
""
"# another comment line"
"option3=value3"

$ zq -i line lines.txt | zq -z 'regexp_replace(this,/(?m)(?P<key>\w+):\s+(?P<value>\w+)$/,"FOO${key}=${value}BAR")' -
"# comment line"
"FOOoption1=value1BAR"
"FOOoption2=value2BAR"
""
"# another comment line"
"FOOoption3=value3BAR"

Thanks @mattnibs!

[philrz]

Also, I'd mentioned when I opened this issue that it'd be worth adding an example to the regexp_replace() docs that uses curly braces. Now that this issue has been fixed, I've drafted that example in #5130.