Introduce a conceptual help topic about calling external programs (native applications)

[mklement0]

Related: #2361, https://github.com/PowerShell/PowerShell/issues/13068#issuecomment-653526374, and #6239

Summary of the new document or enhancement

Many special considerations apply when you call an external command-line executable (aka native application / utility), which aren't currently covered comprehensively, in one place:

• That the only data type supported is text ([string]), both on input and output, and how raw byte data is fundamentally unsupported - both when collecting the output in PowerShell and when piping between native programs.

  • Update: v7.4 introduced raw byte support.

• How there are syntax pitfalls due to PowerShell's extended set of metacharacters (compared to other shells) causing potential misinterpretation of arguments, which must be avoided with quoting (e.g., To pass literal @foo, which works unquoted in cmd.exe and bash, you must use '@foo' in PowerShell).

  • How --% can be used (primarily on Windows) to selectively deactivate PowerShell's parsing.

• How "native globbing" is automatically applied to arguments such as *.txt on Unix-like platforms; that is, *.txt is implicitly replaced with the array of file names / paths matching that wildcard pattern.

• How output data is sent through the pipeline line by line, resulting in an array of strings (lines), if collected in a variable.

• How stderr (standard error) output is passed through to the host rather than going through PowerShell's error stream and can only be captured with a 2> redirection.

• How redirections (>) generally do not pass the native program's output through as-is, but invariably treat it as [Console]::OutputEncoding encoded text that on writing to the target file is written with PowerShell's default encoding (BOM-less UTF-8 in PowerShell 6+, UTF-16LE in Windows PowerShell).

• How external-program calls aren't integrated with PowerShell's error handling and require explicit checking of $? / $LASTEXITCODE to detect failure, except in PowerShell 7, where pipeline chain operators && and || can now be used. See also: the RFC that proposes improvements to the integration.

• How &, the call operator, must be used to invoke executables whose paths are / must be quoted (as a whole) and/or contain variable references or subexpressions (this requirement isn't specific to external programs, but most likely to surface in that context).

• How Start-Process is typically not the right tool for invoking external programs - see #6239.

Details of requested document:

• Proposed title: about_Native_Calls
• Propose location in the TOC: Among the `about_* topics
• Target audience: end users
• Purpose or scenario: guidance for invoking native command-line programs
• List of related articles to link to: about_Parsing, about_Quoting_Rules, about_Redirection, about_Pipelines, about_pwsh, about_Operators (section "Pipeline chain operators && and ||")

[yecril71pl]

C:\'Program Files\Acme Inc.\Your Terrific App.exe' is a perfect invocation and it does not use the & operator.

[mklement0]

True: what matters is whether the token starts with a quote, which in turn only works if the whole token is quoted.

It is for this reason that compound string arguments composed of quoted and unquoted tokens are best avoided in PowerShell - the fact that a quoted first token invariably becomes its own argument is counterintuitive and invites confusion:

# OK, but only because the first token is unquoted.
C:\'Program Files'\nodejs\node.exe

# NOT OK: becomes string literal *plus separate argument*, because the first token is *quoted*
'C:\Program Files'\nodejs\node.exe

# ALSO NOT OK, despite &, for the same reason.
& 'C:\Program Files'\nodejs\node.exe

[yecril71pl]

But that makes & recommended, not required as you have suggested.

[mklement0]

It is required for (a) command names / paths that are quoted as a whole and/or (b) contain variable references or subexpressions. I've inserted "(as a whole)" in the initial post to clarify.

[mklement0]

In short:

• You're free to always use &

• You must use & in the scenarios described above.