Does not support private npm registries with auth

[swells]

Before filling a bug

• [x] have you checked the faq for known issues.
• [x] have you checked existing issues

Describe the bug

On these OneBranch compliant CI/CD pipelines we must use a private internal authenticated npm registry and fails tiring to acquire autorest-core's version during bootstrapping.

Expected behavior

autorest should support this as a typical use-case and use .npmrc on private authenticated npm registries .

Here are the pipeline steps:

network isolation

extends:
  template: v2/OneBranch.Official.CrossPlat.yml@templates
    featureFlags:
      EnableCDPxPAT: false
      WindowsHostVersion:
        Network: KS3

Npm Authenticate via ADO

  - task: npmAuthenticate@0
    displayName: "Authenticate npm"
    inputs:
      workingFile: '$(Build.SourcesDirectory)\src\.npmrc'

.npmrc

registry=https://msdata.pkgs.visualstudio.com/................../npm/registry/
always-auth=true

Set env and run autorest

NOTE: The setting of the env autorest_registry: https://msdata.pkgs.visualstudio.com/................../npm/registry/

 - script: >-
      autorest
         --input-file=.some-spec.json
         --csharp 
         --namespace=SOME-NAMESPACE
         --legacy 
        --debug
    env: # Variables to map into the process's environment.
      autorest_registry: https://msdata.pkgs.visualstudio.com/................../npm/registry/

Failure on getting azure-core version

"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "C:\__w\_temp\548d9984-5f23-47b4-8d71-afa9773675f2.cmd""
AutoRest code generation utility [cli version: 3.7.1; node: v20.11.1]
(C) 2018 Microsoft Corporation.
https://aka.ms/autorest
info    | AutoRest core version selected from configuration: ~2.0.4413.
debug   | [0.97 s] Network Enabled: true
debug   | [1.13 s] No @autorest/core (or @microsoft.azure/autorest-core) is installed.
debug   | [1.13 s] ~2.0.4413 was not satisfied directly by a previous installation.
debug   | [1.39 s] Error trying to resolve @autorest/core version ~2.0.4413: Error: Unable to resolve package '@autorest/core@~2.0.4413'.
Failure:
Error: Unable to find a valid AutoRest core package '@autorest/core' @ '~2.0.4413'.
Error: Unable to find a valid AutoRest core package '@autorest/core' @ '~2.0.4413'.
    at selectVersion (C:\__t\node\20.11.1\x64\node_modules\autorest\dist\src_autorest-as-a-service_ts.js:393:23)
    at resolveCoreVersion (C:\__t\node\20.11.1\x64\node_modules\autorest\dist\src_autorest-as-a-service_ts.js:590:29)
    at main (C:\__t\node\20.11.1\x64\node_modules\autorest\dist\app.js:98:33)

Additional context

I think what is happening is after setting the env autorest_registry autorest'ss use of pacote manifest fetch to get the azure-core version fails since it can not properly authenticate:

export async function fetchPackageMetadata(spec: string): Promise<pacote.ManifestResult> {
  try {
    return await pacote.manifest(spec, {
      cache: `${tmpdir()}/cache`,
      registry: process.env.autorest_registry || "https://registry.npmjs.org",
      "full-metadata": true,
    });
  } catch (error) {
    logger.error(`Error resolving package ${spec}`, error);
    throw new UnresolvedPackageException(spec);
  }
}

There is no way to deal with the _auth scopes from .npmrc and pacote.manifest. This is my speculation anyhow since this is difficult to debug this in CI pipelines.