difficult to set `--cloud-ssh-private`

[casperdcl]

trafficstars

As per https://github.com/iterative/cml/issues/848#issuecomment-996761575, there's an issue with --cloud-ssh-private=--....

• Caused by yargs/yargs-parser#433

[DavidGOrtega]

I use it cloud-ssh-private and works. A better issue title might "--cloud-ssh-private is difficult to be set"?

[dacbd]

What are your thoughts on something to populate an authorized key for you? --public-key= akin to adding something like:

echo "ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBDYd3ssa6L15jQC5bckJ2viWlA1tBygxeWoy3s0S14ZHMxUMfp7u2yqficpHO5b+pjgg7Lz+80Ibw157waTZPM+xbF2/KGqS7aYV0L/R8VbWjVEpzxZEeoxSCwFA1tHWUg==" >> /home/ubuntu/.ssh/authorized_keys

in a startup script? I feel like it might be creeping on the feature too much unless people don't care about a ton of cli args.

Maybe document common debugging options/steps somewhere like the above, TF_LOG env, journalctl -u cml.service etc

[casperdcl]

document common debugging options/steps somewhere

:100: - feel free to create a new page on the wiki

[0x2b3bfa0]

Doesn't the SSH documentation snippet work as expected? https://cml.dev/doc/ref/runner#using---cloud-ssh-private

[0x2b3bfa0]

Publishing some old snippets I wrote months ago:

Terraform with cml runner

The following code snippets produce a full trace-level log of the Terraform provider, useful to diagnose a lot of hard to reproduce bugs related to cml-runner --cloud and cloud instances.

GitLab — .gitlab-ci.yml

debug:
  when: always
  image: iterativeai/cml
  variables:
    TF_LOG: trace
    TF_LOG_PATH: /tmp/terraform.log
  script:
    - cml-runner
      --cloud=aws
      --cloud-region=us-west-1
      --cloud-type=t2.micro
      || true
    - cat "$TF_LOG_PATH"

GitHub — .github/workflows/debug.yml

on: push
env:
  TF_LOG: trace
  TF_LOG_PATH: /tmp/terraform.log
jobs:
  debug:
    runs-on: ubuntu-latest
    steps:
      - uses: iterative/setup-cml@v1
      - run: >-
          cml-runner
          --cloud=aws
          --cloud-region=us-west-1
          --cloud-type=t2.micro
          || true
      - run: cat "$TF_LOG_PATH"

Debugging GitLab CI/CD with tmate

debug:
  when: always
  script:
    - mkdir -p ~/.ssh && printf 'y\n\n' | ssh-keygen -q -t rsa -N '' -f ~/.ssh/id_rsa    
    - apt update && apt install --yes tmate expect    
    - TERM=xterm unbuffer ./tmate -FS /tmp/tmate.sock | cat

Using cml-runner with --cloud-ssh-private

cml runner ··· --cloud-ssh-private="$(cat ~/.ssh/id_rsa)"

You can get the instance address by setting the TF_LOG and TF_LOG_PATH environment variables and searching for instance address in the logs.

Debugging cml-runner --cloud=aws

on: push
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-cml@v1
      - run: >-
          cml-runner
          --labels=test
          --cloud=aws
          --cloud-region=eu-west
          --cloud-type=g4dn.xlarge
          --cloud-spot
        env:
          REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run:
    needs: deploy
    runs-on:
      - self-hosted
      - test
    steps:
      - run: |
          set -x
          cat /var/log/cloud-init.log || true
          cat /var/log/cloud-init-output.log || true
          journalctl -u cml || true
          nvidia-smi || true

[0x2b3bfa0]

What are your thoughts on something to populate an authorized key for you?

We need to have also a private key so the provider can SSH to the instance and wait for runners to be ready. Still, we could add a separate configuration option to provision additional authorized keys from the startup script. 🙈 🙉 🙊

[dacbd]

Doesn't the SSH documentation snippet work as expected? https://cml.dev/doc/ref/runner#using---cloud-ssh-private

I think this is was tried, but unsuccessfully cml runner --cloud=... --cloud-ssh-private="$(cat key.pem)" see their image/last comment in https://github.com/iterative/cml/issues/848#issuecomment-996761575

What are your thoughts on something to populate an authorized key for you?

We need to have also a private key so the provider can SSH to the instance and wait for runners to be ready. Still, we could add a separate configuration option to provision additional authorized keys from the startup script. 🙈 🙉 🙊

I generally have been letting the tool do all of its own key management, my ssh keys are on a hardware token so I couldn't give the private key even if I wanted to, but having a quick way to get your public key or set of keys into the .ssh/authorized_keys could be a nice thing for users.

I am hesitant about adding more to an ever-growing list of -- args

[DavidGOrtega]

@0x2b3bfa0 was not this issue fixed?

[0x2b3bfa0]

@DavidGOrtega, not before merging https://github.com/yargs/yargs-parser/pull/434.

[0x2b3bfa0]

Workaround

Use --cloud-ssh-private="$(cat ~/.ssh/id_rsa) " with an extra space character before the closing quotation mark.

References

• https://github.com/yargs/yargs-parser/issues/433
• https://github.com/yargs/yargs-parser/pull/434

[0x2b3bfa0]

Update: https://github.com/yargs/yargs-parser/pull/434 merged yesterday, pending https://github.com/yargs/yargs-parser/pull/446

[SamKnightGit]

yargs-parser 21.1.0 just released!