CentOS Stream propose-downstream: Unable to push

[sentry[bot]]

Sentry Issue: PCKT-002-PACKIT-SERVICE-78F

GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git push --porcelain --force --no-verify -- fork 1.15-c10s-update-propose_downstream
  stderr: 'fatal: Could not read from remote repository.'
  File "packit/distgit.py", line 297, in push_to_fork
    self.push(refspec=branch_name, remote_name=fork_remote_name, force=force)
  File "packit/base_git.py", line 589, in push
    push_infos_list: Iterable[PushInfo] = self.local_project.git_repo.remote(
  File "git/remote.py", line 1117, in push
    return self._get_push_info(proc, progress, kill_after_timeout=kill_after_timeout)
  File "git/remote.py", line 926, in _get_push_info
    proc.wait(stderr=stderr_text)
  File "git/cmd.py", line 600, in wait
    raise GitCommandError(remove_password_if_present(self.args), status, errstr)

PackitException: Unable to push to remote fork 'fork' using branch '1.15-c10s-update-propose_downstream', the error is:
Cmd('git') failed due to: exit code(128)
  cmdline: git push --porcelain --force --no-verify -- fork 1.15-c10s-update-propose_downstream
  stderr: 'fatal: Could not read from remote repository.'
  File "packit_service/worker/handlers/distgit.py", line 365, in run_for_target
    downstream_pr = self.sync_branch(
  File "packit_service/worker/handlers/distgit.py", line 244, in sync_branch
    downstream_pr = self.packit_api.sync_release(
  File "packit/api.py", line 1137, in sync_release
    pr = self.push_and_create_pr(
  File "packit/api.py", line 1471, in push_and_create_pr
    repo.push_to_fork(repo.local_project.ref, force=True, sync_acls=sync_acls)
  File "packit/distgit.py", line 303, in push_to_fork
    raise PackitException(msg) from ex

[lbarcziova]

we have discussed this a little bit with @mfocko and it looks like an issue with credentials - SSH keys.

[mfocko]

Logs indicate that it's an issue with the authentication against GitLab

• [x] Packit users do have SSH keys uploaded to the GitLab
• [x] There is no entry in the $HOME/.ssh/config though which is probably causing this issue
• [x] Changes have been deployed and affected pods rescaled

[lbarcziova]

This occured again for umockev, see https://github.com/martinpitt/umockdev/issues/241.

[lbarcziova]

it looks like both staging and prod contained outdated ssh-config before https://github.com/packit/deployment/commit/780ab9729074af82278395764541120203d4452c, I have just updated it. Will close this once it is verified to work.

[lbarcziova]

The problem still persists even with the mentioned change being deployed, see https://dashboard.packit.dev/results/propose-downstream/8787.