packit-service icon indicating copy to clipboard operation
packit-service copied to clipboard

Investigate and fix image build failures

Open lbarcziova opened this issue 5 months ago • 3 comments

Our recent automatic image builds are frequently failing due to network errors, requiring multiple manual retries. This process takes too much time to get a usable build, which is unacceptable when we need to deploy an urgent fix.

Example: https://github.com/packit/packit-service/actions/runs/16138635756/job/45547324065

  • [ ] Investigate the root cause of these transient network errors during the build process.
  • [ ] Implement a fix or a reliable workaround to ensure stability, consider alternative solutions to what we use now

lbarcziova avatar Jul 08 '25 12:07 lbarcziova

The errors seem to usually occur in the ansible dnf module that produces a traceback like this:

Traceback (most recent call last):
  File "/src/files/~packit/.ansible/tmp/ansible-tmp-1751970564.7063925-154-159875683464028/AnsiballZ_dnf5.py", line 107, in <module>
    _ansiballz_main()
    ~~~~~~~~~~~~~~~^^
  File "/src/files/~packit/.ansible/tmp/ansible-tmp-1751970564.7063925-154-159875683464028/AnsiballZ_dnf5.py", line 99, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/src/files/~packit/.ansible/tmp/ansible-tmp-1751970564.7063925-154-159875683464028/AnsiballZ_dnf5.py", line 47, in invoke_module
    runpy.run_module(mod_name='ansible.modules.dnf5', init_globals=dict(_module_fqn='ansible.modules.dnf5', _modlib_path=modlib_path),
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                     run_name='__main__', alter_sys=True)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 226, in run_module
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/tmp/ansible_ansible.legacy.dnf5_payload_0aw0s5cy/ansible_ansible.legacy.dnf5_payload.zip/ansible/modules/dnf5.py", line 777, in <module>
  File "/tmp/ansible_ansible.legacy.dnf5_payload_0aw0s5cy/ansible_ansible.legacy.dnf5_payload.zip/ansible/modules/dnf5.py", line 773, in main
  File "/tmp/ansible_ansible.legacy.dnf5_payload_0aw0s5cy/ansible_ansible.legacy.dnf5_payload.zip/ansible/modules/dnf5.py", line 735, in run
  File "/usr/lib64/python3.13/site-packages/libdnf5/base.py", line 867, in download
    return _base.Transaction_download(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
libdnf5.exception.RepoPackageDownloadErrorNested: Failed to download packages

That doesn't give any details about the reason of the failure, we should try to figure out if we can enable some debug logging.

nforro avatar Jul 08 '25 12:07 nforro

Try to use a simple dirty solution here. Create followup cards for a proper longterm fix.

TomasTomecek avatar Jul 10 '25 11:07 TomasTomecek

I wasn't able to replicate this behaviour. Putting it back to the backlog.

majamassarini avatar Jul 21 '25 12:07 majamassarini