Intermittent crash in `ci-dnf-stack/dnf-behave-tests/createrepo_c/zchunk.feature`

[kontura]

Zchunk tests in zchunk.feature crash from time to time.

With sanitizers enabled I was able to extract:

  Scenario: create repository with zck and dictionary metadata with bad package                                                                 # zchunk.feature:113
    Given I copy file "{context.scenario.repos_location}/createrepo_c-ci-packages/x86_64/package-0.2.1-1.fc29.x86_64.rpm" to "/"                # ../common/file.py:149 0.000s
    And I create file "/afilethatlookslike.rpm" with                                                                                            # ../common/file.py:36 0.000s
      """
      gibberish
      """
    Given I create directory "/dictionaries"                                                                                                    # ../common/file.py:30 0.000s
    And I create file "/dictionaries/primary.xml.zdict" with                                                                                    # ../common/file.py:36 0.000s
      """
      primary foobar
      """
    And I create file "/dictionaries/filelists.xml.zdict" with                                                                                  # ../common/file.py:36 0.000s
      """
      filelists foobar
      """
    And I create file "/dictionaries/other.xml.zdict" with                                                                                      # ../common/file.py:36 0.000s
      """
      other foobar
      """
    When I execute createrepo_c with args "--zck --zck-dict-dir {context.scenario.default_tmp_dir}/dictionaries --simple-md-filenames ." in "/" # steps/cmd.py:8 0.162s
    Then the exit code is 2                                                                                                                     # ../common/output.py:60 0.000s
      Assertion Failed: Command has returned exit code 1: createrepo_c --zck --zck-dict-dir /tmp/createrepo_c_ci_tempdir_7jn_6ryi/dictionaries --simple-md-filenames .
      Captured stdout:

      Last Command: createrepo_c --zck --zck-dict-dir /tmp/createrepo_c_ci_tempdir_7jn_6ryi/dictionaries --simple-md-filenames .

      Last Command stderr:
      C_CREATEREPOLIB: Warning: read_header: rpmReadPackageFile() error
      C_CREATEREPOLIB: Warning: Cannot read package: ./afilethatlookslike.rpm: rpmReadPackageFile() error
      =================================================================
      ==470375==ERROR: AddressSanitizer: attempting double-free on 0x607000050350 in thread T10:
          #0 0x7f2a562d7fb8 in __interceptor_free.part.0 (/lib64/libasan.so.8+0xd7fb8) (BuildId: a1bdcca9c70162c2a1e8e7e9e8bc816ab51d214d)
          #1 0x7f2a551b938b in zck_clear_error (/lib64/libzck.so.1+0x1038b) (BuildId: 140a784977179cd62494b17776b5eb5997ddcdc3)
          #2 0x7f2a551b692a in zck_create (/lib64/libzck.so.1+0xd92a) (BuildId: 140a784977179cd62494b17776b5eb5997ddcdc3)
          #3 0x7f2a556e4bc0 in cr_sopen /home/amatej/tmp/src/createrepo_c/src/compression_wrapper.c:676
          #4 0x7f2a5571a97c in cr_get_compressed_content_stat /home/amatej/tmp/src/createrepo_c/src/repomd.c:136
          #5 0x7f2a5571bdb6 in cr_repomd_record_fill /home/amatej/tmp/src/createrepo_c/src/repomd.c:277
          #6 0x7f2a5572f567 in cr_repomd_record_fill_thread /home/amatej/tmp/src/createrepo_c/src/threads.c:186
          #7 0x7f2a56147c31 in g_thread_pool_thread_proxy.lto_priv.0 (/lib64/libglib-2.0.so.0+0x90c31) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #8 0x7f2a56145102 in g_thread_proxy (/lib64/libglib-2.0.so.0+0x8e102) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #9 0x7f2a54896a26 in start_thread (/lib64/libc.so.6+0x8ea26) (BuildId: 21dc323755123cea8c34495928e904bc56fa71e3)
          #10 0x7f2a5491dcab in __clone3 (/lib64/libc.so.6+0x115cab) (BuildId: 21dc323755123cea8c34495928e904bc56fa71e3)

      0x607000050350 is located 0 bytes inside of 72-byte region [0x607000050350,0x607000050398)
      freed by thread T11 here:
          #0 0x7f2a562d7fb8 in __interceptor_free.part.0 (/lib64/libasan.so.8+0xd7fb8) (BuildId: a1bdcca9c70162c2a1e8e7e9e8bc816ab51d214d)
          #1 0x7f2a551b938b in zck_clear_error (/lib64/libzck.so.1+0x1038b) (BuildId: 140a784977179cd62494b17776b5eb5997ddcdc3)
          #2 0x7f2a551b692a in zck_create (/lib64/libzck.so.1+0xd92a) (BuildId: 140a784977179cd62494b17776b5eb5997ddcdc3)
          #3 0x7f2a556e4bc0 in cr_sopen /home/amatej/tmp/src/createrepo_c/src/compression_wrapper.c:676

      previously allocated by thread T8 here:
          #0 0x7f2a562d8cc7 in calloc (/lib64/libasan.so.8+0xd8cc7) (BuildId: a1bdcca9c70162c2a1e8e7e9e8bc816ab51d214d)
          #1 0x7f2a551b6dc5 in set_error_wf (/lib64/libzck.so.1+0xddc5) (BuildId: 140a784977179cd62494b17776b5eb5997ddcdc3)
          #2 0x7f2a551b191d in zck_get_chunk_size (/lib64/libzck.so.1+0x891d) (BuildId: 140a784977179cd62494b17776b5eb5997ddcdc3)
          #3 0x7f2a556e8e02 in cr_get_zchunk_with_index /home/amatej/tmp/src/createrepo_c/src/compression_wrapper.c:1758

      Thread T10 created by T6 here:
          #0 0x7f2a56248956 in pthread_create (/lib64/libasan.so.8+0x48956) (BuildId: a1bdcca9c70162c2a1e8e7e9e8bc816ab51d214d)
          #1 0x7f2a561466ef in g_thread_new_internal (/lib64/libglib-2.0.so.0+0x8f6ef) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #2 0x7f2a56147226 in g_thread_pool_spawn_thread (/lib64/libglib-2.0.so.0+0x90226) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #3 0x7f2a56145102 in g_thread_proxy (/lib64/libglib-2.0.so.0+0x8e102) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #4 0x7f2a54896a26 in start_thread (/lib64/libc.so.6+0x8ea26) (BuildId: 21dc323755123cea8c34495928e904bc56fa71e3)

      Thread T6 created by T0 here:
          #0 0x7f2a56248956 in pthread_create (/lib64/libasan.so.8+0x48956) (BuildId: a1bdcca9c70162c2a1e8e7e9e8bc816ab51d214d)
          #1 0x7f2a561466ef in g_thread_new_internal (/lib64/libglib-2.0.so.0+0x8f6ef) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #2 0x7f2a561468ff in g_thread_new (/lib64/libglib-2.0.so.0+0x8f8ff) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #3 0x7f2a56147536 in g_thread_pool_new_full (/lib64/libglib-2.0.so.0+0x90536) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #4 0x412f26 in main /home/amatej/tmp/src/createrepo_c/src/createrepo_c.c:1616

      Thread T11 created by T6 here:
          #0 0x7f2a56248956 in pthread_create (/lib64/libasan.so.8+0x48956) (BuildId: a1bdcca9c70162c2a1e8e7e9e8bc816ab51d214d)
          #1 0x7f2a561466ef in g_thread_new_internal (/lib64/libglib-2.0.so.0+0x8f6ef) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #2 0x7f2a56147226 in g_thread_pool_spawn_thread (/lib64/libglib-2.0.so.0+0x90226) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #3 0x7f2a56145102 in g_thread_proxy (/lib64/libglib-2.0.so.0+0x8e102) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #4 0x7f2a54896a26 in start_thread (/lib64/libc.so.6+0x8ea26) (BuildId: 21dc323755123cea8c34495928e904bc56fa71e3)

      Thread T8 created by T6 here:
          #0 0x7f2a56248956 in pthread_create (/lib64/libasan.so.8+0x48956) (BuildId: a1bdcca9c70162c2a1e8e7e9e8bc816ab51d214d)
          #1 0x7f2a561466ef in g_thread_new_internal (/lib64/libglib-2.0.so.0+0x8f6ef) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #2 0x7f2a56147226 in g_thread_pool_spawn_thread (/lib64/libglib-2.0.so.0+0x90226) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #3 0x7f2a56145102 in g_thread_proxy (/lib64/libglib-2.0.so.0+0x8e102) (BuildId: d287ded5d6cdc2667dc65f242b4a6546f9017281)
          #4 0x7f2a54896a26 in start_thread (/lib64/libc.so.6+0x8ea26) (BuildId: 21dc323755123cea8c34495928e904bc56fa71e3)

      SUMMARY: AddressSanitizer: double-free (/lib64/libasan.so.8+0xd7fb8) (BuildId: a1bdcca9c70162c2a1e8e7e9e8bc816ab51d214d) in __interceptor_free.part.0
      ==470375==ABORTING

[mcrha]

I might be completely wrong, though zchunk is not thread safe and it looks like you call zck_create from two threads at the same time. You should add locking around the zchunk (and possibly other) library calls or not run in multiple threads. The error handling in zchunk uses its own global context variable including zck_clear_error, where it's used when zck_create calls zck_clear_error with a NULL argument, which triggers that global zck_none global variable use (all the error functions have this "fallback").

[kontura]

This should be now resolved in zchunk: https://github.com/zchunk/zchunk/pull/107, since version 1.5.0.