kani icon indicating copy to clipboard operation
kani copied to clipboard

Add async/await examples from tokio test suite

Open fzaiser opened this issue 2 years ago • 6 comments

I went through the tokio's test suite and identified a couple of test cases that don't use a lot of I/O (or replace it with a mock) so that they seem like a reasonable target for Kani.

Description of changes:

No changes to how Kani works, this only adds tests that are ignored for now.

Testing:

  • How is this change tested? No change of functionality, just adding tests

  • Is this a refactor change? No.

Checklist

  • [x] Each commit message has a non-empty body, explaining why the change was made
  • [x] Methods or procedures are documented
  • [x] Regression or unit tests are included, or existing tests cover the modified code
  • [x] My PR is restricted to a single feature or bugfix

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.

fzaiser avatar Jul 05 '22 16:07 fzaiser

The tests have very few comments making it difficult to understand what they're doing. Can you add some comments, at least for the ones for which you have an idea what the purpose of the test is?

zhassan-aws avatar Jul 05 '22 16:07 zhassan-aws

The tests compile under Kani now, but they take a very long time. I stopped the first harness (io_chain.rs) after two hours.

fzaiser avatar Aug 09 '22 15:08 fzaiser

The tests compile under Kani now, but they take a very long time. I stopped the first harness (io_chain.rs) after two hours.

I saw you're using [kani::unwind(32)] for all harnesses. Have you tried using the minimum required for each harness?

adpaco-aws avatar Aug 10 '22 15:08 adpaco-aws

I saw you're using [kani::unwind(32)] for all harnesses. Have you tried using the minimum required for each harness?

@adpaco-aws I will try this, but it's hard to know upfront what the necessary limit will be. Maybe I should try a smaller limit and successively increase it until it passes.

fzaiser avatar Aug 10 '22 15:08 fzaiser

Progress report

Successful verification: 5 harnesses (basic_usage_fuse, basic_usage_once, empty_box_slice, empty_string, empty_vec)

Failed verification:

  • 9 harnesses: too low unwinding bound (basic_usage_empty, empty_unit, issue_4435, merge_sync_streams, read_buf, read_exact, read, take, write_cursor)
  • 4 harnesses: missing functions (empty, writ_buf_err, async_block, async_fn)
     Check 3880: _ZN3std3sys4unix4rand19hashmap_random_keys17hdd863e684c97ac1dE.1
      - Status: FAILURE
      - Description: "assertion"
      - Location: Unknown File in function _ZN3std3sys4unix4rand19hashmap_random_keys17hdd863e684c97ac1dE
     Check 4474: _ZN3std2io5error5Error4_new17ha2be3d29de30fdd1E.1
      - Status: FAILURE
      - Description: "assertion"
      - Location: Unknown File in function _ZN3std2io5error5Error4_new17ha2be3d29de30fdd1E
    
    missing epoll: 2 harnesses (async_block, async_fn)
    Check 5050: epoll_create1.1
     - Status: FAILURE
     - Description: "assertion"
     - Location: Unknown File in function epoll_create1
    
  • 1 harness: unreachable code (empty)
    Check 1375: std::thread::local::lazy::LazyKeyInner::<std::cell::Cell<(u64, u64)>>::get.unreachable.1
     - Status: FAILURE
     - Description: "unreachable code"
     - Location: ~/.rustup/toolchains/nightly-2022-07-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:627:15 in function std::thread::local::lazy::LazyKeyInner::<std::cell::Cell<(u64, u64)>>::get
    
  • rest: timed out or hit #1499

I will increase the unwinding bound for the ones with too low a bound.

fzaiser avatar Aug 11 '22 22:08 fzaiser

Progress report after raising the unwind bound for some tests

Successful verification: 13 harnesses (basic_usage_fuse, basic_usage_once, empty_box_slice, empty_string, empty_vec, basic_usage_empty, empty_unit, issue_4435, read_exact, read, take, write_cursor, empty_result)

Failed verification:

  • 4 harnesses: missing functions (empty, writ_buf_err, async_block, async_fn)
     Check 3880: _ZN3std3sys4unix4rand19hashmap_random_keys17hdd863e684c97ac1dE.1
      - Status: FAILURE
      - Description: "assertion"
      - Location: Unknown File in function _ZN3std3sys4unix4rand19hashmap_random_keys17hdd863e684c97ac1dE
     Check 4474: _ZN3std2io5error5Error4_new17ha2be3d29de30fdd1E.1
      - Status: FAILURE
      - Description: "assertion"
      - Location: Unknown File in function _ZN3std2io5error5Error4_new17ha2be3d29de30fdd1E
    
    missing epoll: 2 harnesses (async_block, async_fn)
    Check 5050: epoll_create1.1
     - Status: FAILURE
     - Description: "assertion"
     - Location: Unknown File in function epoll_create1
    
  • ~1 harness: memcmp invalid pointer (empty_result) see #1489~ fixed
  • 1 harness: unreachable code (empty)
    Check 1375: std::thread::local::lazy::LazyKeyInner::<std::cell::Cell<(u64, u64)>>::get.unreachable.1
     - Status: FAILURE
     - Description: "unreachable code"
     - Location: ~/.rustup/toolchains/nightly-2022-07-19-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/option.rs:627:15 in function std::thread::local::lazy::LazyKeyInner::<std::cell::Cell<(u64, u64)>>::get
    
  • rest: timed out or hit #1499

fzaiser avatar Aug 12 '22 16:08 fzaiser

So the regresion tests pass now, but they take between 120 min (ubuntu-20.04) and 204 min (macos-11). It probably makes more sense to have those in a separate "slow" test suite.

As an alternative, we could add them as a compile-only test suite, which should be faster.

fzaiser avatar Sep 06 '22 04:09 fzaiser

I'd support adding a "slow" test suite, and running it nightly.

I'm not sure how many concurrent runners we get at the free tier. If its sufficient, we could also make it a "non-required" test so we would get information if it failed, but it wouldn't block work on PRs.

danielsn avatar Sep 06 '22 15:09 danielsn

I've added a new slow test suite as suggested.

fzaiser avatar Sep 08 '22 17:09 fzaiser

Another idea that @adpaco-aws mentioned is to run the slow tests after merging into main. But for now, I think the most important thing is to get the tests into main at all.

fzaiser avatar Sep 08 '22 20:09 fzaiser

CI seems to have failed due to network issues, retrying

fzaiser avatar Sep 13 '22 14:09 fzaiser

@danielsn I don't have rights to merge anymore. Can you merge it for me please?

fzaiser avatar Sep 14 '22 01:09 fzaiser

Results of the slow test suite can be seen here: https://github.com/model-checking/kani/actions/workflows/slow-tests.yml

fzaiser avatar Sep 16 '22 02:09 fzaiser