bento check reports commited-thus-ignored findings as "archived"

[ievans]

Overview

Bento described N archived findings, even if the archive contains 0.

Current Behavior

> git commit -am "foo"
...
◦ Not showing 4 archived finding(s). To view, run $ cat .bento/archive.json
ine@ismcp ~/D/r/wg (master) [1]> cat .bento/archive.json
{
  "bandit": {
    "violations": {}
  },
  "flake8": {
    "violations": {}
  },
  "r2c.click": {
    "violations": {}
  },
  "r2c.flask": {
    "violations": {}
  },
  "r2c.requests": {
    "violations": {}
  },
  "shellcheck": {
    "violations": {}
  }
}⏎

Steps to Reproduce

1. Run bento on a new project without archiving

Possible Solution

Do you have an idea for how to address the issue or feedback?

Additional Context

Bento version (run bento --version): Operation system (e.g. Ubuntu 18.04): Any other context that would help us reproduce or understand the issue:

[underyx]

I couldn't reproduce this on a random repo, @ievans could you provide a specific repo and commit that triggers this issue?

[underyx]

I was told that @brendongo you might know how to reproduce this as well?

[dlukeomalley]

@brendongo will have the most context. My impression is that this is caused by the base commit having results that disappear in the head commit. You should be able to reproduce based on that, though Brendon knows the specific area of code causing this (it's his logic). Thanks @underyx!

[brendongo]

You can probably reproduce this by creating an empty archive file and running bento check on some staged changes (and there are findings that would occur if bento check --all was run before the staged changes were made.

https://github.com/returntocorp/bento/blob/master/bento/commands/check.py#L213

n_archived = n_all - n_all_filtered

n_all_filtered is a count of items that are "filtered". When bento used to run on just a single commit this was fine but when we changed bento to effectively run twice (once to set the base and a second time over the new staged changes) we include that generated base as "filtered" findings even though they are not in the archive.

Relevant code places: orchestrator.py:orchestrate:53 -> runner.parallel_results(tools, baseline) note that baseline includes the archive and head comparison.

We use that filtered flag in our anonymized metrics so you might want to take a look at metrics.py:__get_aggregate_violations.

The solution might be to print out "X findings in archive, Y pre-existing findings in head commit" or something to that effect. Or even to just open the archive file and count the number of findings there to print out in this message. @dlukeomalley for thoughts here.

[nbrahms]

I tried the steps recommended by @brendongo, but could not reproduce this issue. @ievans , I'm going to close this as cannot-reproduce for now. I'll reopen if you end up posting reproduction steps here.

[brendongo]

Reproducing:

git clone https://github.com/pallets/flask
git checkout 38eb5d3b49d628785a470e2e773fc5ac82e3c8e4
bento init
echo "\n" >> tests/test_templating.py
git add tests/test_templating.py
bento check

Prints out

✔ 0 findings due to staged changes in 1.53 s
◦ Not showing 3 archived finding(s). To view, run $ cat .bento/archive.json

when there is no archive file.