crater icon indicating copy to clipboard operation
crater copied to clipboard

Add a way to bless minicrater tests

Open jyn514 opened this issue 3 years ago • 0 comments

Right now, failed tests have output like this:

Output
test minicrater::resource_exhaustion ... FAILED

failures:

---- minicrater::resource_exhaustion stdout ----
Difference between expected and actual reports:
{
  "categories": [],
  "comparison_colors": {},
  "categories": [
    [
      "spurious-fixed",
      {
        "Plain": [
          {
            "name": "memory-hungry (local)",
            "res": "spurious-fixed",
            "runs": [
              {
                "log": "stable/local/memory-hungry",
                "res": 0
              },
              {
                "log": "beta/local/memory-hungry",
                "res": 1
              }
            ],
            "url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
          }
        ]
      }
    ]
  ],
  "comparison_colors": {
    "spurious-fixed": {
      "Striped": [
        "#5630db",
        "#5d3dcf"
      ]
    }
  },
  "crates_count": 2,
  "full": false,
  "info": {
    "test-pass": 2
    "spurious-fixed": 1,
    "test-pass": 1
  },
  "nav": [
    {
      "active": true,
      "label": "Summary",
      "url": "index.html"
    },
    {
      "active": false,
      "label": "Full report",
      "url": "full.html"
    },
    {
      "active": false,
      "label": "Downloads",
      "url": "downloads.html"
    }
  ],
  "result_colors": [],
  "result_names": []
  "result_colors": [
    {
      "Single": "#db3026"
    },
    {
      "Single": "#65461e"
    }
  ],
  "result_names": [
    "build OOM",
    "test OOM"
  ]
}


To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/index.html.context.actual.json tests/minicrater/resource-exhaustion/index.html.context.expected.json

Difference between expected and actual reports:
{
  "available_archives": [
    {
      "name": "All the crates",
      "path": "logs-archives/all.tar.gz"
    },
    {
      "name": "test-pass crates",
      "path": "logs-archives/test-pass.tar.gz"
    },
    {
      "name": "spurious-fixed crates",
      "path": "logs-archives/spurious-fixed.tar.gz"
    }
  ],
  "crates_count": 2,
  "nav": [
    {
      "active": false,
      "label": "Summary",
      "url": "index.html"
    },
    {
      "active": false,
      "label": "Full report",
      "url": "full.html"
    },
    {
      "active": true,
      "label": "Downloads",
      "url": "downloads.html"
    }
  ]
}


To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/downloads.html.context.actual.json tests/minicrater/resource-exhaustion/downloads.html.context.expected.json

Difference between expected and actual reports:
{
  "categories": [
    [
      "test-pass",
      {
        "Plain": [
          {
            "name": "build-pass (local)",
            "res": "test-pass",
            "runs": [
              {
                "log": "stable/local/build-pass",
                "res": 0
              },
              {
                "log": "beta/local/build-pass",
                "res": 0
              }
            ],
            "url": "https://github.com/rust-lang/crater/tree/master/local-crates/build-pass"
          },
          }
        ]
      }
    ],
    [
      "spurious-fixed",
      {
        "Plain": [
          {
            "name": "memory-hungry (local)",
            "res": "test-pass",
            "res": "spurious-fixed",
            "runs": [
              {
                "log": "stable/local/memory-hungry",
                "res": 0
                "res": 1
              },
              {
                "log": "beta/local/memory-hungry",
                "res": 0
                "res": 2
              }
            ],
            "url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
          }
        ]
      }
    ]
  ],
  "comparison_colors": {
    "spurious-fixed": {
      "Striped": [
        "#5630db",
        "#5d3dcf"
      ]
    },
    "test-pass": {
      "Single": "#72a156"
    }
  },
  "crates_count": 2,
  "full": true,
  "info": {
    "test-pass": 2
    "spurious-fixed": 1,
    "test-pass": 1
  },
  "nav": [
    {
      "active": false,
      "label": "Summary",
      "url": "index.html"
    },
    {
      "active": true,
      "label": "Full report",
      "url": "full.html"
    },
    {
      "active": false,
      "label": "Downloads",
      "url": "downloads.html"
    }
  ],
  "result_colors": [
    {
      "Single": "#62a156"
    },
    {
      "Single": "#db3026"
    },
    {
      "Single": "#65461e"
    }
  ],
  "result_names": [
    "test passed"
    "test passed",
    "build OOM",
    "test OOM"
  ]
}


To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/full.html.context.actual.json tests/minicrater/resource-exhaustion/full.html.context.expected.json

Difference between expected and actual reports:
{
  "categories": [],
  "categories": [
    [
      "spurious-fixed",
      {
        "Plain": [
          {
            "krate": {
              "Local": "memory-hungry"
            },
            "name": "memory-hungry (local)",
            "res": "spurious-fixed",
            "runs": [
              {
                "log": "stable/local/memory-hungry",
                "res": "build-fail:oom"
              },
              {
                "log": "beta/local/memory-hungry",
                "res": "test-fail:oom"
              }
            ],
            "url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
          }
        ]
      }
    ]
  ],
  "crates_count": 2,
  "full": false,
  "info": {
    "test-pass": 2
    "spurious-fixed": 1,
    "test-pass": 1
  }
}


To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/markdown.md.context.actual.json tests/minicrater/resource-exhaustion/markdown.md.context.expected.json

thread 'minicrater::resource_exhaustion' panicked at 'invalid report generated by Crater', tests/minicrater/driver.rs:212:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    minicrater::resource_exhaustion

This does have instructions for blessing the failure (cp tests/minicrater/resource-exhaustion/markdown.md.context.actual.json tests/minicrater/resource-exhaustion/markdown.md.context.expected.json), but there's so much output it's easy to miss it. In particular, if more than one test fails it's very easy to miss that there are multiple commands you need to run, since they might have disappeared off your terminal screen, requiring you to run the tests again to see the output.

It would be great to have a cargo test minicrater -- --bless command that automatically updates the expected JSON files.

jyn514 avatar Sep 10 '20 15:09 jyn514