crater
crater copied to clipboard
Add a way to bless minicrater tests
Right now, failed tests have output like this:
Output
test minicrater::resource_exhaustion ... FAILED
failures:
---- minicrater::resource_exhaustion stdout ----
Difference between expected and actual reports:
{
"categories": [],
"comparison_colors": {},
"categories": [
[
"spurious-fixed",
{
"Plain": [
{
"name": "memory-hungry (local)",
"res": "spurious-fixed",
"runs": [
{
"log": "stable/local/memory-hungry",
"res": 0
},
{
"log": "beta/local/memory-hungry",
"res": 1
}
],
"url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
}
]
}
]
],
"comparison_colors": {
"spurious-fixed": {
"Striped": [
"#5630db",
"#5d3dcf"
]
}
},
"crates_count": 2,
"full": false,
"info": {
"test-pass": 2
"spurious-fixed": 1,
"test-pass": 1
},
"nav": [
{
"active": true,
"label": "Summary",
"url": "index.html"
},
{
"active": false,
"label": "Full report",
"url": "full.html"
},
{
"active": false,
"label": "Downloads",
"url": "downloads.html"
}
],
"result_colors": [],
"result_names": []
"result_colors": [
{
"Single": "#db3026"
},
{
"Single": "#65461e"
}
],
"result_names": [
"build OOM",
"test OOM"
]
}
To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/index.html.context.actual.json tests/minicrater/resource-exhaustion/index.html.context.expected.json
Difference between expected and actual reports:
{
"available_archives": [
{
"name": "All the crates",
"path": "logs-archives/all.tar.gz"
},
{
"name": "test-pass crates",
"path": "logs-archives/test-pass.tar.gz"
},
{
"name": "spurious-fixed crates",
"path": "logs-archives/spurious-fixed.tar.gz"
}
],
"crates_count": 2,
"nav": [
{
"active": false,
"label": "Summary",
"url": "index.html"
},
{
"active": false,
"label": "Full report",
"url": "full.html"
},
{
"active": true,
"label": "Downloads",
"url": "downloads.html"
}
]
}
To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/downloads.html.context.actual.json tests/minicrater/resource-exhaustion/downloads.html.context.expected.json
Difference between expected and actual reports:
{
"categories": [
[
"test-pass",
{
"Plain": [
{
"name": "build-pass (local)",
"res": "test-pass",
"runs": [
{
"log": "stable/local/build-pass",
"res": 0
},
{
"log": "beta/local/build-pass",
"res": 0
}
],
"url": "https://github.com/rust-lang/crater/tree/master/local-crates/build-pass"
},
}
]
}
],
[
"spurious-fixed",
{
"Plain": [
{
"name": "memory-hungry (local)",
"res": "test-pass",
"res": "spurious-fixed",
"runs": [
{
"log": "stable/local/memory-hungry",
"res": 0
"res": 1
},
{
"log": "beta/local/memory-hungry",
"res": 0
"res": 2
}
],
"url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
}
]
}
]
],
"comparison_colors": {
"spurious-fixed": {
"Striped": [
"#5630db",
"#5d3dcf"
]
},
"test-pass": {
"Single": "#72a156"
}
},
"crates_count": 2,
"full": true,
"info": {
"test-pass": 2
"spurious-fixed": 1,
"test-pass": 1
},
"nav": [
{
"active": false,
"label": "Summary",
"url": "index.html"
},
{
"active": true,
"label": "Full report",
"url": "full.html"
},
{
"active": false,
"label": "Downloads",
"url": "downloads.html"
}
],
"result_colors": [
{
"Single": "#62a156"
},
{
"Single": "#db3026"
},
{
"Single": "#65461e"
}
],
"result_names": [
"test passed"
"test passed",
"build OOM",
"test OOM"
]
}
To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/full.html.context.actual.json tests/minicrater/resource-exhaustion/full.html.context.expected.json
Difference between expected and actual reports:
{
"categories": [],
"categories": [
[
"spurious-fixed",
{
"Plain": [
{
"krate": {
"Local": "memory-hungry"
},
"name": "memory-hungry (local)",
"res": "spurious-fixed",
"runs": [
{
"log": "stable/local/memory-hungry",
"res": "build-fail:oom"
},
{
"log": "beta/local/memory-hungry",
"res": "test-fail:oom"
}
],
"url": "https://github.com/rust-lang/crater/tree/master/local-crates/memory-hungry"
}
]
}
]
],
"crates_count": 2,
"full": false,
"info": {
"test-pass": 2
"spurious-fixed": 1,
"test-pass": 1
}
}
To expect the new report in the future run:
$ cp tests/minicrater/resource-exhaustion/markdown.md.context.actual.json tests/minicrater/resource-exhaustion/markdown.md.context.expected.json
thread 'minicrater::resource_exhaustion' panicked at 'invalid report generated by Crater', tests/minicrater/driver.rs:212:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
minicrater::resource_exhaustion
This does have instructions for blessing the failure (cp tests/minicrater/resource-exhaustion/markdown.md.context.actual.json tests/minicrater/resource-exhaustion/markdown.md.context.expected.json
), but there's so much output it's easy to miss it. In particular, if more than one test fails it's very easy to miss that there are multiple commands you need to run, since they might have disappeared off your terminal screen, requiring you to run the tests again to see the output.
It would be great to have a cargo test minicrater -- --bless
command that automatically updates the expected JSON files.