substrate icon indicating copy to clipboard operation
substrate copied to clipboard

Force base weights to be the minimum only when the intercept is negative

Open koute opened this issue 3 years ago • 4 comments

Followup of https://github.com/paritytech/substrate/pull/11806 and https://github.com/paritytech/substrate/pull/12386

This PR changes two things:

  1. The base weights are now forced to be equal to the minimum execution time only if the intercept of the linear regression is negative. Looking at https://github.com/paritytech/substrate/pull/12325 there are still issues with doing this unconditionally and not all of our benchmarks are ready for it. So this PR is a compromise that forces the base weights to be the minimum only when it's necessary (when the base weight would otherwise be negative), and when it's not it just uses whatever intercept was calculated. This should hopefully cut down on the number of benchmarks we need to fix.

  2. The minimum execution time is now printed out in the comments generated along with the weights. We often use benchmarks to see whether something got slower/faster, however the weights generated through linear regression are not really an appropriate way to do that. With the base weights always being the minimum execution times we could have used those to see how slow/fast something actually is (since they were not a product of the linear regression), but since after this PR those base weights won't always be equal to the minimum execution times anymore we just move this information into the comments. This doesn't really change anything functionally, and it's just there as extra information for whoever is reading the file with the weights.

koute avatar Oct 12 '22 12:10 koute

/cmd queue -c bench-bot $ pallet dev pallet_contracts

ggwpez avatar Oct 13 '22 14:10 ggwpez

@ggwpez https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1948761 was started for your command "$PIPELINE_SCRIPTS_DIR/bench-bot.sh" pallet dev pallet_contracts. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 9-3e34654e-a477-4628-8238-8e7897430dfd to cancel this command or /cmd cancel to cancel all commands in this pull request.

command-bot[bot] avatar Oct 13 '22 14:10 command-bot[bot]

@ggwpez Command "$PIPELINE_SCRIPTS_DIR/bench-bot.sh" pallet dev pallet_contracts has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1948761 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1948761/artifacts/download.

command-bot[bot] avatar Oct 13 '22 17:10 command-bot[bot]

Update: fixed in b27f42c804fe9d755856ca9ec5cf61c24924dada

The benchbot failed with Error: Io(Custom { kind: Other, error: "Failed to access JSON data." }).
I hope it was not this one https://github.com/paritytech/pipeline-scripts/pull/82

PS: This error seems to come from Handlebar.

ggwpez avatar Oct 14 '22 09:10 ggwpez

/cmd queue -c bench-bot $ pallet dev pallet_assets

ggwpez avatar Oct 20 '22 16:10 ggwpez

@ggwpez https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1972669 was started for your command "$PIPELINE_SCRIPTS_DIR/bench-bot.sh" pallet dev pallet_assets. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 28-43972c72-8f82-4e7f-9da0-5d87650901dd to cancel this command or /cmd cancel to cancel all commands in this pull request.

command-bot[bot] avatar Oct 20 '22 16:10 command-bot[bot]

@ggwpez Command "$PIPELINE_SCRIPTS_DIR/bench-bot.sh" pallet dev pallet_assets has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1972669 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1972669/artifacts/download.

command-bot[bot] avatar Oct 20 '22 18:10 command-bot[bot]

/cmd queue -c bench-bot $ pallet dev pallet_uniques

ggwpez avatar Oct 20 '22 18:10 ggwpez

@ggwpez https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1973082 was started for your command "$PIPELINE_SCRIPTS_DIR/bench-bot.sh" pallet dev pallet_uniques. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 29-0fef2a97-b085-4e14-90af-ada602a12367 to cancel this command or /cmd cancel to cancel all commands in this pull request.

command-bot[bot] avatar Oct 20 '22 18:10 command-bot[bot]

@ggwpez Command "$PIPELINE_SCRIPTS_DIR/bench-bot.sh" pallet dev pallet_uniques has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1973082 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/1973082/artifacts/download.

command-bot[bot] avatar Oct 20 '22 19:10 command-bot[bot]

bot merge

ggwpez avatar Oct 20 '22 20:10 ggwpez