qiskit icon indicating copy to clipboard operation
qiskit copied to clipboard

more useful output for update_fake_backends.py

Open 1ucian0 opened this issue 3 years ago • 3 comments

This PR extends tools/update_fake_backends.py to warning the user when the snapshot has errors of 1.

@nonhermitian noticed that some calibration data was bad when I was creating FakeGeneva in https://github.com/Qiskit/qiskit-terra/pull/8322#issuecomment-1181595235

Indeed, many property json files contain gates with gate_error==1

$ rgrep '"gate_error", "unit": "", "value": 1}' qiskit -l
qiskit/providers/fake_provider/backends/rochester/props_rochester.json
qiskit/providers/fake_provider/backends/washington/props_washington.json
qiskit/providers/fake_provider/backends/cambridge/props_cambridge_alt.json
qiskit/providers/fake_provider/backends/cambridge/props_cambridge.json
qiskit/providers/fake_provider/backends/manhattan/props_manhattan.json

Moreover, many exploratory backends have errors==1 somewhere, but some calibrations are better than others. So it is important to show the relative amount of gates with error==1.

Additionally, the script tools/update_fake_backends.py might silently fail and it is hard to see why is not doing what it suppose to do. Take the situation of trying to update a backend you dont have access to. It runs without any output, but does not generate the expected files.

In this PR, the script outputs when some of the args.backends did not get updated or if some of the json files were skipped.

This is how it looks:

$ ./tools/update_fake_backends.py --hub <redacted> ibm_washington

ibm_washington
==============
Readout errors == 1:
 * qubit 3
Gate errors == 1:
 * cx 96, 109
 * cx 109, 96
 * cx 118, 119
 * cx 3, 4
 * cx 10, 9
 * cx 9, 10
 * cx 119, 118
 * cx 4, 3
╒═══════════════╤═════════════════════════════════╤═════════════════════════════════════╕
│    error == 1 │  new 2022-07-24 09:02:23+02:00  │  current 2022-04-12 23:42:47+09:00  │
╞═══════════════╪═════════════════════════════════╪═════════════════════════════════════╡
│ readout error │          1/127 (0.79%)          │            0/127 (0.00%)            │
├───────────────┼─────────────────────────────────┼─────────────────────────────────────┤
│      cx error │          8/284 (2.82%)          │            6/284 (2.11%)            │
├───────────────┼─────────────────────────────────┼─────────────────────────────────────┤
│     all gates │          8/919 (0.87%)          │            6/919 (0.65%)            │
╘═══════════════╧═════════════════════════════════╧═════════════════════════════════════╛

In addition, I added a parameter --datetime to be able to fetch specific snapshots, like suggested here.

1ucian0 avatar Jul 13 '22 10:07 1ucian0

Thank you for opening a new pull request.

Before your PR can be merged it will first need to pass continuous integration tests and be reviewed. Sometimes the review process can be slow, so please be patient.

While you're waiting, please feel free to review other open PRs. While only a subset of people are authorized to approve pull requests for merging, everyone is encouraged to review open pull requests. Doing reviews helps reduce the burden on the core team and helps make the project's code better for everyone.

One or more of the the following people are requested to review this:

  • @Qiskit/terra-core

qiskit-bot avatar Jul 13 '22 10:07 qiskit-bot

Pull Request Test Coverage Report for Build 3146527252

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 11 unchanged lines in 1 file lost coverage.
  • Overall coverage increased (+0.003%) to 84.492%

Files with Coverage Reduction New Missed Lines %
qiskit/compiler/transpiler.py 11 89.71%
<!-- Total: 11
Totals Coverage Status
Change from base Build 3146373335: 0.003%
Covered Lines: 60199
Relevant Lines: 71248

💛 - Coveralls

coveralls avatar Jul 13 '22 10:07 coveralls

I added some extra explanation and follow the advice in https://github.com/Qiskit/qiskit-terra/pull/8336#discussion_r947289397

ibm_washington
==============
Broken operations (gates with errors == 1)
 * cx 96, 109
 * cx 109, 96
 * cx 118, 119
 * cx 3, 4
 * cx 10, 9
 * cx 9, 10
 * cx 119, 118
 * cx 4, 3
╒═════════════════════════════════════╤═════════════════════════════════╤═════════════════════════════════════╕
│                                     │  new 2022-07-23 23:10:17+02:00  │  current 2022-04-11 14:24:11+02:00  │
╞═════════════════════════════════════╪═════════════════════════════════╪═════════════════════════════════════╡
│                broken cx / total cx │          8/284 (2.82%)          │            6/284 (2.11%)            │
├─────────────────────────────────────┼─────────────────────────────────┼─────────────────────────────────────┤
│ broken operations / total operation │          8/919 (0.87%)          │            6/919 (0.65%)            │
╘═════════════════════════════════════╧═════════════════════════════════╧═════════════════════════════════════╛
The calibration you are pulling (in the "new" column) contain some broken error values (when the error is 1). However, if the rate of broken errors is better than the current one, you might want to update the files.

1ucian0 avatar Sep 16 '22 16:09 1ucian0

Closing in favor of https://github.com/Qiskit/qiskit-ibm-runtime/pull/1262 as IBM specific fake backends are moving to https://github.com/Qiskit/qiskit-ibm-runtime

1ucian0 avatar Dec 06 '23 12:12 1ucian0