QCHackers
[Question]: How to use detector database in demos?
Question: Suppose I want to present a demo of tqec generating and running a Stim circuit. I create a Jupyter notebook with all the code needed for this, just like our gallery examples. Running the notebook puts a database.pkl file locally. As I understand, this database has pre-computed the detectors which Stim samples from. We want an external user to clone my program and the database so that they can simply perform the sampling rather than waiting to re-compute the detectors.
What is the standard operating procedure for how the notebook should be written so that the user does not encounter bugs? Bugs are often a pickle.load() failure, where either a plaquette, circuit, or operation was missed, or a database versioning issue.
Apologies if this has already been explained. I want to make sure I understand what is supposed to work everytime so I can nail down any bugs with the caching if there are any.
Related: #681
the JSON format with a to_file line seems promising
@BSchelpe @Zhaoyilunnn let me know if you have any thoughts
the JSON format with a to_file line seems promising
@BSchelpe @Zhaoyilunnn let me know if you have any thoughts
If a pickle database built in one environment cannot be loaded correctly in another environment, I believe using json is ok.
The expected user experience should be like (if I understand correctly):
- Build a logical operation
- Simulate it, and the database will be built/updated in local env
- If the user simulate the same operation again, it will reuse the existing database
In this process, the user should not need to manually specify the database path (unless he really want a customized path), and should not need to delete/updated the database file manually.
I haven't looked into the issues related to #681 but I plan to investigate it. There must be some way to make the above process smoother
I agree with everything @Zhaoyilunnn says.
The only thing I thought was worth mentioning from @KabirDubey's description of what bugs show up: the database versioning issue superficially sounds like a different sort of issue from "plaquette, circuit, operation" being missed, but I think it is likely to have the same cause. If the version number is missing, the code assumes it is 0.0.0. This would then be incompatible with the current version number in the code and so lead to a versioning issue.
Reading @purva-thakre's comments on issue #681, it would be nice to narrow down whether the problems were due to using pickle instead json, rather python version differences or something else...Then it could be clear what the recommended workflow is. I guess this will be easier to test once @Zhaoyilunnn has made his improvements to how json is selected and it can be tested whether using json eliminates the problems.
I thought one thing to maybe check is what the default database path is on mac and linux - I think they may be different, and so when copied across the database may need saving in a different folder. Although it sounds as though you already did that and the code could find the database, just not load it correctly.
Thanks all. Will circle back to this when I am updating the demos from QCE, but will report the following in the meantime. I was testing Yiming's S gate notebook and got a database issue relating to the PauliBasis object.
Click to expand code
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[12], line 4
1 from tqec import NoiseModel, compile_block_graph
3 compiled_graph = compile_block_graph(graph)
----> 4 circuit = compiled_graph.generate_stim_circuit(
5 k=1, noise_model=NoiseModel.uniform_depolarizing(p=0.001)
6 )
File ~/Projects/dev/ytqec/tqec_pr719/src/tqec/compile/graph.py:529, in TopologicalComputationGraph.generate_stim_circuit(self, k, noise_model, manhattan_radius, detector_database, database_path, do_not_use_database, only_use_database)
493 def generate_stim_circuit(
494 self,
495 k: int,
(...) 501 only_use_database: bool = False,
502 ) -> stim.Circuit:
503 """Generate the ``stim.Circuit`` from the compiled graph.
504
505 Args:
(...) 527
528 """
--> 529 circuit = self.to_layer_tree().generate_circuit(
530 k,
531 manhattan_radius=manhattan_radius,
532 detector_database=detector_database,
533 database_path=database_path,
534 do_not_use_database=do_not_use_database,
535 only_use_database=only_use_database,
536 )
537 # If provided, apply the noise model.
538 if noise_model is not None:
File ~/Projects/dev/ytqec/tqec_pr719/src/tqec/compile/tree/tree.py:319, in LayerTree.generate_circuit(self, k, include_qubit_coords, manhattan_radius, detector_database, database_path, do_not_use_database, only_use_database, lookback)
317 if detector_database is None: # Nothing passed in,
318 if database_path.exists(): # look for an existing database at the path.
--> 319 detector_database = DetectorDatabase.from_file(database_path)
320 else: # if there is no existing database, create one.
321 detector_database = DetectorDatabase()
File ~/Projects/dev/ytqec/tqec_pr719/src/tqec/compile/detectors/database.py:507, in DetectorDatabase.from_file(filepath, format)
501 if format not in DetectorDatabase._READERS:
502 raise TQECError(
503 f"Could not read the database: the provided format {format} is not supported. "
504 + "Supported formats are:\n -"
505 + "\n -".join(DetectorDatabase._READERS.keys())
506 )
--> 507 return DetectorDatabase._READERS[format](filepath)
File ~/Projects/dev/ytqec/tqec_pr719/src/tqec/compile/detectors/database.py:222, in _DetectorDatabaseIO.from_pickle_file(filepath)
220 try:
221 with open(filepath, "rb") as f:
--> 222 database = pickle.load(f)
223 except pickle.PickleError as e:
224 moving_location = filepath.parent / f"faulty_database_{hash(e)}.pkl"
File ~/.pyenv/versions/3.12.5/lib/python3.12/enum.py:757, in EnumType.__call__(cls, value, names, module, qualname, type, start, boundary, *values)
755 if names is not _not_given:
756 value = (value, names) + values
--> 757 return cls.__new__(cls, value)
758 # otherwise, functional API: we're creating a new Enum type
759 if names is _not_given and type is None:
760 # no body? no data-type? possibly wrong usage
File ~/.pyenv/versions/3.12.5/lib/python3.12/enum.py:1171, in Enum.__new__(cls, value)
1169 ve_exc = ValueError("%r is not a valid %s" % (value, cls.__qualname__))
1170 if result is None and exc is None:
-> 1171 raise ve_exc
1172 elif exc is None:
1173 exc = TypeError(
1174 'error in %s._missing_: returned %r instead of None or a valid member'
1175 % (cls.__name__, result)
1176 )
ValueError: 'x' is not a valid PauliBasis
I searched to see where the pickle database was being written to:
python3 -c "from platformdirs import user_data_path; print(user_data_path(appname='TQEC'))"
and found detector_database.pkl in a system directory: Library/Application Support/TQEC/detector_database.pkl.
Removing it and then rerunning the cell worked. I think this is an issue with bumping the database version. I see commit 26699d93 “Bump the detector database version as a breaking change,” but my old database from before must have persisted.
I have not checked if PR #727 fixes this.
I have not checked if PR #727 fixes this.
No this PR aims to enable users to use json format database.
Removing it and then rerunning the cell worked. I think this is an issue with bumping the database version.
This should be another issue, if the database version has changed, the old database should not be read. I'll look into this issue.
I searched to see where the pickle database was being written to: python3 -c "from platformdirs import user_data_path; print(user_data_path(appname='TQEC'))"
and found detector_database.pkl in a system directory: Library/Application Support/TQEC/detector_database.pkl.
@KabirDubey I think this might be related to where your local directory is, how you have set it up, and how tqec loads it from a file. For me, it is a bit different on a Linux device. This is will be different on each user's local device OS. Do you mind verifying this when you get to resolving the other QCE tutorial-related issue?
Do you mind verifying this when you get to resolving the other QCE tutorial-related issue?
Makes sense, will do.