[WIP] Data storage interface

[WilliamHPNielsen]

Summary

Presents an interface to abstract over the storage media in DataSet. This is the main feature branch which means that the RMQ interface will also end up here.

This PR strips out any mention of the SQLite database from the DataSet and thereby inevitably introduces non-backwards compatible API changes. Note that we guarantee 100% compatibility of all (non-corrupt) existing DB files with this new interface.

Replaces #1383.

List of most important changes being introduced:

• DataStorageInterface is defined to abstract storage and retrieval of datasets (runs). The DataStorageInterface consists of two parts, a reader and a writer, and they may be of different type. This is to allow writing to RabbitMQ but reading directly from SQLite.
• An SQLiteReaderInterface and an SQLiteWriterInterface are implemented. When used in the same DataStorageInterface they provide the same service/performance for direct local sqlite DB storage that is currently implemented directly into DataSet on master branch. A notable novelty is that the reader and the writer have a connection each, which the tests must take extra care to close.
• Remove deprecated methods of DataSet: ...
• Do not allow to add_results (with 's') if DataSet is completed
• Introduce pristine and running properties of DataSet which correspond to its possible state, started property is removed (although an attribute remains)
• RMQStorageInterface - pending in PR #1388

ToDo-list:

• [x] Split DSI into reader and writer - for DSI and for SQL DSI
• [ ] fix type: ignore problems in store_results and DatSet.get_setpoints
• [ ] make sure that DataSet methods handle the case of results being Dict[str, ndarray] properly
• [ ] Add notion of result_counter into DataSet and DSI:
  • When you pass results to store_results, you must specify: "Dear DSI, these are results number N to M, meaning that they were recorded (by the Measurement) as the Nth point, the N+1st point etc."
  • This specification of result_number is what enables the DSI to play back the points in the same absolute order (meaning that I don't only know that point A is before B, I also know that A is first, meaning that there is nothing before A) as they were recorded.
  • results = {'x': [1, 2, 3], 'y': [2.3, -4, 5.1], '__start__': 20}, 0-based, length defines "stop".
• [ ] Replay_results should use python indexing, get_data and other DataSet has to use old school 1-based sql indexing :(
• [ ] Sql DSI Retrieve_results waits for Jens' PR/branch
• [ ] _set_run_description uses sqlite_base.get_parameters to get currently inserted parameters. Perhaps, it shoud do it via the metadata (meaning, via current run_description)
• [ ] ParamSpec, _depends_on and _inferred_from properties should NOT return freaking strings, they should return the lists.
• [ ] Fix "add_results lets you add a parameter value without setpoint values"
• [ ] Refactor "parameter dependency validation" into InterDependencies and use it in add_parameter and add_result(s) (i.e. remove fugly checks)
• [ ] Remove _started and substitute it with "not pristine" in the code everywhere
• [ ] Implement removing metadata?
• [ ] Test that the not_given logic for store_meta_data works
• [ ] What to do with the idea of VERSION of interface classes? do we care?
• [ ] Make the SQLiteReaderInterface have a read-only connection.
• [ ] Disable the Experiment API from making any changes to the database. All writing must happen through the DSI (but we can allow some non-authorized reading for now)

• ...

[codecov[bot]]

Codecov Report

Merging #1415 into master will increase coverage by 3.99%. The diff coverage is 92.42%.

@@            Coverage Diff             @@
##           master    #1415      +/-   ##
==========================================
+ Coverage   70.52%   74.51%   +3.99%     
==========================================
  Files         103       87      -16     
  Lines       11608    10293    -1315     
==========================================
- Hits         8186     7670     -516     
+ Misses       3422     2623     -799

[WilliamHPNielsen]

This PR has been superseded by the in-memory cache as the data interface.