databroker

databroker copied to clipboard

Reinstate just enough deep copying to guard against cache corruption

1

We should make it impossible for callers to modify objects that are cached internally. This will involve deep copying somewhere and might carry a significant performance cost, so we may...

danielballan

## Expected Behavior ## Current Behavior ``` MongoAdapter._build_event_stream(self, run_start_uid, stream_name, is_complete) 1466 # We need each of the sub-dicts to have a consistent length. If 1467 # Events are still...

tacaswell

Record created_at and updated_at timestamps in experimental Document

1

It may be useful to sort analysis results by the most recently created one and also note when metadata has last been updated.

danielballan

Too many open files SRX

2

We are getting a "too many open files" error with databroker-tiled when loading data at SRX. This can be worked around by using databroker-intake and running ``` db._catalog._entries.cache_clear() ```

gwbischof

Send xarrays with tiled

1

Edit from @danielballan: This PR prompted a re-think in how we handle xarrays in Tiled in general. We'll leave it open for now as there may be something useful to...

jmaruland

An incremental step to an "embedded" implementation

2

## Status Quo We encode Bluesky documents directly into MongoDB documents. There is one MongoDB collection per document type, and each Bluesky document becomes one MongoDB document. The MongoDB representation...

danielballan

Out of memory HXN

1

"There seems to be a memory leak when loading a batch of data with PyXRF using Data Broker V1 (no memory leak when using Data Broker V0). After loading 13...

gwbischof

"It takes about 35 seconds to load one 75x75 2D scan (e.g. 92276) with PyXRF using Data Broker with tiled V1 compared to 3.5-4.0 seconds using Data Broker V0. It...

gwbischof

Handle case where there is a 'time' data key.

2

In databroker, we make a best effort to fit the streaming "document model" into standard PyData types. In v1, we use DataFrames (`.table(...)`) or generators (`.data(...)`). In v2, we use...

danielballan

Update indexes to use tie-break ObjectId

17

Paginated requests are always sorted so that the pages are deterministic. To avoid nondeterministic "ties" in the sorting parameters, they use ObjectId as a tie-breaker. https://github.com/bluesky/databroker/blob/de3a2e9b2974353946af938ff6d6128623f1e625/databroker/mongo_normalized.py#L1561-L1568 This means that we...

danielballan