Nicholas Smith

Results 88 comments of Nicholas Smith

I'm happy to comment where I can, but I may already be behind you after your thorough survey. From a far-away view, here are some random ideas: - The ray...

> For INT, delta encoding would not allow decompress-to-target For FLOAT, byte_stream_split likewise These encodings are something we will want in our data, especially the split float (due to our...

> are all possible values of SliceFuture._parent an Uproot future? In the case of FSSpec (the only source currently using the coalescing algorithm), none of them are. They are python...

So as I understand, the main bottleneck is having to issue `get_dataset_locks` for each block in a large CMS dataset-container. And the way around appears to be not to cross-check...

I had hoped that using [list_replica_locks](https://rucio.readthedocs.io/en/latest/api/rule.html#rucio.client.ruleclient.RuleClient.list_replica_locks), despite being file-level, would work better since you instantly get the distribution of replicas for a given rule, and we already have effective filtering...

Ok, we can probably just cache the cluster stop in this subroutine and reduce this bottleneck. On the subject of MemmapSource, recall the main issue with it is each such...

It's @dr-stringfellow Such cases I think we need to invalidate the data in DBS so it is not used in workflows again. Would this advancing the rule cause that to...

I'm confused, why do we need to have `truncate=True` in the touch command if it is only called when the file doesn't exist?

Since you later want to open a file with truncate on open, I would suggest to use that interface in fsspec-xrootd. Open the file with mode `w` once to touch/create/truncate...

Furthermore, I suspect interacting in read-write mode with a remote filesystem is not a great idea. We should encourage users to add the `simplecache::` (see fsspec [docs](https://filesystem-spec.readthedocs.io/en/latest/features.html#remote-write-caching)) prefix to the...