Jack Kelly comments

Results 383 comments of


                                            Jack Kelly

What would user's code would look like the for simple use-cases? (After dropping `async`, and instead only using `channels` and `Rayon`)?

> can we use a flat structure We can't have a flat list, where list items define the end of groups, because Rayon processes things out-of-order. So the following code...

What would user's code would look like the for simple use-cases? (After dropping `async`, and instead only using `channels` and `Rayon`)?

I should test the channel of channels idea in my Rust playground

What would user's code would look like the for simple use-cases? (After dropping `async`, and instead only using `channels` and `Rayon`)?

Yup, using channels of channels works for both submission and completion! ```rust use rayon::prelude::*; #[derive(Debug)] enum Operation { Get(u8), } fn main() { let (completion_tx, completion_rx) = crossbeam::channel::bounded(4); { let...

Multiple threads (each with a uring) & `crossbeam::deque` & a pipeline of I/O steps

What happens if we want to read 1 million chunks from a single file. As currently designed, that will all be handled by a single thread, and other threads will...

Multiple threads (each with a uring) & `crossbeam::deque` & a pipeline of I/O steps

On the injector queue, there will be on file per Operation. Reading a million chunks from one file will be represented by one Operation. The thread that receives that Operation...

Multiple threads (each with a uring) & `crossbeam::deque` & a pipeline of I/O steps

See `fio` results in [this comment](https://github.com/JackKelly/light-speed-io/issues/50#issuecomment-1983479259) for evidence that multiple threads are required to achieve top speed

Multiple threads (each with a uring) & `crossbeam::deque` & a pipeline of I/O steps

Some updates: I think most of the ideas above are still relevant. One big change is that I'm now hoping to use `Rayon` to manage the threadpools and injector queue...

Multiple threads (each with a uring) & `crossbeam::deque` & a pipeline of I/O steps

This is all implemented in PR #125!

Cache chunks in RAM

## Simplify caching! And restrict it to a single layer! Caching should exist only in a thin layer ~after the planning, and before reading~ UPDATE: actually, shouldn't caching exist between...

Cache chunks in RAM

## Consider using a `Map` where each `File` struct would store cached filesize, and cached data Then, maybe assign one `File` to each `FileChunks`, so that we can safely plan...