k6 icon indicating copy to clipboard operation
k6 copied to clipboard

Add Streams API support to k6

Open oleiade opened this issue 2 years ago • 0 comments

Problem Statement

The Streams API is a specification for JavaScript which aims to provide users with programmatically access streams of data and process them as desired by the developer. It was initially designed for networking and with the Fetch API in mind, however, in light of recent work made on both improving the handling of large files in k6 and our future HTTP API, it appears that it might be a good candidate to improve and streamline data consumption and manipulation in k6 too.

Proposal

We propose to provide a subset of the streams API as a k6 module, and expose it to both k6 internals as well as our public API. Judging by the scope of the Streams API itself, we propose that only ReadableStream would be implemented at first. If necessary at a later time, we could then implement support for writable and transformable streams too.

One of the main use case for streams as of this day would be around data processing, either from a file, or from a source in memory. It would likely allow us to consume data from various sources, without requiring to load it as a whole beforehand as it often is the case now.

Beyond the optimized resources consumption it would offer, we could also extend our existing APIs to have the ability to consume Streams. We believe beyond implementing and exposing streams to JavaScript user scripts, we would need to make sure our underlying API remains usable and intuitive from the Go side too to facilitate adoption from modules and extensions.

Problem space

  • https://github.com/grafana/k6/issues/2977 would probably be a good first candidate to return a stream to underlying file data, as opposed to reading it as a whole.
  • https://github.com/grafana/k6/issues/592

Execution

### Tasks
- [ ] https://github.com/grafana/k6/issues/3666
- [ ] https://github.com/grafana/k6/issues/3659
- [x] Expose a `streams` module to k6 scripts
- [ ] #3576
- [ ] (maybe) Implement the `ReadableStreamBYOBReader`

oleiade avatar Mar 13 '23 13:03 oleiade