feat: add streamedListObjects for unlimited object retrieval
Summary
Adds streamedListObjects method for retrieving unlimited objects via the streaming API. Node.js-only implementation with NDJSON parsing, error handling, and automatic resource cleanup.
Requires OpenFGA server v1.2.0+
Fixes #236
Changes
- Add
streaming.tswith NDJSON parser for Node.js streams - Add
StreamedListObjectsResponseinterface toapiModel.ts - Add
OpenFgaClient.streamedListObjects()async generator method - Add
createStreamingRequestFunctiontocommon.tsfor streaming requests - Export
parseNDJSONStreamfromindex.ts - Add streaming tests and examples
- Update CHANGELOG with feature and version requirement
Implementation
- Streams beyond 1000-object limit
- Proper error propagation through async iterators
- Automatic stream cleanup on early exit
- Supports Readable, AsyncIterable, string, Buffer, and Uint8Array inputs
- Telemetry integration via API layer
Usage
const fgaClient = new OpenFgaClient({
apiUrl: "http://localhost:8080",
storeId: "01H0H015178Y2V4CX10C2KGHF4"
});
for await (const response of fgaClient.streamedListObjects({
user: "user:anne",
relation: "owner",
type: "document"
})) {
console.log(response.object);
}
Testing
All 153 tests passing (10/10 suites)
- 17 streaming tests covering parsing, errors, cleanup, edge cases
- 90.8% coverage on streaming.ts
- Live verified: 3-object and 2000-object streaming
Related
- Fixes #236
- Parent issue: openfga/sdk-generator#76
- Related PR: openfga/sdk-generator#654
[!IMPORTANT]
Review skipped
Auto incremental reviews are disabled on this repository.
Please check the settings in the CodeRabbit UI or the
.coderabbit.yamlfile in this repository. To trigger a single review, invoke the@coderabbitai reviewcommand.You can disable this status message by setting the
reviews.review_statustofalsein the CodeRabbit configuration file.
Walkthrough
This PR adds a new streaming variant of the ListObjects API to the JavaScript SDK. It includes a new streamedListObjects method that streams NDJSON-formatted results from the server, allowing retrieval of more than 1000 objects. The implementation spans the API layer, client wrapper, streaming parser, and includes comprehensive examples and tests.
Changes
| Cohort / File(s) | Summary |
|---|---|
Core API Layer src/api.ts |
Adds streamedListObjects method across all API interfaces (AxiosParamCreator, Fp, Factory, and OpenFgaApi class) to support streaming requests via createStreamingRequestFunction. |
API Model Types src/apiModel.ts |
Introduces new StreamedListObjectsResponse interface with object property for streamed response elements. |
Client Wrapper src/client.ts |
Adds streamedListObjects method returning AsyncGenerator<StreamedListObjectsResponse> that parses NDJSON stream and yields results. |
Streaming Support src/common.ts, src/streaming.ts, src/index.ts |
Implements createStreamingRequestFunction for axios streaming with telemetry; adds parseNDJSONStream utility to parse line-delimited JSON from Node.js Readable streams; exports parseNDJSONStream publicly. |
Test Mocks & Coverage tests/helpers/nocks.ts, tests/streaming.test.ts |
Adds streamedListObjects mock helper for NDJSON endpoint responses; comprehensive test suite for NDJSON parser covering chunked data, various input types, and error handling. |
Examples example/streamed-list-objects/README.md, example/streamed-list-objects/model.json, example/streamed-list-objects/streamedListObjects.mjs |
Complete example demonstrating streaming 2000 tuples, comparing streamed vs. standard endpoints, with access control model definition. |
Local Example example/streamed-list-objects-local/README.md, example/streamed-list-objects-local/streamedListObjectsLocal.mjs |
Simplified local example showing streamedListObjects with user-relation queries and store lifecycle. |
Documentation CHANGELOG.md |
Documents new streamedListObjects feature for Node.js environments with NDJSON parsing support. |
Sequence Diagram(s)
sequenceDiagram
actor User
participant Client as OpenFgaClient
participant API as OpenFgaApi
participant HTTP as axios
participant Server as OpenFGA Server
participant Parser as parseNDJSONStream
User->>Client: streamedListObjects(request)
Client->>API: streamedListObjects()
API->>HTTP: GET /stores/{id}/streamed-list-objects<br/>(responseType: stream, telemetry)
HTTP->>Server: Request with streaming
Server-->>HTTP: NDJSON stream (line-delimited JSON)
HTTP-->>API: Response stream
API-->>Client: Response.data (stream)
Client->>Parser: parseNDJSONStream(stream)
loop For each line in stream
Parser->>Parser: UTF-8 decode, buffer, split on \\n
Parser->>Parser: Parse JSON object
Parser-->>Client: yield StreamedListObjectsResponse
end
Client-->>User: AsyncGenerator<StreamedListObjectsResponse>
User->>User: Iterate and process results
Estimated code review effort
🎯 3 (Moderate) | ⏱️ ~35 minutes
-
src/streaming.ts: The NDJSON parser handles multiple input modalities (Readable, Buffer, string, async iterable) with UTF-8 decoding, line buffering, and error handling. Verify edge cases around partial UTF-8 sequences, decoder state management, and listener cleanup. -
src/common.ts: NewcreateStreamingRequestFunctionmirrors existing request flow but with streaming response. Verify telemetry capture (fromRequest/fromResponse) and histogram emission work correctly with streaming. -
tests/streaming.test.ts: Extensive test suite validates chunked data, error scenarios, and various input types. Spot-check for completeness of edge cases (e.g., incomplete UTF-8 at buffer boundary). - Example files: Ensure model.json schema is valid and examples execute correctly with 2000 tuples to verify streaming advantage over standard pagination.
Possibly related issues
- openfga/sdk-generator#76: Directly addresses the "Support streaming endpoints (StreamedListObjects)" work item with implementation of streaming API, NDJSON parser, and async generator interface.
Suggested reviewers
- jimmyjames
Pre-merge checks and finishing touches
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | ⚠️ Warning | Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. | You can run @coderabbitai generate docstrings to improve docstring coverage. |
✅ Passed checks (4 passed)
| Check name | Status | Explanation |
|---|---|---|
| Title Check | ✅ Passed | The pull request title "feat: add streamedListObjects for unlimited object retrieval" clearly and directly describes the primary change in the changeset. The title references the new streamedListObjects feature and explains its key benefit (unlimited object retrieval beyond the standard 1000-object limit). The title is specific, concise, and uses appropriate conventional prefix (feat:) without vague language or excessive noise. A developer reviewing the commit history would immediately understand that this PR adds streaming capability to retrieve objects without the standard limitation. |
| Linked Issues Check | ✅ Passed | The pull request successfully implements both primary requirements from linked issue #236. First, SDK methods to call the StreamedListObjects API have been added across all layers: the api.ts exposes the streamedListObjects method through OpenFgaApi, OpenFgaApiFp, OpenFgaApiFactory, and OpenFgaApiAxiosParamCreator; the client.ts provides OpenFgaClient.streamedListObjects() as an AsyncGenerator interface for consumers. Second, the implementation aligns with the Python SDK's async-generator pattern as noted in the PR objectives. The code includes supporting infrastructure (parseNDJSONStream for NDJSON parsing, createStreamingRequestFunction for streaming requests, StreamedListObjectsResponse model), comprehensive examples demonstrating usage, and test coverage validating the streaming behavior. |
| Out of Scope Changes Check | ✅ Passed | All changes in the pull request are directly scoped to the objectives of adding StreamedListObjects support. The modifications span necessary layers: API contract changes (api.ts), client-facing implementation (client.ts), data models (apiModel.ts), streaming utilities (streaming.ts), request infrastructure (common.ts), public exports (index.ts), test infrastructure (tests/helpers/nocks.ts, tests/streaming.test.ts), usage examples (example/ directory), and documentation (CHANGELOG.md). No unrelated bug fixes, general refactoring, or tangential features are present. The examples and supporting files are integral to demonstrating the feature rather than out-of-scope additions. |
| Description Check | ✅ Passed | Check skipped - CodeRabbit’s high-level summary is enabled. |
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.
Codecov Report
:x: Patch coverage is 90.54054% with 14 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 89.56%. Comparing base (847b9ff) to head (0d47948).
| Files with missing lines | Patch % | Lines |
|---|---|---|
| streaming.ts | 90.21% | 7 Missing and 2 partials :warning: |
| common.ts | 86.36% | 1 Missing and 2 partials :warning: |
| api.ts | 90.90% | 2 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #280 +/- ##
==========================================
+ Coverage 89.45% 89.56% +0.11%
==========================================
Files 24 25 +1
Lines 1299 1447 +148
Branches 234 266 +32
==========================================
+ Hits 1162 1296 +134
- Misses 81 91 +10
- Partials 56 60 +4
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
- :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
Can you also review @SoulPancake's PR here: https://github.com/openfga/go-sdk/pull/252
Ideally both would have the same semantics, example, tests, config options, README, etc..
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
| Diff | Package | Supply Chain Security |
Vulnerability | Quality | Maintenance | License |
|---|---|---|---|---|---|---|
| @openfga/syntax-transformer@0.2.0 |