js-sdk icon indicating copy to clipboard operation
js-sdk copied to clipboard

feat: add streamedListObjects for unlimited object retrieval

Open daniel-jonathan opened this issue 4 months ago • 4 comments

Summary

Adds streamedListObjects method for retrieving unlimited objects via the streaming API. Node.js-only implementation with NDJSON parsing, error handling, and automatic resource cleanup.

Requires OpenFGA server v1.2.0+

Fixes #236

Changes

  • Add streaming.ts with NDJSON parser for Node.js streams
  • Add StreamedListObjectsResponse interface to apiModel.ts
  • Add OpenFgaClient.streamedListObjects() async generator method
  • Add createStreamingRequestFunction to common.ts for streaming requests
  • Export parseNDJSONStream from index.ts
  • Add streaming tests and examples
  • Update CHANGELOG with feature and version requirement

Implementation

  • Streams beyond 1000-object limit
  • Proper error propagation through async iterators
  • Automatic stream cleanup on early exit
  • Supports Readable, AsyncIterable, string, Buffer, and Uint8Array inputs
  • Telemetry integration via API layer

Usage

const fgaClient = new OpenFgaClient({ 
  apiUrl: "http://localhost:8080", 
  storeId: "01H0H015178Y2V4CX10C2KGHF4" 
});

for await (const response of fgaClient.streamedListObjects({
  user: "user:anne",
  relation: "owner",
  type: "document"
})) {
  console.log(response.object);
}

Testing

All 153 tests passing (10/10 suites)

  • 17 streaming tests covering parsing, errors, cleanup, edge cases
  • 90.8% coverage on streaming.ts
  • Live verified: 3-object and 2000-object streaming

Related

  • Fixes #236
  • Parent issue: openfga/sdk-generator#76
  • Related PR: openfga/sdk-generator#654

daniel-jonathan avatar Oct 28 '25 04:10 daniel-jonathan

[!IMPORTANT]

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This PR adds a new streaming variant of the ListObjects API to the JavaScript SDK. It includes a new streamedListObjects method that streams NDJSON-formatted results from the server, allowing retrieval of more than 1000 objects. The implementation spans the API layer, client wrapper, streaming parser, and includes comprehensive examples and tests.

Changes

Cohort / File(s) Summary
Core API Layer
src/api.ts
Adds streamedListObjects method across all API interfaces (AxiosParamCreator, Fp, Factory, and OpenFgaApi class) to support streaming requests via createStreamingRequestFunction.
API Model Types
src/apiModel.ts
Introduces new StreamedListObjectsResponse interface with object property for streamed response elements.
Client Wrapper
src/client.ts
Adds streamedListObjects method returning AsyncGenerator<StreamedListObjectsResponse> that parses NDJSON stream and yields results.
Streaming Support
src/common.ts, src/streaming.ts, src/index.ts
Implements createStreamingRequestFunction for axios streaming with telemetry; adds parseNDJSONStream utility to parse line-delimited JSON from Node.js Readable streams; exports parseNDJSONStream publicly.
Test Mocks & Coverage
tests/helpers/nocks.ts, tests/streaming.test.ts
Adds streamedListObjects mock helper for NDJSON endpoint responses; comprehensive test suite for NDJSON parser covering chunked data, various input types, and error handling.
Examples
example/streamed-list-objects/README.md, example/streamed-list-objects/model.json, example/streamed-list-objects/streamedListObjects.mjs
Complete example demonstrating streaming 2000 tuples, comparing streamed vs. standard endpoints, with access control model definition.
Local Example
example/streamed-list-objects-local/README.md, example/streamed-list-objects-local/streamedListObjectsLocal.mjs
Simplified local example showing streamedListObjects with user-relation queries and store lifecycle.
Documentation
CHANGELOG.md
Documents new streamedListObjects feature for Node.js environments with NDJSON parsing support.

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant Client as OpenFgaClient
    participant API as OpenFgaApi
    participant HTTP as axios
    participant Server as OpenFGA Server
    participant Parser as parseNDJSONStream

    User->>Client: streamedListObjects(request)
    Client->>API: streamedListObjects()
    API->>HTTP: GET /stores/{id}/streamed-list-objects<br/>(responseType: stream, telemetry)
    HTTP->>Server: Request with streaming
    Server-->>HTTP: NDJSON stream (line-delimited JSON)
    HTTP-->>API: Response stream
    API-->>Client: Response.data (stream)
    Client->>Parser: parseNDJSONStream(stream)
    
    loop For each line in stream
        Parser->>Parser: UTF-8 decode, buffer, split on \\n
        Parser->>Parser: Parse JSON object
        Parser-->>Client: yield StreamedListObjectsResponse
    end
    
    Client-->>User: AsyncGenerator<StreamedListObjectsResponse>
    User->>User: Iterate and process results

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~35 minutes

  • src/streaming.ts: The NDJSON parser handles multiple input modalities (Readable, Buffer, string, async iterable) with UTF-8 decoding, line buffering, and error handling. Verify edge cases around partial UTF-8 sequences, decoder state management, and listener cleanup.
  • src/common.ts: New createStreamingRequestFunction mirrors existing request flow but with streaming response. Verify telemetry capture (fromRequest/fromResponse) and histogram emission work correctly with streaming.
  • tests/streaming.test.ts: Extensive test suite validates chunked data, error scenarios, and various input types. Spot-check for completeness of edge cases (e.g., incomplete UTF-8 at buffer boundary).
  • Example files: Ensure model.json schema is valid and examples execute correctly with 2000 tuples to verify streaming advantage over standard pagination.

Possibly related issues

  • openfga/sdk-generator#76: Directly addresses the "Support streaming endpoints (StreamedListObjects)" work item with implementation of streaming API, NDJSON parser, and async generator interface.

Suggested reviewers

  • jimmyjames

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (4 passed)
Check name Status Explanation
Title Check ✅ Passed The pull request title "feat: add streamedListObjects for unlimited object retrieval" clearly and directly describes the primary change in the changeset. The title references the new streamedListObjects feature and explains its key benefit (unlimited object retrieval beyond the standard 1000-object limit). The title is specific, concise, and uses appropriate conventional prefix (feat:) without vague language or excessive noise. A developer reviewing the commit history would immediately understand that this PR adds streaming capability to retrieve objects without the standard limitation.
Linked Issues Check ✅ Passed The pull request successfully implements both primary requirements from linked issue #236. First, SDK methods to call the StreamedListObjects API have been added across all layers: the api.ts exposes the streamedListObjects method through OpenFgaApi, OpenFgaApiFp, OpenFgaApiFactory, and OpenFgaApiAxiosParamCreator; the client.ts provides OpenFgaClient.streamedListObjects() as an AsyncGenerator interface for consumers. Second, the implementation aligns with the Python SDK's async-generator pattern as noted in the PR objectives. The code includes supporting infrastructure (parseNDJSONStream for NDJSON parsing, createStreamingRequestFunction for streaming requests, StreamedListObjectsResponse model), comprehensive examples demonstrating usage, and test coverage validating the streaming behavior.
Out of Scope Changes Check ✅ Passed All changes in the pull request are directly scoped to the objectives of adding StreamedListObjects support. The modifications span necessary layers: API contract changes (api.ts), client-facing implementation (client.ts), data models (apiModel.ts), streaming utilities (streaming.ts), request infrastructure (common.ts), public exports (index.ts), test infrastructure (tests/helpers/nocks.ts, tests/streaming.test.ts), usage examples (example/ directory), and documentation (CHANGELOG.md). No unrelated bug fixes, general refactoring, or tangential features are present. The examples and supporting files are integral to demonstrating the feature rather than out-of-scope additions.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Oct 28 '25 04:10 coderabbitai[bot]

Codecov Report

:x: Patch coverage is 90.54054% with 14 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 89.56%. Comparing base (847b9ff) to head (0d47948).

Files with missing lines Patch % Lines
streaming.ts 90.21% 7 Missing and 2 partials :warning:
common.ts 86.36% 1 Missing and 2 partials :warning:
api.ts 90.90% 2 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #280      +/-   ##
==========================================
+ Coverage   89.45%   89.56%   +0.11%     
==========================================
  Files          24       25       +1     
  Lines        1299     1447     +148     
  Branches      234      266      +32     
==========================================
+ Hits         1162     1296     +134     
- Misses         81       91      +10     
- Partials       56       60       +4     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov-commenter avatar Oct 28 '25 04:10 codecov-commenter

Can you also review @SoulPancake's PR here: https://github.com/openfga/go-sdk/pull/252

Ideally both would have the same semantics, example, tests, config options, README, etc..

rhamzeh avatar Oct 29 '25 16:10 rhamzeh

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Added@​openfga/​syntax-transformer@​0.2.09910010082100

View full report

socket-security[bot] avatar Nov 03 '25 17:11 socket-security[bot]