Yielding rows lazily

[bts]

Hi Antti,

Thanks for this library. I was wondering if there are any plans to yield rows lazily? I see in PgResultSet that iterator() is currently backed by an eager List<Row>.

Thanks! Brian

[bts]

Could we pass stream at https://github.com/alaisi/postgres-async-driver/blob/92300b6f03ceefc186cb8cfbe29243814af89960/src/main/java/com/github/pgasync/impl/PgConnection.java#L79 into a new ResultSet impl that takes a Stream<Message> instead of List<Row>? In that ResultSet, size() and updatedRows() would either have to block on completion of the streaming, or return some sort of future instead.

[alaisi]

Hi,

interesting idea.

The library has no blocking methods, but a callback-based API (or future or promise) could be added. NettyPgProtocolStream currently passes the backend messages to PgConnection only after ReadyForQuery message is received, but there is no limitation why the rows could not be processed as soon as each individual row is available. The API could probably look something like Rx Observable with onNext/onComplete/onError (I'm planning on creating an Observable wrapper for the library as a separate project).

Note that the implementation would not truly be lazy as the postgres backend always sends the entire result set (and cancelling is expensive as it opens a new physical connection), but the library could provide a Stream- or Observable-like API for consuming individual rows as early as possible.

I'll take a look. Pull requests are also welcome. :)

[bts]

Cool.

I'm a little tied-up with other work at the moment (and I don't write much Java) so I was thinking about tackling the smaller task of adding a manifold interface (as an alternative to the core.async interface) to postgres.async first, if you'd be interested in having that in the library? The two interfaces could live side-by-side. Here's manifold's rationale. If you haven't played with it yet, the new version of aleph has switched to a manifold interface and it's really nice to use.

[alaisi]

Manifold looks interesting. That interface could be included in postgres.async in a separate namespace and with provided scope manifold leiningen dependency.

[hadronzoo]

:+1: for a manifold interface

[sdeleuze]

Hi, I am one of the developer of the Reactive support of Spring Framework. As part of the spring-reactive project, I am working on a spring-reactive-playground experiment which is about integrating MongoDB, Couchebase and PostgreSQL async drivers in order to have an example of a Reactive application end to end (web to database).

Thanks a lot for your work on postgres-async-driver. I think being able to multiple rows in a non-blocking way with a stream would be really useful. I don't think we need manifold for that, exposing Rxjava Observable<Row> or Reactive Streams Publisher<Row> is perfect for this use case.

mongo-java-driver-reactivestreams, mongo-java-driver-rx or Couchbase AsyncN1qlQueryResult are good example of such API/implementation.

Any thoughts?

[alaisi]

Hi, release 0.7 (just now syncing to maven central) returns query results as Rx Observables and can emit individual rows to the subscriber before the entire result set is received.

See queryRows: https://github.com/alaisi/postgres-async-driver/blob/master/src/main/java/com/github/pgasync/QueryExecutor.java

@sdeleuze: This probably fits your use case perfectly?

[sdeleuze]

@alaisi Awesome, perfect match cf. https://github.com/sdeleuze/spring-reactive-playground/commit/b90efcb363931ba8cc025f81a03be2af65eb00bf

Thanks!