Add list of related projects to readme

[saulshanabrook]

We should make a list in the readme of other related work and how they compare

• CRDTs:
  • Lumino datastore
  • automerge
  • Microsoft Fluid framework
• Other approaches:
  • https://github.com/jaredly/local-first (Jared Forsyth has a few interesting posts looking at libraries out there: https://jaredforsyth.com/)
  • William Stein's implementation in cocalc

[saulshanabrook]

See convergence as well! https://github.com/jupyterlab/jupyterlab/issues/5382#issuecomment-632116871

[saulshanabrook]

Logux: https://logux.io/ QuicTransport (websockets alternative): https://web.dev/quictransport/ CRDT in redux with automerge: https://github.com/DevResults/cevitxe ORM in redux: https://github.com/redux-orm/redux-orm

[saulshanabrook]

Recoil JS (best practices for state management with hooks in React): https://recoiljs.org/

[saulshanabrook]

• Cocalc uses https://github.com/google/diff-match-patch instead of CRDTs
• distributed databases
  • https://github.com/pouchdb/pouchdb
  • https://github.com/citusdata/citus
  • https://materialize.io/docs/
  • https://github.com/biokoda/actordb
  • https://rethinkdb.com/faq
• JupyterLab sharing on dat: https://github.com/deathbeds/jupyterlab-dat/blob/master/packages/jupyterlab-dat-notebook/src/model.ts
• https://github.com/share/sharedb

[williamstein]

Cocalc uses https://github.com/google/diff-match-patch instead of CRDTs

To expand on this a little, in CoCalc, we do use https://github.com/google/diff-match-patch for the plain text part of sync (but something else that is custom for structured objects). Also, we do not use the differential synchronization algorithm by Neil (who made diff-match-patch); we did use differential sync for a couple of years, but it had major shortcomings for us, so I came up with something much better, which is what we use now. See https://blog.cocalc.com/2018/10/11/collaborative-editing.html for more details.

We use PostgreSQL for longterm storage of sync-related data, but we do not actively use a database for anything interesting related to sync. In particular, we no longer use database triggers for sync, and instead make the job of broadcasting changes to all clients the responsibility of a websocket server that is running in the same Docker container as the Jupyter kernel.

[jasongrout]

We use PostgreSQL for longterm storage of sync-related data, but we do not actively use a database for anything interesting related to sync. In particular, we no longer use database triggers for sync, and instead make the job of broadcasting changes to all clients the responsibility of a websocket server that is running in the same Docker container as the Jupyter kernel.

Thanks for the update, that's interesting. Is the websocket server also responsible for logging the sync-related data to the database, then?

[williamstein]

Yes it periodically writes patches to the database for more longterm storage only. Some synced tables - eg for ipywidgets - just skip this step.

On Tue, Jun 23, 2020, 1:01 PM Jason Grout [email protected] wrote:

We use PostgreSQL for longterm storage of sync-related data, but we do not actively use a database for anything interesting related to sync. In particular, we no longer use database triggers for sync, and instead make the job of broadcasting changes to all clients the responsibility of a websocket server that is running in the same Docker container as the Jupyter kernel.

Thanks for the update, that's interesting. Is the websocket server also responsible for logging the sync-related data to the database, then?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jupyterlab/rtc/issues/15#issuecomment-648387481, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJXS5WI4WI7SEACHWAWFBLRYEC2TANCNFSM4NGDFGZQ .

[saulshanabrook]

Had another look at graphql to see if it would be able to be used, but they haven't formalized some things we need, like subscriptions primarily. Also worried that it seems to be mostly a dumping ground for internal FB things as they bring them out. Not much other company contributions it seems like on the main focus. Might be a good candidate for integration with but not building around it as source of truth. This is another relevant issue: https://github.com/graphql/graphql-spec/pull/733

[blink1073]

Relevant HN thread: https://news.ycombinator.com/item?id=23802208

[benjie]

Apologies in advance for replying out of context (and delayed), I saw the link in my GraphQL Spec PR and was curious.

Also worried that it seems to be mostly a dumping ground for internal FB things as they bring them out. Not much other company contributions it seems like on the main focus.

I don't think this is a fair assessment of GraphQL. If you look at the recent contributors to the working groups and the spec itself you'll find it's a large mix of various companies and independent contributors. The GraphQL Foundation itself is independent of Facebook and is headed by Lee Byron who no longer works for Facebook. If you'd like to join a GraphQL WG call, the instructions are in the agenda file and I encourage you to do so if you'd like to help us evolve GraphQL.

they haven't formalized some things we need, like subscriptions primarily

Subscriptions are in the GraphQL specification and have been for a while, what specifically do you feel is insufficiently formalized here?

This is another relevant issue: graphql/graphql-spec#733

I'd love feedback on this; if there's something that this enables, or could enable if it were tweaked a bit, now is the time to raise it! At the moment it's the lead option for input polymorphism in GraphQL. Please discuss via that issue if you have feedback (or reach out to me directly if you prefer, you can find me on the Graphile Discord or GraphQL Slack as @benjie).

[saulshanabrook]

Hey @benjie thank you for responding so thoroughly!

I apologize for my rough/incorrect language above. I don't know the community very well, and I was sharing an off the cuff impression not informed by much experience. It's great to know there is active movement from outside Facebook on the spec.

If you'd like to join a GraphQL WG call, the instructions are in the agenda file and I encourage you to do so if you'd like to help us evolve GraphQL.

Thank you for the inviting me! I will try to attend the next meeting.

Subscriptions are in the GraphQL specification and have been for a while, what specifically do you feel is insufficiently formalized here?

Thank you for that pointer. I had just seen this RFC document that said "working draft" so I assumed they weren't standardized yet. I will take a closer look at that.

I'd love feedback on this; if there's something that this enables, or could enable if it were tweaked a bit, now is the time to raise it!

We are still at an early stage here, but I think what might be helpful is for me to do a deeper assessment of what is currently missing from the GraphQL spec to support our desired use case.

I linked out to that issue, because I had been thinking some of our fields are represented as tagged unions, and act more like state machines. So it was nice to see some movement there on the GraphQL side for this kind of typing. We have the flexibility currently of falling back to any Typescript type definition to describe JSON data, so we use that to get the typed unions currently. But having built in support at the data model level for tagged unions would be a nice addition, so they don't have to be nested under a status field.

[benjie]

Thank you for that pointer. I had just seen this RFC document that said "working draft" so I assumed they weren't standardized yet. I will take a closer look at that.

Right; that's a historic document - thanks for sharing this, I've raised a PR to add a note to the top of the document to try and avoid future confusion: https://github.com/graphql/graphql-spec/pull/754

For the latest specs; please view: https://spec.graphql.org/ - note that many people work off of the Working Draft, and that another GraphQL Spec release is imminent.

some of our fields are represented as tagged unions, and act more like state machines

I'm glad to hear my spec proposal is solving a pain point; however the tagged types as currently specified only allow a single key to be set, so it would result in something more like the following rather than what you have linked to; would that be acceptable or is it a deal breaker?

| { requested: true }
| { inProgress: true }
| {
    ok: {
      execution_count: number | null;
      result: null | {
        data: ReadonlyJSONObject;
        metadata: ReadonlyJSONObject;
      };
    };
  }
| { abort: true }
| {
    error: {
      ename: string;
      evalue: string;
      traceback: Array<string>;
    };
  }

(I've put true where there are no further keys, but you could use objects there too.)

[blink1073]

Another relevant HN thread: https://news.ycombinator.com/item?id=24176455

[saulshanabrook]

I started looking into some more of the ecosystem around GraphQL optimistic UI (i.e. updating local state without having to wait for server response). I was looking for a way to say "Hey if you write your mutations and schemas in this standard way, then different frontends will know how to optimistically update them!" but I didn't find that to be the case. Instead, each client seems to have it's own custom syntax for optimistic updates, which you have to hand code per field:

• Offix a way to augment Apollo to provide offline operations
• Amplify Datastore: An open source AWS project to provide basically exactly what we want! Too bad it's all tied to AWS
• Apollo optimistic UI updates a way to edit their local cache after sending mutation
• urql an alternative to Apollo that also provides optimistic updates... Didn't look too deep but looks nice! Curious to hear more about why they built their own tool
• overmind a state store which you can use to put your graphql state and then optimistically update

I am curious with Graphile, since there is so much explicit domain information in the database schemas, whether you have explored using that to create a framework for optimistic client side updates that takes care of that? @benjie Or do you just see people hand rolling little bits of optimistic updates for different part of the DB?

[benjie]

Alas I've not had time to build anything like that yet; mostly people hand-roll (or don't even bother because PostGraphile's so fast the optimistic updates are only necessary for things like "favouriting" something where the feedback needs to be instant; normally response times are dominated by network latency, which is only a few hundred milliseconds even from one side of the planet to the other).

[blink1073]

Microsoft just released: "The Fluid Framework is a TypeScript library for building distributed, real-time collaborative web applications.". https://github.com/microsoft/FluidFramework. https://news.ycombinator.com/item?id=24417482

[echarles]

Microsoft just released: "The Fluid Framework is a TypeScript library for building distributed, real-time collaborative web applications.". https://github.com/microsoft/FluidFramework. https://news.ycombinator.com/item?id=24417482

thx @blink1073 We were already tracking this https://jupyter-rtc.readthedocs.io/en/latest/about/libraries.html#microsoft-fluid

Good to know that it is released. I was wondering if that can be used outside of Office365? Now we will know, as soon as the Azure static website host will be "unthrottled" ...

[saulshanabrook]

I was wondering if that can be used outside of Office365? Now we will know, as soon as the Azure static website host will be "unthrottled"

Yeah I believe it doesn't require any MS services:

Yes. Fluid Framework is designed to stand alone. It has no dependencies on other services.

---

From a first glance it looks very complete technically, as one would imagine as the bedrock for an office product. And it might be interesting to see what a Jupyter experience would be like on top of this. And there is likely some great insights in it to learn from!

But as a community, probably not the kind of thing we want to make a part of the core stack right now b/c of single stakeholder control of project...