Understanding Norm's Concurrency Story

[winrid]

Hello,

I'm trying to understand norm's concurrency/async story. I'm new to Nim as well. It looks like Norm is blocking? So if you use it with Jester a few slow queries will block the entire API. Is that correct?

Thanks (cool library btw)

[moigagoo]

Norm doesn't have a concurrency story of its own. Connecting to a DB is blocking because it's io. You can alleviate this by writing async code which let's you do something else while waiting for io or using a connection pool which puts the burden of concurrent DB access on the DB server.

[winrid]

Connecting to a DB doesn't have to be blocking. Norm does have a concurrency story - as you just said - it's blocking :)

If you use the examples, or follow the docs, and use this library with any major async nim framework, the performance would be terrible/unpredictable... This is why Node performs well since all the IO is async/deferred on the event loop. Modern Java also uses async DB access (Vertx, Quarkus, etc). Rust also has sqlx/diesel which is completely async.

[moigagoo]

Async doesn't mean not blocking though :-) Async just means you can tell the program to switch to a different task while the current one is blocked because it's waiting for the io.

If you use Norm with Prologue, which is an async web framework, your controllers will run concurrently: while one is waiting for the DB transaction to finish, the other can parse the HTTP request or so do computations or send response.

[winrid]

If Norm isn't using async IO there's no magic that Prologue can do.

I you only have 4 network IO threads and each thread is waiting on a query, the app will appear non responsive, unless you offload the db queries to a separate threadpool, but I'm not sure why you'd want to do that yourself, when that's kinda the whole point of async syntax sugar.

Examples of other libraries:

• https://github.com/SSPkrolik/nimongo#usage-of-synchronous-client
• https://github.com/treeform/pg#concurrency

[moigagoo]

Sorry, you are right, I'm wrong.

Norm relies on ndb package for the actual DB interaction. So Norm's DbConn type is actually ndb's.

Since ndb is a blocking package, so is Norm.

[winrid]

No worries, thanks for replying :)

I wonder if ndb would have to be forked to change this. Then we could use { .multisync . } which would allow the simple blocking usage we show today in examples, but also allow async usage for performance oriented scenarios. I think sqlite would always be blocking unless the library could use some kind of threadpool behind the scenes or something. This is what Drogon does for its async ORM - PG/MySQL have native async implementations but the sqlite driver uses a threadpool. The vertx sqlite driver does a similar thing, although with Java's new virtual threads it will be interesting to see what happens with that.

Probably a good place to start would be benchmarks so we can track our progress. Let me know if you ever want to start down this path and if you want help!

[PhilippMDoerner]

In this regard it should be noted that ndb is also currently what's holding norm back for compatibility with nim 2.0 (Currently nim 1.9.X, the current devel branch of the compiler) (As a sidenote for winrid just to be complete: I already made a fork around a week ago for myself and my own project nimstoryfont that makes use of norm. So the potential for it is there).

Personally I think it's interesting, though I'm approaching this more from the sqlite angle than the postgres one (because dealing with postgres containers for testing is fiddly and annoying).

I'm not sure why sqlite would need to be blocking though. You already linked treeforms Postgres driver that shows it's really not that difficult. He just implemented the public API through the async pragma, should we play around with multisync even that would become unnecessary. A few of the helper procs that are built on top of the more base-level API (like selectManyFromOne etc.) will require some tweaking, but even there it looks like it could be easy.

It also could still be beneficial to do this for sqlite as well, because there are some bits and tweaks you can apply to sqlite to make it more capable of dealing with concurrency (Write Ahead Log as well as some options to reduce locking etc., not that I've done it I just stumbled over the options here and there while browsing through the sqlite docs for my own purposes). Though in those cases you're really putting your sqlite DB under some stress.

[winrid]

I'm not sure why sqlite would need to be blocking though.

Because sqlite itself is AFAIK, and everything is just a wrapper around that library. If you look at treeform's library it's using the nim stdlib db_postgres module, which is an async client. The nim stdlib db_sqlite does not use async io. That's why people do stuff like this: https://github.com/drogonframework/drogon/blob/master/orm_lib/src/sqlite3_impl/Sqlite3Connection.cc#L103

EDIT - I just looked again and I don't see anything about async in the db_postgres module.. so maybe there is something about nim I don't understand as to how treeform's library can be async but the underlying driver is not.

Although sqlite could probably just be blocking, since it's mostly used for testing.

Though in those cases you're really putting your sqlite DB under some stress.

FWIW I've actually used sqlite in WAL mode for 24+ hr cron jobs, with tens of thousands of multi-kb events a second, and it never broke a sweat. sqlite is awesome. :)

[winrid]

Ah, I found how treeform's library works. The pg client in the nim stdlib does indeed provide an async API:

• https://nim-lang.org/docs/postgres.html#pqsendQuery%2CPPGconn%2Ccstring
• https://www.postgresql.org/docs/current/libpq-async.html

So at least supporting PG is easy.

[moigagoo]

@winrid @PhilippMDoerner sorry for the slow response guys. I've been really overwhelmed at work lately.

Given that ndb looks abandoned, doesn't support async, and doesn't support Nim 2.0, I think the best course of action would be:

1. Short-term: fork ndb and patch it. This way, Norm will be able to get upgraded for Nim 2.0.
2. Long-term: implement low-level DB connection primitives and types in Norm. This means either accurately copying the code from ndb to Norm or reimplementing DbConn and all the related procs from scratch.

I don't want to the the bottleneck in these efforts and I'll be happy to grant admin permissions to this repo to @winrid and @PhilippMDoerner if you are interested in getting involved.

[PhilippMDoerner]

Happy to accept the admin permission there.

I disagree on parts of the long-term strategy though. I would like to keep ndb functionality separate from norm, because I would like others to still be able to build on top of it starting from nim 2.0 and I'm not seeing an immediate benefit of having them in the same repo. I also think it might help with separation of concerns.

Therefore I suggest the following steps primarily regarding the ndb situation:

• Publish my ndb fork as a nimble package, either while it's my own repository or we can move ownership to you moigagoo so that norm and ndb are under the same umbrella (low effort)
• Update norm to use the new ndb fork package (low effort)
• Reimplement or Update ndb to provide async capabilities (high effort)
• Implement async capabilities in norm (high effort)

What are your thoughts?

Edit: I maybe should note that I'm not sure I'll find the time to help all that much with the postgres async implementation, as I'm also jumping around in other parts of the nim ecosystem and all of that is eating at my available time, particularly as I don't use postgres much myself. I'm happy to help oversee/review the efforts though.

[winrid]

Assuming we go with @PhilippMDoerner suggestions, I'd be happy to help with the high effort stuff. Although I wonder how hard it is to copy/reuse some of treeform's work for postgres.

[PhilippMDoerner]

@winrid I'll be honest, that was me wild-guessing the effort. I know that the effort for the fist two is trivial (for me) because I know I have the code already and I know what to do there.

For postgres we're running into the issue that we can't exactly use std/db_postgres because they represent NULL with "" in the nim objects, that was precisely the issue that ndb was made to solve. I don't have that much experience with C-bindings and looking at C-api's in general tends to make me annoyed with the API.

Similarly I haven't done a ton of async-programming in nim so while I can write it, I'm not overly comfortable in it and it still takes a decent amount of time, so I estimate those two higher. Could very well be that both of those are comparatively simple, they might just not be for me ;-)

[PhilippMDoerner]

I've started the process. I'm not the greatest of namers, so I just named it lowdb for now. If anyone has a better name suggestion, throw it out, for now I'll prepare file changes etc. under that name

https://github.com/PhilippMDoerner/lowdb/issues/1#issue-1578443488

@moigagoo I'd still like answers to the following questions:

1. Do you have a preference if we have lowdb(name pending) under my name or yours? I'm leaning towards transferring ownership of my fork to you solely because it's mainly used with norm and so we have it under one umbrella, but I'm not having particularly strong feelings in one direction or another
2. Are you okay with my long-term desire to maintain lowdb(name pending) separate from norm ?

[moigagoo]

@PhilippMDoerner I'm totally fine with lowdb being your independent project under your name. We can create an organization on GitHub and put both Norm and lowdb (and Norman, and whatever emerges in the future) in it.

Also, lowdb is a fine name IMHO 😊

[moigagoo]

Back in the day, I used to have a vision that Norm would be a part of a larger project for web development. Like, Norman would be the scaffolder fornit, Norm would be the ORM, Jester or Prologue would be the server, and Karax the frontend. I even had a name for it: Normandy.

Could this be the time we try and make it happen?

[PhilippMDoerner]

I'm very much in favour of making a github organization to put them under the same umbrella, mostly because in my mind they just "belong" together (well I haven't seen anything else use ndb). I'll be honest though that I have no proper idea what that means because I never dealt with those :smile:

As for Normandy: If I'm understanding this right the vision is Django like - an opinionated, battery-included framework. Maybe? If we went down that route the tech-stack I personally see would be Prologue - Norm - Karax (for outputting HTML) / Jsony (for outputting JSON/parsing JSON). I don't see Jester there because afaik with dom leaving there's just no maintainer there and I wouldn't want to start providing support or fixes for an httpserver, I just don't have the time.

Basically I can see the vision, I'm just uncertain about having the manpower to support the vision. Though even without committing to it, we can none the less work towards it of course. First with stabilizing norm for nim 2.0, later with ways to facilitate usage, plugins or just provide a cli-tool (that could be easily integrated into norman) that can set-up a tech-stack project for you.

[Clonkk]

Integrating with https://github.com/HapticX/happyx might be worth too :)