on windows, server crashes after adding a source

[bcolloran]

Describe the bug on windows in dev mode after adding a source, the following error is repeated over and over:

Script "npm run dev:backend" exited with code 3221226505 null
Starting "npm run dev:backend"

> @rilldata/[email protected] dev:backend
> ts-node-dev --project tsconfig.node.json -- src/server/server.ts

[INFO] 12:49:11 ts-node-dev ver. 1.1.8 (using ts-node ver. 9.1.1, typescript ver. 4.6.3)
Server started at http://localhost:8080
Script "npm run dev:backend" exited with code 3221226505 null
Starting "npm run dev:backend"

> @rilldata/[email protected] dev:backend
> ts-node-dev --project tsconfig.node.json -- src/server/server.ts

[INFO] 12:49:21 ts-node-dev ver. 1.1.8 (using ts-node ver. 9.1.1, typescript ver. 4.6.3)
Server started at http://localhost:8080
Script "npm run dev:backend" exited with code 3221226505 null
Starting "npm run dev:backend"

> @rilldata/[email protected] dev:backend
> ts-node-dev --project tsconfig.node.json -- src/server/server.ts

[INFO] 12:49:30 ts-node-dev ver. 1.1.8 (using ts-node ver. 9.1.1, typescript ver. 4.6.3)
Server started at http://localhost:8080
Script "npm run dev:backend" exited with code 3221226505 null
Starting "npm run dev:backend"

> @rilldata/[email protected] dev:backend
> ts-node-dev --project tsconfig.node.json -- src/server/server.ts

[INFO] 12:49:38 ts-node-dev ver. 1.1.8 (using ts-node ver. 9.1.1, typescript ver. 4.6.3)
Server started at http://localhost:8080
Script "npm run dev:backend" exited with code 3221226505 null
Starting "npm run dev:backend"

To Reproduce with the changes from PR #764 in place, load a new source by way of the plus icon next to the sources heading in the navigation sidebar. The source loads, and shows up in the UI, but the console begins showing the errors above over but the console begins showing the errors above over and over.

Expected behavior the back end should run without error

Desktop (please complete the following information):

• OS: windows

• version 0.7.0 – e03abf5

[bcolloran]

Note that the binary is affected as well, I have checked versions 0.6 and 0.7, and both show similar behavior. You are able to run the rill start command, the local host server starts, and you can open the page in the browser ; however , when you attempt to load data through the UI the program crashes. Unlike in dev mode it doesn't repeat the error output from above, it just crashes silently-- i.e., the command prompt just shows:

C:\...\rill\binary-testing> rill start
Server started at http://localhost:8080

C:\...\rill\binary-testing>

here is a video of what I see when I download the binary on this page (https://docs.rilldata.com/install/binary), and then run the commands in the command prompt, but omitting the import-source command and instead importing a source using the app UI

https://user-images.githubusercontent.com/2380975/185245369-39d789b1-0578-4709-9e0f-6e339063fc5f.mp4

and here is the browser console contents corresponding to those steps:

Response { type: "basic", url: "http://localhost:8080/api/file/table-upload", redirected: false, status: 200, ok: true, statusText: "OK", headers: Headers, body: ReadableStream, bodyUsed: false }
index.svelte-b3b8ad88.js:4:43317
The connection to ws://localhost:8080/socket.io/?EIO=4&transport=websocket&sid=jwjW5mKlH-oo5FrlAAAC was interrupted while the page was loading. store-root-e652662c.js:1:22049

@AdityaHegde -- I don't really have any idea what's going on here, but it feels like it might be some kind of issue with profiling or sending data back and forth. It looks like at least a couple profiling steps happened successfully, which indicates that the data is successfully imported, and even a couple profiling operations seem to succeed, but then it stops working. please let me know if I can provide any other information that might be useful, thanks

[AdityaHegde]

I was able to pin point the issue to the histogram query. This is an issue with duckdb itself on windows, or the way we are using it.

[AdityaHegde]

We need to check this now with runtime using the CAPI directly. A couple of issues to fix before we verify,

1. Windows executable is not available, so cannot test the exe directly.
2. npm i fails because the script assumes linux based OS.
3. Runtime is not build properly for windows - the windows zip file has nothing in it.

[begelundmuller]

Going to use this issue to write down a a few things on the backend side necessary to support Windows:

• [ ] Add Windows builds in the runtime release workflow
• [ ] Test SQL library is correctly loaded on Windows (code here)
• [ ] (Optional) Add pre-built static libraries for Windows to the go-duckdb driver (to speed up builds)

[hamilton]

@begelundmuller @AdityaHegde what do you both think the lift is to get Rill Developer building for windows, with these investigations in mind? It doesn't look like the issue is Node-related based on what y'all have found

[begelundmuller]

On the Go side, it doesn't feel like more than a few days of work to make things work "on paper", but given how much time we already spent on unexpected bugs, I wouldn't commit to a time estimate.

As far as I recall, @kaspersjo was also experiencing hard-to-debug errors with vercel/pkg on Windows and felt it wasn't worth pursuing until we deprecate Node on the backend.

One thing we should be mindful of about Windows is that it's not a one-time investment. It's a different platform, and given how many syscall-level things we're doing (DuckDB, Calcite native library, file and network IO), it will likely continue to require special treatment compared to Mac/Linux. I worry it becomes a permanent 10-20% tax on dev velocity (at least until our feature set and automated tests stabilize).

[hamilton]

A 10-20% tax on dev velocity feels worth the much larger developer TAM (we've had many, many data people reach out to us telling us Rill Developer doesn't work on Windows), but I agree that we should be mindful of how we invest atm.

[begelundmuller]

Closing since we direct people to WSL support