golem icon indicating copy to clipboard operation
golem copied to clipboard

First-Class Windows Development Environment

Open jdegoes opened this issue 11 months ago • 28 comments

In the Golem project currently, Windows is not a first-class development experience. The problems are many and range from:

  1. How Golem is built and tested
  2. Which Rust libraries are used, and to what degree they support Windows
  3. Random scripts and tools that may not support Windows fully or at all

This ticket is to enhance Golem and its dependencies so that developers who are using Windows may have a first-class development experience, without having to install Windows Subsystem for Linux (WSL).

When completed, a clean checkout of the Golem code base will:

  1. Fully build with clear and simple instructions for Windows developers
  2. Reliably pass all tests locally without requiring docker or WSL
  3. Require no more setup for development than on Linux or Mac
  4. All Golem binaries (including single executable) run without issue on Windows

Moreover, Github-based CI must be modified to automatically run tests on Windows to ensure no future regressions, so that Windows developers have an ongoing guarantee of support for development.

NOTE: It is expected that as part of this work, you will have to submit to other repositories that are neither owned nor managed by Golem. These could include, for example, wasmtime, sozu-proxy & poule, etc. The actual changes required in the Golem repository should be relatively minimal.

jdegoes avatar Dec 24 '24 17:12 jdegoes

/bounty $5000

jdegoes avatar Dec 24 '24 17:12 jdegoes

💎 $5,000 bounty • Golem Cloud

Steps to solve:

  1. Start working: Comment /attempt #1210 with your implementation plan
  2. Submit work: Create a pull request including /claim #1210 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

❗ Important guidelines:

  • To claim a bounty, you need to provide a short demo video of your changes in your pull request
  • If anything is unclear, ask for clarification before starting as this will help avoid potential rework
  • Low quality AI PRs will not receive review and will be closed
  • Do not ask to be assigned unless you've contributed before

Thank you for contributing to golemcloud/golem!

Attempt Started (UTC) Solution Actions
🟢 @abhishek12201 Mar 08, 2025, 08:31:19 AM WIP
🟢 @mrsbeep Oct 09, 2025, 09:00:26 PM WIP
🟢 @vishalpatil1899 Apr 13, 2025, 10:26:51 AM WIP
🟢 @Mayank77maruti Feb 14, 2025, 08:32:38 AM WIP
🟢 @SamarthShukla17 Sep 15, 2025, 11:57:13 PM WIP
🟢 @chethanuk Aug 16, 2025, 09:41:49 AM WIP
🟢 @naaa760 Jan 18, 2025, 05:44:05 AM WIP
🟢 @dohrisalim Aug 19, 2025, 02:54:19 PM WIP
🟢 @Rutik7066 Mar 20, 2025, 06:55:32 AM WIP
🟢 @uurl Dec 25, 2024, 12:00:47 AM WIP
🟢 @varshith257 Dec 25, 2024, 01:03:54 AM WIP
🟢 @laz-001 May 26, 2025, 07:00:20 PM WIP
🟢 @parvit Apr 26, 2025, 11:12:38 PM WIP
🟢 @zelosleone Dec 28, 2024, 04:34:31 PM WIP

algora-pbc[bot] avatar Dec 24 '24 17:12 algora-pbc[bot]

/attempt #1210

Options

uurl avatar Dec 25 '24 00:12 uurl

/attempt #1210

Algora profile Completed bounties Tech Active attempts Options
@varshith257 25 bounties from 11 projects
Scala, Rust,
TypeScript & more
Cancel attempt

varshith257 avatar Dec 25 '24 01:12 varshith257

Following up with sozu maintainers closely to fix it in upstream

varshith257 avatar Dec 27 '24 02:12 varshith257

Truth to be told the major problem is wasmtime package versioning making it hard to build-test, linking version is extremely hard to pass unless default features of wasmtime is completely disabled, which could make it unable to connect on HTTP via python. But there are also at least 4 years old crates like poule that is not updated recently, I am pretty sure some golem repo code could be changed to fit the newer/more maintained crates.

zelosleone avatar Dec 27 '24 14:12 zelosleone

@zelosleone The issue of conflicting wasm appearing in both Windows and Linux after the merge of #1200. Before it worked fine without any conflicts of wasm versions. The problem stems from wasm-wave likely pulling older version of wasm as its dependency

varshith257 avatar Dec 28 '24 07:12 varshith257

@zelosleone The issue of conflicting wasm appearing in both Windows and Linux after the merge of #1200. Before it worked fine without any conflicts of wasm versions. The problem stems from wasm-wave likely pulling older version of wasm as its dependency

Nope, its the wit-parser that pulls the old versioning, which breaks the wit module of the golem if upgraded. 0.6.0 is p old for wit-parser crate since it went more than 5 updates from that time, so golem code needs to change and be tested that "Yeah, this proves upgrading it didnt break anything after we updated the code and it works like old"

zelosleone avatar Dec 28 '24 07:12 zelosleone

/attempt #1210

Options

zelosleone avatar Dec 28 '24 16:12 zelosleone

@zelosleone The wit parser conflict, I think not under consideration of this issue. It happening with other os linux too not sure what about Mac though.

I have clearly discussed with jdegoes and vigoo for the compatibility of windows and the alternatives and following them to solve keeping in mind of future work of Golem

varshith257 avatar Dec 28 '24 17:12 varshith257

@zelosleone The wit parser conflict, I think not under consideration of this issue. It happening with other os linux too not sure what about Mac though.

I have clearly discussed with jdegoes and vigoo for the compatibility of windows and the alternatives and following them to solve keeping in mind of future work of Golem

Yeah I was working for 10 days on windows with golem and discussed the errors i encountered and frustration as well on discord with jdgoes. so i had a headstart, thats fine, I almost finished the PR.

zelosleone avatar Dec 28 '24 18:12 zelosleone

/attempt #1210

Options

naaa760 avatar Jan 18 '25 05:01 naaa760

/attempt #1210

Algora profile Completed bounties Tech Active attempts Options
@Mayank77maruti 1 bounty from 1 project
CSS, Python,
JavaScript & more
Cancel attempt

Mayank77maruti avatar Feb 14 '25 08:02 Mayank77maruti

/attempt #1210

Options

abhishek12201 avatar Mar 08 '25 08:03 abhishek12201

/attempt #1210

Algora profile Completed bounties Tech Active attempts Options
@Rutik7066 12 bounties from 9 projects
Go, TypeScript,
Rust & more
Cancel attempt

rutikthakre avatar Mar 20 '25 06:03 rutikthakre

/attempt #1210

vishalpatil1899 avatar Apr 13 '25 10:04 vishalpatil1899

/attempt #1210

@jdegoes Do you require for the second point of eliminating docker/WSL requirement to provide also scripts for setup of services (eg. postgres) or should this change assume the services already installed on the test machine ? thank you.

parvit avatar Apr 26 '25 23:04 parvit

@jdegoes

  1. Reliably pass all tests locally without requiring docker or WSL

The ci tests depend on docker, e.g. launching postgres via docker.

@parvit

Do you require for the second point of eliminating docker/WSL requirement

be aware of the above

You seem to have misinterpreted my comment, what i meant is that to make the change to not require docker anymore (not used only in ci but also locally), you can either:

  1. require to have services installed and setup by the user

  2. install and setup the services via a script in the project

The tests would then operate by only calling the services like at the moment and reset the state inbetween (which would also be compatible with other platforms).

The choice is for @jdegoes to make but currently i'm working on the other points so there's no rush on my part, however you .

parvit avatar Apr 29 '25 20:04 parvit

The choice is for @jdegoes to make but currently i'm working on the other points so there's no rush on my part, however you .

@jdegoes is (quite obviously) overworked. Response times here and in other issues are low.

And he should not bother with such tiny details "scripts or not" (though he should be bothered with refining the requirements a bit, and of course with splitting down this issue/bounty or extending it with a management-issue/bounty at ~$1K)

Thing is:

  • removing WSL dependency: required
  • removing docker dependency: NOT required (the opposite, devs should be able to simply use docker, if prefered, otherwise go with an installed postgres.

I've recently installed postgres on Win10, it quite trivial, not much to script there.

, however you

?

(I comment often on counties, just to make then reach fulfillment sooner. This does not mean always that I want to take the bounty over. Mostly I just assist colleagues - we bounty-workers are kind of colleagues).

Sure you may see the matter however you like and implement accordingly, i'll do the same myself.

parvit avatar May 01 '25 13:05 parvit

Neither Docker nor WSL should be necessary.

It should be possible to do a plain-vanilla Windows install, check out this project, and build and execute all tests, with all tests passing, without any effort by a new Windows-based developer.

jdegoes avatar May 01 '25 14:05 jdegoes

@jdegoes so we can assume the service dependencies (tests do interact with postgres/minio) can be left to the user and only the configuration of the tests are to be part of the test (eg. username/password/hostname etc)?

parvit avatar May 01 '25 14:05 parvit

I want cargo build and cargo test to work immediately after checkout on any Windows machine, without the developer having to do anything at all other than type these commands after checkout.

jdegoes avatar May 01 '25 14:05 jdegoes

Would it satisfy your requirement if the tests that right now require interaction with docker containers instead use mock calls so its not necessary anymore to install and configure anything ? I think that would be an improvement also for the other platforms.

parvit avatar May 01 '25 14:05 parvit

@parvit Looks like I am missing something. I don't understand why we need Docker when running tests.

rutikthakre avatar May 01 '25 14:05 rutikthakre

@Rutik7066 An example: https://github.com/golemcloud/golem/blob/2b0dd21bf1461d59c9353e0a84b0f825d222056f/golem-service-base/tests/blob_storage.rs#L111-L119

This spins up a MinIO container for the test and same is done in other tests for postgres etc. Removing docker requirement means to fix these tests to work without it. I assume all tests unit and integration are required to be working on windows so that is why i've asked for guidance previously.

parvit avatar May 01 '25 15:05 parvit

@parvit Thank you so much for pointing this out. We can work together on this. What say?

rutikthakre avatar May 01 '25 15:05 rutikthakre

@parvit Thank you so much for pointing this out. We can work together on this. What say?

I'd like first to have confirmation on how to proceed for removing the wsl/docker dependency, that integration tests need to use mocks or just services local to the machine.

parvit avatar May 01 '25 20:05 parvit

/attempt #1210

laz-001 avatar May 26 '25 19:05 laz-001

wasm-rpc-derive uses backtrace-on-stack-overflow

The backtrace-on-stack-overflow crate version 0.3.0 has Windows compatibility issues. Specifically: Unix-only APIs: The crate is trying to use Unix/Linux-specific APIs that don't exist on Windows: nix::sys::signal libc::stack_t libc::sigaltstack

This means cargo make build is no longer possible in Windows, it was possible before cloud merge. I am also not sure if wasm-rpc-derive is needed for local, maybe building/check/fix on wasm-rpc-derive can be skipped

golem-common, worker-service, worker-executor, integration-tests all have dependancy on wasm-rpc-derive

Nanashi-lab avatar May 30 '25 06:05 Nanashi-lab

backtrace-on-stack-overflow was added to debug a stack overflow which is now fixed, can be removed. I was not aware it's not windows compatible.

vigoo avatar May 30 '25 08:05 vigoo