crater

crater copied to clipboard

There's a large number of crates out there which are Windows specific, so it is essential that crater be able to test those crates.

CLI usage on Windows

• [x] #267: Have Crater successfully build on windows
• [x] #275: Run cargo check on Appveyor
• [x] #280: Run cargo build on Appveyor
  • C dependencies needs to be downloaded in the Appveyor image
• [x] Implement and test all the functions in src/native/windows.rs
  • [x] #332: It would be nice to isolate all platform-specific code into a native module
• [x] Create the Docker image used by Crater to test crates
• [x] Successfully execute a demo run on Windows
• [x] Run cargo test on Appveyor
  • The test suite needs to be working on Windows
• [ ] Successfully execute a full run on Windows
  • This probably needs to be done on a server, full runs takes ages

Craterbot support for Windows

• [ ] Add a way to categorize agents by platform
• [ ] Add support for scheduling runs on multiple platforms at the same time
• [ ] Add support for rendering multiple platforms together in the report
• [ ] Ensure the agent works on Windows

Reference material

• https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/

Some unordered thoughts:

• there are a lot of assumptions of linux in Crater, like joining paths with format!("{}/{}"). I'm guilty of being lazy and doing this recently :( Not a big problem, 'just' a bunch of papercuts
• isolation is essential so we need to use Docker (or similar). I have a lot of experience with Docker on Linux, but none yet on Windows (and it's not too easy for me to get access), so I have a some questions:
  • is Docker on Windows fast enough at starting containers to be able to do start a container for every operation (generate-lockfile,test) on crates?
  • what environment do Windows crates assume? I'm vaguely aware of nanoserver - is that 'full' enough? But what about libraries that create GUIs on Windows, are they supported? For Linux we install a ton of random dependencies to help out crates that may require random libraries - do we need similar on Windows, e.g. the MSVC toolchain or whatever?
  • are all the Docker features we use supported (mainly volumes off the top of my head)? Are there important differences to be aware of?
• more generally, I develop on Windows about once every 3 months, and have a commensurate amount of experience with useful things like scripting - I know the basics of a Windows dev environment, but little more, so a committed volunteer with Windows experience may be necessary
  • relatedly, we have crater linux machines. I know a bit about Linux admin, but nothing about Windows

There's nothing insurmountable, but Linux issues are higher yield for me at the moment. I will keep this tracking issue in mind though to at least try and stop the gap from getting worse.

It's clearly important to make sure we get to Windows at some point, but I don't know how long it's going to take - any help on this front would be much appreciated!

nanoserver would not work. It is so extremely stripped down that I have my doubts over whether even libstd would work.

Windows Server 2016 has containers which use Docker. I don't know how fast they are but it is certainly something we could benchmark. There is plenty of documentation though so anyone who wants to work on this can fairly easily learn what they need to know. https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/

As long as we have VC++ installed and the environment configured using vcvars, then that should take care of almost all crates which are capable of working.

Following up on this: given how much we rely on crater to detect potential breakage, we really do need this. Would it make sense to try to seek out some help from an appropriate Windows developer? If someone popped up willing to do this work, would the crater developers have time to work with them to get this in place? If so, it might make sense to put out a call for help here.

Added a bunch of tasks that needs to be done at the top of the issue. Appveyor with cargo check is also going to be up soon (#275) so at least we know Crater compiles on Windows.

Now that Craterbot is sort of running adding proper Windows support is going to be feasible, we "just" need to get an agent running on a Windows machine. Unfortunately I have almost no experience developing on Windows, but I'm happy to review PRs if someone wants to do the work.

I've been doing a bit of work on this, so I thought I should post my progress here:

I've published a proof-of-concept Docker container which installs the requirements for rustc on Windows as well as vcpkg, a package manager for native code libraries. I learned some lessons while provisioning the host system (an Windows 10 Pro Azure VM), which are recorded in NOTES.md, perhaps these will be useful when provisioning a Windows machine for crater runs.

I've had some trouble building crater on the host , but was able to get far enough to encounter what I believe to be a rustup bug. Once this is resolved, I'll try get the minicrater runs passing with my changes to the native module.

There's a single failing unit test on Windows (it expects the result to be a path using "/" as a separator) which I will submit a fix for at some point :).

Whoa, thank you a lot for working on this! :heart:

I've published a proof-of-concept Docker container which installs the requirements for rustc on Windows as well as vcpkg, a package manager for native code libraries.

Great! Can you send a PR to rust-lang/crates-build-env?

I learned some lessons while provisioning the host system (an Windows 10 Pro Azure VM), which are recorded in NOTES.md, perhaps these will be useful when provisioning a Windows machine for crater runs.

It would be awesome to get this into the docs on the repo.

Another progress update:

The modifications in #399 along with the new docker image are enough to get the unit tests passing and minicrater working (modulo a bug where Docker on Windows does not report whether a container was killed due to an OOM condition).

I'm planning to do a Windows crater run on the top 100 to find some more bugs, as well as write an AppVeyor script in the next few days.

@pietroalbini I thought I had seen an issue where you explained how to do a top 100 crater run, but I can't seem to find it now. Am I imagining things? Is this easy to do?

Also, the latest commit to #399 run the unit tests on AppVeyor. Once rust-lang/crates-build-env#7 is merged, I'll enable the minicrater tests. Unfortunately, I believe that actually running containers requires an AppVeyor machine with nested virtualization. One exists, but I don't know if it's available in the free tier.

@pietroalbini I thought I had seen an issue where you explained how to do a top 100 crater run, but I can't seem to find it now. Am I imagining things? Is this easy to do?

Just pass --crate-select=top-100 instead of --crate-select=demo. small-random (20 random crates) also exists.

Also, the latest commit to #399 run the unit tests on AppVeyor. Once rust-lang/crates-build-env#7 is merged, I'll enable the minicrater tests.

:tada:

Unfortunately, I believe that actually running containers requires an AppVeyor machine with nested virtualization. One exists, but I don't know if it's available in the free tier.

I'll try to figure if we can enable nested virtualization on our account, thanks for pointing this out.

Now that #399 is merged, crater runs are possible on Windows. Work to build as many crates as possible by adding dependencies to crates-build-env-windows is ongoing. There's also a few outstanding issues with Docker for Windows that need to be worked around (see #426 and this feature request).

The next step is to make craterbot aware of alternate platforms. @pietroalbini I think this should go in a new issue? Both @rylev and I would need some mentoring instructions to implement this.

Opened an issue with rough intructions to make craterbot aware of other platforms: https://github.com/rust-lang/crater/issues/430

Ping me on that issue or Discord if y'all need mentoring.

I opened a PR (#450) with an initial implementation of #430. It doesn't address scheduling for multiple platforms simultaneously or merging reports between platforms, however.