GRIB.jl
GRIB.jl copied to clipboard
weech
Support for Windows
We noticed that GRIB.jl does not support Windows yet.
Can you comment on why this is the case? Any plans to get it fixed?
Hi, I'd be glad if this package had Windows support, but I don't have a computer that runs that operating system, so I have no way to debug the build script for eccodes_jll. I was able to verify that it didn't work out of the box on a library computer I had access to when I wrote this, but I didn't have the tools or knowledge of the operating system to investigate it further.
Curious if docker images would help? If so, I want to try my hands on it with a little help of yours 🙌🏽
@weech any update on this front? @LakshyaKhatri can help, he is very skilled with test infra on GitHub.
Thanks for the reminder on this one @juliohm
I can spin up a windows docker image with Julia configured in it (have to look how to do that) and then maybe @weech can run and debug GRIB.jl in it?
PS: I'm resisting understanding the entire GRIB.jl codebase secretly. 🤫🙈
@LakshyaKhatri you can also try to use BinaryBuilder.jl to get a reproducible Windows environment with Julia, it should be even simpler. Check this cross-compilation of OpenCL I've written: https://github.com/juliohm/cross-platform-opencl
You can launch the script on any specific platform, or you can use BinaryBuilder.jl directly to activate a platform from the REPL.
The main problem is that eccodes itself doesn't support Windows fully. At some point someone got it working with msvc, but there was something different about the tools in BinaryBuilder that meant that the Windows eccodes_jll artifact doesn't work. It has been two years since I last looked at it, so things may have changed since then.
Okay did some googling, if I'm understanding it correctly, we are talking about: https://juliahub.com/ui/Packages/eccodes_jll/NxZch/2.25.0+0
and they mention on their docs that these windows platforms are supported (last commit was 7 March 2022):
- Windows x86_64 {libgfortran_version=3.0.0} (x86_64-w64-mingw32-libgfortran3)
- Windows x86_64 {libgfortran_version=4.0.0} (x86_64-w64-mingw32-libgfortran4)
- Windows x86_64 {libgfortran_version=5.0.0} (x86_64-w64-mingw32-libgfortran5)
Does that mean with that I should be able to use https://github.com/juliohm/cross-platform-opencl to build a windows package for GRIB.jl and then test it in a separate PR?
@LakshyaKhatri the repository I linked is unrelated to GRIB.jl, it was just an example of how you can build binaries with BinaryBuilder.jl or how you can activate specific platforms even if your host platform is not Windows.
That list of platforms on the page you linked is automatically generated and doesn't mean that those artifacts actually work properly. If I remember correctly, I tried having it so no artifact was generated for Windows at all, but the maintainers of Yggdrasil didn't want that.
What needs to happen is someone needs to be able to iterate on the eccodes_jll build script and test the produced artifact in a Windows environment. It could also be that some environment variable needs to be set. I can do the iteration (at some point after Thanksgiving), I just don't have access to a Windows environment. If someone knows how to create one with Docker, I'd be happy to use that process.
Okay, did some digging on this end:
Docker Image: Docker image for Windows on Linux platforms is not possible. Tldr; I forgot that Docker don't virtualize containers, it just runs it using the native resources. So, for spinning up a Windows container we would need Windows machine.
ecCodes: I went through the build script for eccodes_jll and through ECMWF's official website from there to realize that ecCodes isn't officially supported on Windows:
As @weech mentioned, they made it work on Windows by creating what looks like some kind of mapping to me and generating static dlls and exes from it. (please check this update). But it's usable only via Python and Conda 🤷🏽♂️
As they pointed, maybe we can play a bit with the Magics and ecCode repos?:
If you find problems, we invite the Windows user community to use our new set-up on GitHub to fork/contribute and issue pull request changes to ecCodes and Magics. Old Reliable Tech has set up automatic tests and builds for Windows on GitHub, which should ensure continued releases for this new platform.
Simplest solution would be to build ecCodes from source to generate Windows and Linux artifacts separately. 💭
I've been looking at this too for quite some time. I do have a Windows and Linux machines at home. The code doesn't run under Windows as it comes back with an error message from GRIB.jl. I know it is related to eccodes.jl. However I was able to use a different approach by using the binary decoders under Julia by reading the file as a binary. I was able to convert the data back to decimal values which do correspond to what is stored in the file. The only thing that I was not able to decode yet is the openjpeg compression format which is not supported in Windows. I can read the different messages and indexes under Windows as of now but I didn't develop the code for reading the data part yet.
I also created a function that converts the ECMWF data tables copied in a text file to a CSV file where it is saved locally.
So cool human! We would absolutely love it if you could share the snippet here 🙏🏽❤️
My idea was to create pure Julia code to access Grib2 files. Just started to scratch the surface. I will need to continue to work on the code now that I know a lot more how Julia works. That might go against the development being already done by ECMWF but I was interested in finding if Grib2 files can be decoded directly without using C++ code. It does work but more coding will be needed before I can assemble it together. I didn't come across any issues in Windows when reading the file. Also need to use struct in Julia to make it easier to extract values.
Created this code as of now:
- opening Grib2 files
- read sections directly for each message in one file following ECMWF code
- read ECMWF code directly in each section. Need to export remaining ECMWF tables into CSV format. Can be exported in other format if necessary
- Can adjust the code to use the same functions as grib.jl
- Need to find ways to use Openjpeg in Windows if it is possible
Any comments or suggestions would be appreciated.
I am just starting to understand how to use Julia but I haven't look at artifacts just yet. Some help will be needed with this. I am able to read Grib2 as a binary file which is decoded into decimal values. Got the inspiration on the Python side where there is a simple function to do this work. However there is no equivalent function in Julia but the data can be decoded with the use of array with Uint8 type values. I was surprised that it worked. Done some similar work in Python before. I am quite experienced with Python code as I am developing web pages with the use of Fastapi and Bootstrap.
So I borrowed a Windows computer for the weekend, and I got most of the tests passing today. The only feature that isn't working yet is the Index class. I did try to get that in today, but there was a lot that I couldn't figure out. The problem is that the Index class opens files for you, but because it's Windows they have to be opened in binary mode, and ecCodes doesn't do that. I tried linking in a library that was supposed to fix that, but then the ReadOnlyMemoryError returned. I'll see if I can get it working in the time I have left with this computer.
If anyone thinks that I should release a new version that includes the partial Windows support, just say so. Otherwise, I'll hold off until all the tests pass.
From my point of view it would be better to hold off until all the tests passed. Maybe someone will figure it out with C code but I won't hold my breath. I'm still new at programming in Julia but I learned it a lot quicker than Python. I do love to program in Python but I find that Julia is closer to C in terms of speed than Python.
However I am not a big fan of C pointers in Julia. I've continued to develop the code for Grib in pure Julia by using binary data. I'm only in the early stages right now as I'm trying to learn how to use struct types, reading binary data and creating Ecmwf code tables in CSV or JSON format. I too can't figure out how Eccodes work in C code.
As I said before I'm using a different approach that seems to work in Windows. I'm also creating dispatch methods by reusing Julia base methods. The only thing I'm seeing is that it's not type stable. According to documentation the code needs to be refactored or not depending on the benefits.
Can we have a new release of this package with what is working currently? We noticed that the latest registered release is emitting warnings in downstream packages that don't even attempt to load GRIB files. The warning is sitting in the init function of the module.
Appreciate if you can tag a new version with the latest advances in the master branch.
Please release a new patch or minor without the warning in the init::
┌ Warning: You are using GRIB.jl on Windows which is currently is not supported. You will likely hit a ReadOnlyMemoryError, and, if you know how to get around it, please let us know!
└ @ GRIB C:\Users\julio\.julia\packages\GRIB\6rlik\src\GRIB.jl:129
The registration system was down for a while, but it seems to be working now. The new version is now in the registry.