pyvroom
pyvroom copied to clipboard
VROOM-Project
Support for libosrm
this would be awesome! just a few comments from recent experience:
automated source builds with osrm as submodule might be a PITA without cmake (also maintenance-wise), which would bastardize setup.py. though I'm not 100% familiar with osrm config/compiler args for platforms. I'd like to add valhalla library support eventually to vroom upstream (and/or the bindings) as well and that's definitely a horror show without cmake. (edit: I guess what I wanted to say it might be easiest to expect osrm(/valhalla) installed on the build system and just distribute wheels, no source distro)
happy to see what you come up with as architecture:)
Yeah, I am seeing the cmake script now and it looks like an uphill battle. Might give up on it if it is too much a challange.
Do you have an issue for valhalla library support that I can follow?
Nope it’s not that far yet.. I’ll open one when I actually do it. It’ll be more painful than osrm, so maybe I’ll wait until a client pays for it;)
I just created pyvalhalla: https://github.com/gis-ops/valhalla-py. and will also do a pyosrm (any existing py osrm project is unmaintained and using either cython (no thanks) or wraps osrm in with some C (also no thanks)). didn't give it too much thought, but maybe there's a way to leverage that (future) package for pyvroom to work with libosrm directly? adds a level of hard-to-wrap-your-head-around indirection, but the needed setup to get libosrm built for all platforms is sorta evil and would be a pity to pollute 2 repos with that (here and pyosrm).
in that valhalla package you'll see the way I had to go for valhalla: I checked in all dependencies into the repo for all platforms, including libvalhalla (mac still to come, don't have one). reason is that win kills CI performance, either it runs > 1 hour to install all dependencies on every run or I run out of cache while trying to cache the vcpkg packages which are > 1 GB. linux & mac only would be ok though, ditching win is not an option for us however. bad point is obviously that I have to version dependencies alongside the bindings and check in new valhalla libs for all platforms every time I have to change smth in valhalla core. not so much an issue for me though, won't happen too often.
That is really cool!
Getting a compatability layer between valahalla-py and pyvroom shouldn't be an issue, I belive.
Take a look at the current github action workflow in pyvroom. I have three set up: pull-request (runs often and fast), main-push (more validation for some platforms), and release (full builds for all platforms).
This way the 1 hour run can be limited to just when doing a full release and you avoid using up all your gh action minutes.
I am not familiar with the windows compiler, and I don't know if this is relevant for you, but this trick reduced memory and cpu usage for me.
thanks! yeah I also have 3 workflows now, much like yours (which also gave me that inspiration:)). true, the release one could build win as the only workflow, but then I'd never test windows unless releasing which is dangerous in my experience.
luckily my binding code is very trivial (purely high-level, equivalent of having only vroom's JSON API exposed), so that builds in 20 secs. it's the dependencies which suck big time on win (again..)
I'm guessing thats boost causing trouble?
Have you consider using the dedicated gh action installer: https://github.com/MarkusJx/install-boost Precompiled so should be less stressfull compared to conan.
unfortunately it's not boost, but everything else. boost we only use header-only, so the conan solution is pretty easy. it's the protobuf, lua, sqlite/spatialite etc binary dependencies, here we can't use conan bcs not all packages are on there. so we have to vcpkg. which is a terrible dep manager IMO, which not only doesn't offer pre-compiled but also compiles all libraries in release and debug mode. there's a (pretty opaque) option to disable debug mode, but then some packages won't compile at all bcs some of their build logic depends on the debug stuff.
I do really appreciate suggestions though. took me ~3 full days to trial/error my way out of all those disasters and ended up with this dep-in-repo crap, the only option I really saw to satisfy everything except some slight maintenance penalty.
Ah, right. I just looked at your conan dependencies, not the include folder. that looks a lot more brutal, yes.
Not sure if this is relevant as I'm following the discussion a bit remotely, but for the core CI builds using GitHub Actions we cache the libosrm builds per version/compiler, so the time it takes to build libosrm only happens once (per version) and for all. The cached versions are probably available from this repo too if it can help, see https://github.com/VROOM-Project/vroom/blob/master/.github/workflows/vroom_libosrm.yml.
