CI: Test downstream

[lithomas1]

It would be nice if meson-python tested downstream projects(e.g. pandas, scipy) to make sure changes here aren't unintentionally breaking them.

[rgommers]

Thanks for the suggestion @lithomas1. We tested SciPy in CI in this repo for a while. There are two issues with that:

1. SciPy and Pandas are very heavy, while meson-python is very lightweight (installs in seconds, full test suite runs in ~1 min)
2. They have lots of configurations and platforms, and any breakage is likely to be localized to a specific config.

I also ran Meson master + meson-python main + SciPy on Linux in my own fork for about half a year. It never broke there, so I stopped it.

SciPy does have two CI jobs that test pre-release versions of dependencies, pointing at https://anaconda.org/scipy-wheels-nightly and using pip install --pre. We should make sure meson-python from the main branch of this repo is used there I think, that's a more maintainable solution (it's a 1 line code change, and doesn't cost extra CI time).

@lithomas1 would that work for Pandas as well?

[lithomas1]

I would really prefer meson-python test pandas, instead of the other way around.

I don't think meson-python's coverage is very good right now(FFY00/meson-python#120 is a good example of this, where meson-python couldn't even build itself from an sdist). I also think the coverage covers more scipy-like projects, which is probably why scipy's build hasn't failed.

End-to-end testing like with pandas is a good way to fill in the gaps in the meantime (You already test SageMath anyways).

While I could add another job to pandas that checks meson-python main, like we do with numpy, I would prefer not to ATM since we are basically already at capacity in terms of CI jobs with pandas(we have 43 jobs).

Also, pandas doesn't take that long to build(it's around 5 minutes ish). We don't have to run the full test suite, but we should probably check that importing pandas works.

[rgommers]

Also, pandas doesn't take that long to build(it's around 5 minutes ish). We don't have to run the full test suite, but we should probably check that importing pandas works.

Okay, that's a good middle ground. I agree that that's a light enough load that it's a good idea for the extra coverage - at least right now.

[rgommers]

After https://github.com/scipy/scipy/pull/18270 we're now testing meson-python's main branch in SciPy CI. We were already testing Meson's master branch (no regressions for the past year IIRC).

[rgommers]

@lithomas1 FYI I have now set up a scheduled CI job for Pandas + meson-python main. Which I can also use to easily test meson-python PRs against Pandas. See https://github.com/rgommers/pandas/actions/workflows/test-mesonpy-dev.yml

Building Pandas is pretty simple, that is highly unlikely to find any regressions in meson-python at this point. Running the Pandas test suite is perhaps more likely to uncover some thing. So I'll keep that CI job running for a while on my fork.

I'll close this issue.