djangoproject.com icon indicating copy to clipboard operation
djangoproject.com copied to clipboard

[Proposal] Scheduled daily github workflow to run tests using Django's main branch

Open sarahboyce opened this issue 3 weeks ago • 13 comments

Was listening to DjangoChat and how we don't get enough testers for pre-releases. It was also mentioned that folks may not know how to do this

I think djangoproject.com should lead by example. We should have our tests run against Django main everyday with warnings enabled. Other folks can then look at this workflow as an example for their own project. We are also contributing to testing pre-released Django 👍

sarahboyce avatar Dec 13 '25 18:12 sarahboyce

I like the idea.

SaptakS avatar Dec 15 '25 06:12 SaptakS

I think djangoproject.com should lead by example

Yes, but I don't think that best practice applies to this project. One important point is that this repo is a project, not a library. It will not be used with multiple Django versions at any given time, so I think it's perfectly okay to have a single version in tests.

ulgens avatar Dec 15 '25 11:12 ulgens

Also, it's not clear how long we will be able to keep this project up to date with the latest version of Django, due to Python compatibility issues with Trac. I'll recommend finalizing

  • https://github.com/django/djangoproject.com/pull/2234
  • https://github.com/django/djangoproject.com/issues/2265

first so that we can have a healthier discussion.

ulgens avatar Dec 15 '25 11:12 ulgens

Django projects, not just libraries, should test pre-releases ideally on a regular basis 🙂 I think a test matrix doesn't fit this use-case so well, as there is no need to have this run on every PR. It's more about having our tests run periodically against Django main (could even be once a week, once a month etc) to test that Django itself hasn't accidentally broken something for our project

sarahboyce avatar Dec 15 '25 12:12 sarahboyce

I don't disagree with the intention, but I still don't think it's needed or beneficial. We can always create a new branch with a new Django version (or main) and can run tests on it.

ulgens avatar Dec 15 '25 12:12 ulgens

I agree that it's not needed. More of a nice-to-have than a true need 👍

In case we are discussing the benefit of doing anything at all, then the benefit is, if something got merged into Django that broke some existing functionality accidentally that we relied on, we would catch this early and raise a ticket. They would then fix it before that breaking change is even included in a release. Otherwise, we would need to wait a month for a patch release of Django before we would upgrade versions (if we were only checking once a final release is out). That's the benefit of anybody (whether it's a library or a normal Django project) testing against Django main on a regular basis. Having a scheduled job would mean this happens automatically. In theory, someone could do this manually every now and again

However, you may be trying to discuss an implementation detail rather than the benefit of doing something here? I'm open to other ways to approach this :)

Things to consider:

  • that something is broken on Django main but works on the current version of Django we are using should not block a PR, hence I wouldn't have Django main tests ran on PRs
  • how regular doesn't matter too much, the more regular makes it easier to find the breaking commit of Django
  • I prefer having something automatically scheduled than trusting some folks would do this locally manually every now and again

sarahboyce avatar Dec 15 '25 12:12 sarahboyce

@sarahboyce I really do understand where you are coming from, and I'd just approve this if we were talking about a project that is applying updates often and quickly. Given the project's development experience, I don't see a case where we would have to struggle with an issue that could have been caught earlier, thanks to being tested with main.

My vote will be negative, so we won't be adding unnecessary complexity. If the other people supports the idea and see no issues with the additional maintenance load, I'll comply with the majority.

ulgens avatar Dec 15 '25 12:12 ulgens

@sarahboyce Thanks for putting this forward. I think using the djangoproject.com repo to demonstrate testing against pre-releases is a great idea. In his lightning talk at DjangoCon US 2025, @adamzap suggested this repo should be an example for others; I trust this will inspire more developers to test pre-releases and provide feedback to Django. Perhaps it's even worth a blog post on djangoproject.com? Regardless, I plan to use this workflow as a reference for my own projects. Thanks!

tobiasmcnulty avatar Dec 15 '25 14:12 tobiasmcnulty

I absolutely agree with Tobias and Sarah. I think this proposal has clear benefits, including:

  1. As Tobias and Sarah noted, using the djangoproject.com repository to demonstrate testing against Django pre-releases.
  2. As Sarah said, providing the Django framework with much-needed early testing during the pre-release phase.
  3. Nice side effect but not necessarily main driver: enabling earlier detection of potential compatibility issues between djangoproject.com and Django releases.

Thank you @sarahboyce for pushing this initiative!

nessita avatar Dec 15 '25 17:12 nessita

I like this idea!

adamzap avatar Dec 15 '25 18:12 adamzap

+1 for some canary testing here!

jacobtylerwalls avatar Dec 15 '25 19:12 jacobtylerwalls

+1

pravintargaryen avatar Dec 16 '25 02:12 pravintargaryen

I only found this issue today and it really made me smile because yesterday I was listening to the same Django Chat episode with Natalia and had exactly the same idea. I even shared a quick thought about it on Mastodon before noticing this issue: https://fosstodon.org/@paulox/115723883286082489

So +1 from me as well, and kudos to Sara for being much faster than me at actually opening the issue, clearly that episode was very inspiring. 😂

pauloxnet avatar Dec 16 '25 09:12 pauloxnet