django
Migrate to renovate
Unfortunately, dependabot's development has been stagnant for a while. It works well enough with pip / requirements.txt, but its uv support has many issues that weren't resolved, even though they announced it ~1.5 years ago.
Comments on details of this process and active issues can be found in this issue's comments: https://github.com/dependabot/dependabot-core/issues/10478
The migration is a blocker for https://github.com/django/djangoproject.com/issues/1901
One small extra benefit is pre-commit hooks support: https://docs.renovatebot.com/modules/manager/pre-commit/
Why do we need this?
The core idea is to keep the automation working for dependency updates. Currently, we have dependabot in place (integrated to GitHub) but...
- It has multiple issues with modern Python tooling (https://github.com/dependabot/dependabot-core/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22L%3A%20python%3Auv%22)
- It doesn't support all the ecosystems we use (missing pre-commit support).
- This is more of a personal experience but I've been a contributor of dependabot since 2020; the Python ecosystem was always suffering from a lack of maintenance effort. The Python-related modules were not in a usable state without the volunteer effort. The situation got even worse after uv's initial release. I've been in discussions with different GitHub employees / Dependabot team members over different channels for the last 1.5 years and it saddens me to tell that that effort amounted into nothing in the end. The current state of Dependabot's Python ecosystem tools poses a risk to supply chain security (https://github.com/dependabot/dependabot-core/issues/10478#issuecomment-2863472205)
Can't we just do it manually?
We can, but I believe that we shouldn't. It requires a not-so-small amount of effort. To ensure consistent security for the project, we need to be actively watching for all 3rd party updates and create the PRs manually. Right now, I'm handling this for a tiny portion of our dependencies. It's repetitive, and it requires the maintainer to stay vigilant. It's a perfect task for automation.
A new integration
Thanks to the benefits it provides, I don't see Renovate as an optional 3rd party - what it does is vital for any software project in this day and age. I'd be happier if Dependabot (an integrated GitHub product) were still an option, but unfortunately, that's not the case. Also, with Renovate's pre-commit support, we won't need pre-commit.ci integration for hook updates.
Required permissions
Renovate creates issues and PRs, and uses security warnings to do its job.
- The feature that needs issue creation can be disabled, but in my experience it's a useful feature: https://docs.renovatebot.com/configuration-options/#dependencydashboard
- The feature that needs to create PRs can be disabled, and we can create PRs ourselves (I wouldn't prefer that): https://docs.renovatebot.com/configuration-options/#prcreation
- The feature that needs access to vulnerability alerts can be disabled too, but I think having it enabled is a good and useful thing: https://docs.renovatebot.com/configuration-options/#vulnerabilityalerts
Please let me know if there is any detail/concern that I missed and I'll try my best to address that.
A nice reading about Renovate's position in its area: https://www.jvt.me/posts/2024/04/12/use-renovate/
Hi @ulgens. I chatted with @bmispelon and @nessita about this at our monthly Ops team checkin. We agreed we can't grant the renovate bot's requested perms at this time. To flesh out what was already alluded to on Slack:
- the Django org includes private repos managing sensitive infrastructure and handling security reports, so we're much more careful in this regard when a bot requests org-level perms.
- we're actually going through a period of formalizing matters like who has the GitHub ownership to grant these kinds of requests. Even I don't have access to grant this kind of request.
We're not against tooling modernization, and that part is not even our call, the website WG can decide what tools it wants to use. I hear you that Dependabot has a lot of gaps supporting uv. I think that's something the website WG can decide on: the roadmap for adopting uv lockfiles vs. staying on dependabot vs. some other automation solution vs. wait and see. I hope that makes sense.
@jacobtylerwalls Thanks for taking the time to discuss this, but I'm sorry that the main concern here is just a misunderstanding.
- Renovate doesn't need org-level umbrella permissions that gives access to everything that org owns. The request I sent (and yes it was premature, sorry for that) was for only djangoproject.com and code.djangoproject.com repos. I tried to explain this to @nessita in Slack and also in this issue, but it seems it wasn't enough to clarify. I believe we already have multiple integrations with similar permissions, like Sentry and pre-commit.ci. If there is anything I'm missing, the best way to move forward is to discuss it together.
- A formalization would be nice. Right now, I'm confused about how to move forward with this discussion. On one hand, I do understand that there should be an approval consensus in place and it shouldn't be a blind approval. On the other hand, the discussion already caused a waste of time on your end because of this permission misunderstanding - the goal wasn't to integrate it into any resource that the Website WG doesn't already own. If we say that "when an integration is needed, the WG group that will use it needs to decide on it officialy (in a meeting and or with a voting)", I would support that decision, but I also think that WGs deserve a level of autonomy for the resources they own so I'd expect that voting to be final and shouldn't require another set of voting/discussion in the org level.
think that's something the website WG can decide on: the roadmap for adopting uv lockfiles vs. staying on dependabot vs. some other automation solution vs. wait and see.
We already have an issue to discuss the adaptation of the modern standard: https://github.com/django/djangoproject.com/issues/1901. Under that issue, we have a supporting comment from @tobiasmcnulty (here) and no opposition. By the nature of the process, we will need approvals before a PR like https://github.com/django/djangoproject.com/pull/2321 got merged, but I'm not sure there will be an actual roadmap because this is not even a controversial change (and if there is any controversy, I still think that issue is a good place to share concerns). Apart from the pyproject and lock file, I can and will present the topic of dependency updates in a website WG meeting.
All in all, I'm happy we are having this conversation and pointing out the missing org-level policies/permissions (like the officialization of ownership for these types of requests), but it's also sad because a simple misunderstanding led to a misguided concern. For the future, I think it's much better to include the owner of the request in the discussion because otherwise the communication process is too fragmented and things can go wrong (as it's happening now)
Thanks for the reply -- I rechecked with the Ops team, and it turns out I didn't state the concern correctly. You are correct that only repo-specific permissions were requested for Renovate here, but part of granting a repo-specific permission involves approving an org-wide integration that has the effect of listing it in the approved/active integrations and generating emails when the integration asks for approval of updated permissions. This is what the Ops team is concerned about--although I take your point there's an existing precedent for Sentry and maybe more (I don't have the perms to check & find out)--in other words, that it's "too easy" [for this use case] for an "accident" to happen where the permissions are elevated because someone approves a request from the integration "since the integration is already there, so someone must have approved this already".
"For this use case" being a judgment call that could be informed by the view of the website WG and others.
I also think that WGs deserve a level of autonomy for the resources they own so I'd expect that voting to be final and shouldn't require another set of voting/discussion in the org level.
Ultimately the ops team does take responsibility for all of our org infrastructure, including this site’s infrastructure. So clearly there are topics where both teams need to be in agreement for things to proceed, and that can involve different votes that can go different ways. Obviously it’s in everyone’s interest that we keep the bureaucracy in check, and that different teams are more autonomous but it’s not always super feasible.
Back to Renovate, for what it’s worth, it has a lot of options to run it without the SaaS GitHub app. For example within GitHub Actions on a schedule, or manually on someone’s machine. I have used both of those options for projects with restricted access, though it was about 4-5 years ago. It still requires some forms of access to do some of its logic - from memory, I’d set it up with a "repo:public" token on my bot account, with that account having its access limited to specific repositories, and those repositories having branch protection set up. This PR is an example if it helps, and here’s the corresponding Renovate + GHA configuration.
I’m not familiar with the threat model here nor permissions granularity between SaaS GitHub Apps vs. PATs vs. setting up one’s own GitHub App, so can’t say more about its suitability for our needs. In the above example I ended up disabling it for that specific repo just because I decided for my use cases batch manual updates would work better.
