open-build-service icon indicating copy to clipboard operation
open-build-service copied to clipboard

Published 20 hours ago • verified publisher icon openSUSE

Downloads often fail hdrmd5 checks due to dod urls

Open Firstyear opened this issue 3 years ago • 6 comments

It's extremely common to see:

[46/68] (SUSE:SLE-15:Update) libldap-data-2.4.46-9.64.1.noarch.rpm: 100%|...| Time: 0:00:00 SUSE:SLE-15:Update/libldap-data: attempting download from api, since the hdrmd5 did not match

This is caused by .osc/_buildinfo-standard-x86_64.xml which has an invalid hdrmd5 contained (d0d0dd0d0d0d ... I think). I assume this is meant to mean it's a place-holder md5, but in reality what this causes is that when attempting to download from a mirror, these downloads will ALWAYS fail because the downloaded file (which is valid) does not match what the hdrmd5 in the xml has. This then forces the downloaded files to be discarded, and the buildservice api is used instead, which is MUCH slower compared to using the mirrors (especially when you live in Australia).

Firstyear avatar Mar 22 '22 00:03 Firstyear

Thanks for the report, the d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0 hash really seems to have a special meaning in OBS. @Firstyear could you provide a reproducer?

dmach avatar Mar 22 '22 08:03 dmach

On Dienstag, 22. März 2022, 09:23:46 CET Daniel Mach wrote:

Thanks for the report, the d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0d0 hash really seems to have a special meaning in OBS. @Firstyear could you provide a reproducer?

it references to not yet downloaded packages configured via download on demand repositories.

--

Adrian Schroeter @.***> Build Infrastructure Project Manager

SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev

adrianschroeter avatar Mar 22 '22 08:03 adrianschroeter

@dmach I'm not sure of the exact reproduction steps, beside "I just use osc and it does this about 1/3rd of the time". About the only special setup I have is that I'm using the option:

[https://api.opensuse.org]
downloadurl = https://mirrorcache-au.opensuse.org

This is because downloads in Australia are significantly faster with a local mirror. So I'm assuming it's some interaction there where the fact that the downloaduri option turns of api downloads (but so does download_api_only = 0 I think).

Anyway, the point is that the downloaded package is valid, but the xml we get is invalid because the hdrmd5 is wrong. If the point is to flag that the package is from a dod url, then that should be a seperate flag, rather than trashing the hdrmd5 value - especially since this breaks local/regional mirrors that are valid content holders.

Firstyear avatar Mar 22 '22 22:03 Firstyear

@Firstyear could you try patch from https://github.com/openSUSE/osc/pull/1019 ? Files with the d0d... checksums are no longer checked, I've also fixed indentation that probably caused even worse problems than you're describing (deleting all downloaded files and downloading them from API again).

dmach avatar Mar 29 '22 08:03 dmach

@Firstyear could you try patch from openSUSE/osc#1019 ? Files with the d0d... checksums are no longer checked, I've also fixed indentation that probably caused even worse problems than you're describing (deleting all downloaded files and downloading them from API again).

The patch fixes the issue from my side.

Thaodan avatar Nov 27 '22 21:11 Thaodan

we agreed in our meeting not to deliver the d0d md5sum in that situation.

This means osc can not verify the correct version at that point of time, it can still verify the gpg signature for security though.

(the issue goes away when a server side build triggered the download of the rpm)

adrianschroeter avatar May 09 '23 11:05 adrianschroeter