aqtinstall
aqtinstall copied to clipboard
miurahr
Mirrors don't provide sha256
aqtinstall requires .sha256 hashes to verify Updates.xml and 7z archives.
Only https://download.qt.io/ provides .sha256, .sha1, .md5 and .meta4 files for every visible file.
Mirrors provide only visible files, i.e. .sha1 for 7z archives which aren't *meta.7z.
If https://download.qt.io/ is not available, it's impossible to download any archive with aqtinstall.
There should be an option to use .sha1 and skip verification of Updates.xml.
Thanks for letting us know about the lack of sha256 hashes on the mirrors.
IMHO, we should be very careful about allowing aqt to use sha1 or md5 hashes in any way. It’s not hard for an attacker to create a .7z file with a hash collision with a legitimate sha1 or md5 hash; this is not true of sha256. If an attacker has control over any of our mirror sites, that attacker can convince aqt that their malicious copy of the .7z file is the real thing, and suddenly the attacker can get their code to run in your Qt program. This is a very big deal.
If you want to add the ability to use sha1, you will need to be able to prove without any doubt that there is no code path that uses sha1 by mistake. This feature will add attack surface to aqt that will need lots of developer attention.
There are other potential solutions to this problem, other than using insecure checksums. We could host our own copies of the expected sha256 checksums here on Github, and use our copies as a fallback when download.qt.io is unavailable.
I have a project at https://github.com/ddalcino/qt-repo-cache that could be modified to do this job; right now, it hosts copies of the Updates.xml files at download.qt.io, stored as json. This project runs a workflow once a day that scans 8 HTML files at download.qt.io: 3 for the Windows targets, 2 for the Linux targets, and 3 for the Mac targets. It reads the 'Last Modified' column for each folder to decide whether or not anything has been updated. If a folder has been updated since the last time the workflow has run, it fetches the Updates.xml file and caches it; otherwise, it skips the folder. This approach will only make 8 HTTP GET requests per day if nothing has changed, and if something has changed, it makes the minimum number of requests required to fetch the updated files.
We could do something similar: Check the 'Last Modified' column for each folder, and if anything has changed recently, go into that folder and fetch all the sha256 files for modules that have been updated.
I must note that this approach also increases attack surface. I have the workflow at https://github.com/ddalcino/qt-repo-cache set up to commit changes automatically, rather than by pull request. The repo changes so often that I would not be able to keep up with it if I had to approve every change manually. If an attacker could somehow convince the workflow to download a bad checksum (via MITM, etc), there would be no way to prevent the workflow from committing those values, and no way to know what had happened afterward.
I don't know if this is a good solution, but IMHO it's better than using sha1. If download.qt.io is unavailable a lot of the time, maybe we should consider something like this.
As a temporary workaround, I wrote this https://github.com/mite-user/mite-qtinstall The idea is to prefetch SHA256 hashes as @ddalcino suggested. Hope this helps.
Last night I tried to install qt 6.4.0 in CI, but the job failed because there’s no sha256 hashes for any of the files in that directory, on the download.qt.io server. Hopefully this is just a temporary situation, and they will upload the hashes later. However, there’s a very real possibility that the qt repo will stop uploading sha256 hashes for new versions. We don’t have any guarantees here.
In this case, caching sha256 hashes downloaded from download.qt.io will not work.
We may have no choice but to add an option to use one of the insecure hashes like md5/sha1 that appear to exist on both download.qt.io and the mirrors. I propose that we call it something like --insecure to discourage people from using it unless absolutely necessary, and to warn them about what they’re doing.
Quick update on the sha256 situation: There are now sha256 hashes available for Qt 6.4.0. The CI job that I observed as failing (due to missing hash) now passes: https://dev.azure.com/miurahr/github/_build/results?buildId=5742&view=logs&j=b86c6ba8-325a-58b4-9c91-e9a8f44b4793
It’s not hard for an attacker to create a .7z file with a hash collision with a legitimate sha1 or md5 hash; this is not true of sha256. If an attacker has control over any of our mirror sites, that attacker can convince aqt that their malicious copy of the .7z file is the real thing, and suddenly the attacker can get their code to run in your Qt program.
I'm not security expert but I don't see a problem here. Even if attacker creates a hash collision, that "malicious" copy will contain a garbage, so no code can be injected that way? Just my 2 cents, sorry if I'm wrong.
I'm not security expert but I don't see a problem here. Even if attacker creates a hash collision, that "malicious" copy will contain a garbage, so no code can be injected that way? Just my 2 cents, sorry if I'm wrong.
No, the malicious copy may contain whatever code the attacker wants. It could be padded with garbage to make the hash come out right, but the padding would likely be hidden in a way that you would not see it.
You can search for hash collision generators on any search engine to find tools that allow you to do this; several are hosted on GitHub.
Thanks, didn't know that. Does it mean all the mirrors can be considered insecure right now?
No, the mirrors are fine, as long as the binaries they host match the sha256 hashes hosted on download.qt.io. It’s relatively easy to fake a sha1 hash, and extremely difficult to fake a sha256 hash.
FYI: qt.download.io uses MirrorBrain for its distribution server. MirrorBrain is a download redirector and generates cryptohashes, and metalinks.
Aqtinstall depends on metalinks which MirrorBrain generated, and detect blacklisted mirror site, not just redirect by 304 response.
- ref. https://en.wikipedia.org/wiki/Metalink
- ex. https://download.qt.io/online/qtsdkrepository/linux_x64/desktop/tools_qt3dstudio/Updates.xml.meta4
MirrorBrain automatically generates Updates.xml.meta4 .sha256 .sha1 .md5 and even .torrent
Hi! Is it OK then if I propose a PR that adds one --insecure option which disables checksum verification?
Reading all issue tickets related to the problem. But as i get it, there is no solution? It's not possible to use aqtinstall, if https://download.qt.io/ is not available?
Reading all issue tickets related to the problem. But as i get it, there is no solution? It's not possible to use aqtinstall, if https://download.qt.io/ is not available?
You can help #684 to be a state of quality that pass all the CI checks.
Reading all issue tickets related to the problem. But as i get it, there is no solution? It's not possible to use aqtinstall, if https://download.qt.io/ is not available?
You can help #684 to be a state of quality that pass all the CI checks.
That's what i did. I took head from here, added edits from that pull request, added lines to settings.ini and it worked.
[requests]
hash_algorithm: sha1
INSECURE_NOT_FOR_PRODUCTION_ignore_hash: False
Lost that executable. Now trying to compile a new one.
python build_standalone.py ends with no errors
But compiled exe showing error
./aqt
Traceback (most recent call last):
File "launch_aqt.py", line 2, in <module>
File "PyInstaller\loader\pyimod02_importers.py", line 385, in exec_module
File "aqt\__init__.py", line 31, in <module>
ModuleNotFoundError: No module named 'aqt.version'
[31404] Failed to execute script 'launch_aqt' due to unhandled exception!
Pure master branch shows same
aqt/version.py is generated when python -m pip install . by setuptools_scm.
you can see CI/CD script. https://github.com/miurahr/aqtinstall/blob/master/.github/workflows/upload-release-artifacts.yml#L36C11-L36C34
setuptools_scm configuration is here.
https://github.com/miurahr/aqtinstall/blob/master/pyproject.toml#L90-L96