minutor
minutor copied to clipboard
mrkite
Definition Update not working
Triggered by issue #102 I tested functionality of definition update: It fails silently
During debugging I get a QNetworkReply::ProtocolUnknownError in definitionupdater.cpp::checkTime()
It seems that for HTTPS it is necessary to self-compile Qt with OpenSSL support linked to the library: https://forum.qt.io/topic/44117/qnetworkaccessmanager-and-https/18 http://doc.qt.io/qt-5/ssl.html
EtlamGit writes:
It seems that for HTTPS it is necessary to self-compile Qt with OpenSSL support linked to the library:
I don't think we want to bundle Qt so that isn't really an option.
Are we left with:
- Find a cross-platform https client
- Mirror the files on another protocol (http, ftp) Or do we have more options?
seancode's website works with http but I don't know if a mirror can be set up there and whether or not the host wants to.
I could offer a mirror myself but then everyone would have to trust in me to stay available and not serve anything malicious.
If it's just definitions, and not a program file, then plain http is okay, I guess, there's very little room for anything malicious. If it were for the main program files, though, that's a different story.
Anyway, I do have Qt self-compiled because of Minutor already, because I wanted to link Qt statically, rather than dynamically. I could have a look at enabling OpenSSL in my build and then provide the executables for the releases.
Yeah, I already statically link Qt for Windows and OSX releases. Linux releases on the other hand are dynamically linked (package manager for the win).
I'd like to keep the definitions on github, because it's just easier for everyone... so getting https working is definitely the way to go.
Github only supports HTTPS. So moving the definition files there caused this problem.
I'm able to build OpenSSL from the sources. Versions > 1.0.0 are claimed to be compatible with Qt. But versions 1.1.x changed a lot and I'm not able to build it anymore. Pre-build DLLs are also available for download from openssl.org.
From what I read during the day is that Qt "should" load provided OpenSSL DLLs automatically. But with my static build it does not . . . It seems that without specifying it, configuration defaults to -no-openssl whereas automatic DLL loading would be -openssl (next Qt build should be finished tomorrow...)
It is also possible to link OpenSSL static at compile time of Qt library. But as OpenSSL is some high security stuff with frequent updates and hacker attacks to older versions, it could by wise to use DLLs that can be replaced with latest versions more easily.
@EtlamGit Wouldn't it be easier and more reliable to bundle something like libcurl instead of an OpenSSL DLL or the whole of Qt?
P.S. I've made an HTTP mirror that updates once an hour using a simple git pull so you don't need to do anything special. It's there and available even if nobody wants to use it.
Minutor is a Qt application, so you have to "bundle" Qt with it. This is the case now and was the case since several years. The problem now is just that for some erratic US export restrictions Qt feels that they are not allowed to bind to OpenSSL by default. So we just have to do that part ourself.
From Qt side the intended approach for HTTPS is to use OpenSSL. Everybody using Qt is doing that. Curl would not make it easier, as it would be an external application. We need something compatible to the current code. And the current update code is using Qt network methods.
It is just a matter of recompiling Qt. We are doing that since years. But it takes several hours. And fails when you replace your SSD meanwhile...
A HTTP (without S) mirror would be a fall back solution, in case everything else fails. But I think that in some years all the internet will be HTTPS anyways. And only mrkite should be responsible to provide official files.
Seems like Qt has a rather strange deployment model but if that's the case then please disregard everything my ignorant self said in this thread.
I agree that relying on a third party (like I am) to distribute files is poor form. I set it up because I misunderstood the scope of the problem and already had the infrastructure available. MrKite said SSL support should be fixed which to me sounded like refactoring and when DLLs were mentioned I got portability nightmares. Glad it ain't so.
Intermediate result: Recompiling Qt with OpenSSL support is possible. Provided DLLs are loaded automatically and requests can be done.
End of good news, cut message here -----%
But github.com does not provide a "Last-Modification" header in the response. So the current code is not able to detect updated files. It may be possible to parse the version info inside the definition file itself. But in that case we should do the update only weekly or some other longer time frame.
You can check header responses with this tool: http://www.webconfs.com/http-header-check.php
Agreed. Checking for updates every run isn't ideal anyways, people start to think you're tracking them or something. Weekly is good for me.
What change to the QT configuration command did you do? I'll verify it works with MacOS as well, and if everything is good, I'll release new binaries later this week (after the long recompile)
I updated: https://github.com/mrkite/minutor/wiki/Self-Compile
(I'm not sure if I find the time to finish and test all code changes for the new definition update during this week.)
Just tried to repeat my steps on a different computer ... the instructions were missing the step where you configure what and how to build. I updated the self compilation instructions with this step. Now it should work and generate the two DLLs that have to be placed in the same folder than Minutor (e.g. the Debug folder). This page can also be used as reference: https://wiki.openssl.org/index.php/Compilation_and_Installation#W64
@mrkite we can close this issue, when you include libeay32.dll and ssleay32.dll in the next release. I just updated the build instructions, as it was not clear, that these two files have to be located in same folder than Minutor.exe. A source for pre-build OpenSSL binaries would be: https://indy.fulgan.com/SSL/ It is possible to copy the DLLs from there into the same folder than Minutor. But during configuring Qt the OpenSSL sources have to be present also.
Ah.. makes sense. I can update the wix setup to copy those dlls as part of the install as well. I'll try to get a build going.
In my opinion for a 1.13 compatible release only this is missing and possible to do short term:
- decoding of Structure Overlay (update to new storage format)
Other stuff is more cosmetic or requires access to Minecraft source code:
- handling of Blocks with changing top side based on attributes
- hostile Mob spawn detection
As Minutor 2.2.0 is out (together with the OpenSSL libraries) automatic update should work now via HTTPS. We can test that in real live with the upcoming changes for 1.14 . . .
I still need to test it on Mac OS.. Qt recommended using Apple's native SecureTransport libraries instead of OpenSSL on MacOS.. so I compiled the release with SecureTransport enabled, but haven't actually had time to test if it actually works.
Definition updates are working for Windows and Linux. But I have no way to test it on MacOS.
To solve that, I prepared a test:
- I just added the first 1.19 Entities and we could observe if they get updated on MacOS
- ensure that you use a recent release of Minutor, but NOT the latest Nightly
- ensure that you have a Entity definition file with version number starting with 17 (last update before)
- force a definition update or wait 7 days
- verify that a Entity definition file with version 19.0.22w13a is now used
To get some "volunteer" testers I just randomly trigger people that participated in other MacOS related discussions . . . : @PeacePlays @Macfelon @bighairydave @afontenot @magicus @rjp @Down10 @bskendig
Hi all, updates for 1.19 definitions are done. Does anybody see these files automatically updated on a Mac?
I'm giving this a try. I haven't run Minutor for a very long time. On my Mac, I downloaded the 2.19.0 release (Minutor.macOS-5.12.zip) and ran it.
It gave me a "Failed to add Block from definition file, as it might be a duplicate or generates the same hash as an already existing Block" error over and over and over (each time I would dismiss it, another modal would appear with the same message) until I killed the app. I deleted ~/Library/Application Support/seancode and launched it again and it worked this time.
Without loading a world yet, I was able to find Help -> Manage Definitions... which shows me that it has entity version 1.19.22w15a loaded. I hope that helps you!
It loads my 1.19.0 world and performs smoothly and quickly with it, too! My only wish is that it could zoom out more, but otherwise I've got no complaints.
"Zoom out more" -> use a modifier key while scrolling to allow zooming beyond 1:1 resolution (CTRL on PC, no idea if this works on MacOS)
Regarding update on MacOS, this is not a working proof. Release 2.19.0 does contain exactly these definitions, so they are copied from the release and not via update mechanism. We have to try again with the next changes.
(A proof would be: delete the definition folder, use release 2.18.0, do Settings->Check now, => get 1.19 definition files with older release)
- I deleted the Minutor app and the
seancodedirectory - I downloaded Minutor 2.18 (Minutor.MacOS-5.12.zip) and launched it
- Help -> Manage Definitions... shows 'entity' at 1.17.21w13a2
- I went into Preferences... and clicked the 'Check now' button; nothing seemed to happen
- 'entity' is still at 1.17.21w13a2
- I waited a few minutes then checked again; same entity version
- I quit Minutor and relaunched; now the entity version is 1.19.22w15a (all the versions now are as in my screenshot from yesterday)
Great! That was the proof I was waiting for. Thank you!
As explanation: the auto update is loading the most recent definitions here from GitHub and copy them to that local folder. But these files are only checked at start-up. So you can see them only after a restart.
Did you succeed with zooming out? When not, please open a new issue so we can solve that for MacOS.
You might want to add a note somewhere in the UI saying something like 'new definitions found, will take effect after restarting the app' so that users won't think nothing happened.
I'll open a new ticket for issues with zooming out. 😄