Multiple improvements, fixed up conflicts

[lj3954]

Includes PRs:

• #860 - Replace macrecovery with inbuilt downloading & alternate method of verification
• ~~#875 - Fix error where VMs fail to launch when running command from outside VM directory~~
• #879 - Fix quickget when wget2 replaces wget with a symlink (Fedora is experimenting with this, other distros usually follow); prioritize wget2 over wget for downloading if installed on system
• #880 - Add new release Zorin OS 17
• #884 - ~~Implement "download-iso" option which only downloads the ISO (or other image type) into the current directory, without creating the VM config.~~ Also add windows & macOS support for download-iso, not included in original PR because of dependence on #860
• Added sha256sum image verification for consumer windows (8, 10, 11)
• Added languages for Windows back
• Added Windows Server 2012R2, 2016, 2019, and 2022

Previous pull requests closed, due to the extra improvements I've added here on top of the work in those branches.

[zen0bit]

I already included some of your PRs (Thanks for contributing) in my bigger one check it... #885

PS: If you can exclude changes in my PR please. Include just the rest and check against my branch?

To make it bit easier for Wimpy and quicker for us 😉

[lj3954]

That should do it.

[zen0bit]

LGTM

[lj3954]

Amended README to include download-iso and replace outdated warning that show-iso-url doesn't work for specific operating systems (no longer accurate). a037322

[zen0bit]

Currently I'm working on support for Windows Server releases (#843) as well as support for different languages of Windows ISOs in quickget on this branch. I discovered that checksums are on Microsoft's download page, and it's quite important to verify file integrity, so I'm adding it to this PR so it can be merged before I finish up on the other changes.

windows languages was already in quickget if you missed... look few versions back. If can help you with reimplementation.

If you already know about, or is useless for you, just ignore this comment.😉

[lj3954]

Windows 8 wasn't there previously (it supports a different set of languages), and I don't really like the array solution. Currently my best idea is to just make the languages "Editions" and make it act just like any other OS. Also the parenthesis can't be inputted without quotation marks (which is bad UX) so I'm using dashes to replace spaces & removing parenthesis.

You can look at my current progress here. https://github.com/lj3954/quickemu/tree/winserver

[lj3954]

Fixes #843 Fixes quickemu-project/quickgui#116

[lj3954]

Windows 10 & 8 fail verification because Windows 11 hashes are used. Working on diagnosing the issue.

I'm not sure there's a solution for this. Perhaps image verification will be Windows 11-only, unless there's an alternate page that can be used (such as the regular consumer download page, rather than this api).

[lj3954]

Fixed the previous issue by only checking hashes for Windows 11.

[TuxVinyards]

I have to close my eyes to what are now 1k+ shellcheck warnings and errors on the quickget script.

Thought this one might need flagging up though.

Also quotes on the echo lines ...

:heart:

[TuxVinyards]

Any ideas here? Happens by the point I have got to the boot/recovery drive selector first stage bit.

qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]

Took three attempts to get the installer to work. Although that was before last nights commits ... Otherwise, Sonoma does boot for me and seems to be okay. FWIW

Edit: I also get

usb_desc_get_descriptor: 2 unknown type 33 (len 10) usb_desc_get_descriptor: 1 unknown type 33 (len 10)

after logging in.

[lj3954]

qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.hle [bit 4] qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.rtm [bit 11]

I've also seen this with quickemu macOS VMs. My pull request has nothing to do with it. Those warnings did not show for me on a laptop with an Intel 1135g7, but it does show on both my desktop and laptop with AMD processors. I'm not sure exactly what causes it. I don't believe performance is affected by this in any way.

Took three attempts to get the installer to work. Although that was before last nights commits

Did it give you any information? I installed Sonoma on two different machines and didn't get any errors. Yesterday's changes just fix the variable I forgot the $ on, as well as improving the formatting.

[TuxVinyards]

This is all on a series 11 intel with avx2, and avx512 ... There were a few things that I noticed.

First, and probably the most dis-concerting, was that I used to be able to have an error free working version of Mojave. I retried Mojave this morning and found that it now won't boot. The only thing it does now is load the installer and tell me that I have 47 mins to go ...

On the previous three sonoma attempts:

The first one showed me two qemu drives. One was large, the other quite small. I formatted the larger, in the usual way and started the install. It shortly after crashed. Said something about not being able to find a recovery server.

After that it wouldn't restart until I had deleted 'disk.qcow2'

As I used qqX which logs the boot args, I did later notice that on the first boot quickemu had tried to use the Penryn formula without avx2. The subsequent boots used the Haswell one.

Moving on, second time around, quickemu created a new main drive for me but I was presented this time with the large drive being labelled as Apple. I formatted again and it went ahead. At the 'finished in about 40 mins' mark I went for a break.

When I came back it had crashed. I can't remember the error. But the restart took me to the 29 mins point and carried on until it said less than a minute to go. It then sat at this point for over 10 mins with no no hard drive activity light showing, so I reset it. Unfortunately I found myself back at the 29 mins point again.

Third try, I left it at the less than a minute point for probably an hour or so and it finally managed to sort itself out.

[TuxVinyards]

Looking at

function check_cpu_flag() { local HOST_CPU_FLAG="${1}" if lscpu | grep -o "^Flags\b.*: .*\b${HOST_CPU_FLAG}\b" > /dev/null; then return 0 else return 1 fi }

I wonder if just double grepping might be the better route:

[TuxVinyards]

Little bit of good news. I got Mojave back. Not sure what went wrong there. I think it was down to the difference between the Mojave and the Sonoma boot screens that was the cause:

On the Sonoma one, the first icon on the left was the main drive to boot up. On Mojave the first one is the same colour and similar to look at but is actually the installer being presented as default first option .... :roll_eyes:

On the negative side, I now have the USB error "usb_desc_get_descriptor: 2 unknown type 33 (len 10)" as with Sonoma.

But (edit) I have that USB error now, even when I switch back to the current version quickemu 4.9.2 .... This could be just down to recent changes to qqX whereby it is now highlighting qemu errors a bit more.

Just need to fix function 'check_cpu_flag' so it works properly:

xxx@System-X:~$ Test="avx2"

xxx@System-X:~$ lscpu | grep  "Flags" | grep -o "$Test"

avx2

[TuxVinyards]

@lj3954 Going carefully through what you have done, I have to say that this is quite a big contribution you have made. :rocket:

If I knew enough about GitHub's intricacies I would PR a commit for you to help. However ...

I have managed to solve the USB problem. This comes down to specifying sse4.2 in the -cpu line. We are only testing for and need 4.1 anyway. Change that parameter and the error stops. You also had '+avx2' in there twice btw.

On the cpu feature warning, my thoughts are that something is not being enabled. I have Intel hard metal, so using just "-cpu host" works as a cure for me. Obviously not necessarily good on all Intel. Certainly not good on AMD.

It is only a warning but it would be nice to fix it. Something in Sonama is requesting something that is not present in Mojave is my feeling on this.

Lastly, I tested a rewrite of function check_cpu_flag. The -q option will make grep return exit codes, so all we actually need is a simple one liner.

function check_cpu_flag { lscpu | grep "Flags" | grep -q -o "$1" ; }

Hope this all helps.

Also, some other experimental bits and notes:

                if [ "${HOST_CPU_VENDOR}" == "GenuineIntel" ]; then
                    CPU="-cpu host"
                else
                    # CPU="-cpu Haswell,kvm=on,vendor=GenuineIntel,+sse4.1,+aes,+xsave,+avx,+avx2,+xsaveopt,+xsavec,+xgetbv1,+bmi2,+smep,+bmi1,+fma,+movbe,+invtsc"
                    # CPU="-cpu Broadwell,kvm=on,vendor=GenuineIntel,+ssse3,+sse4.1,+avx,+avx2,+aes,+xsave,+xsavec,+xsaveopt,+xgetbv1,+bmi1,+bmi2,+smep,+fma,+movbe,+invtsc"
                    # CPU="-cpu Haswell,kvm=on,vendor=GenuineIntel,+sse4.1,+avx2"
                    CPU="-cpu Broadwell,kvm=on,vendor=GenuineIntel,+ssse3,+sse4.1,+avx,+avx2,+hypervisor,+kvm_pv_eoi,+kvm_pv_unhalt,+popcnt,+aes,+xsave,+xsavec,+xsaveopt,+xgetbv1,+bmi1,+bmi2,+smep,+fma,+movbe,+invtsc,check"
                fi
                # CPU="-cpu host" # works on intel hardware without causing cpu feature warnings (tested on series 11)
                # See  https://unix.stackexchange.com/questions/710944/qemu-warning-host-doesnt-support-requested-feature-cpuid-80000001hecx-svm-b
                # https://en.wikipedia.org/wiki/List_of_Intel_Core_processors   
                # https://www.reddit.com/r/hackintosh/comments/141wnjk/state_of_macos_14_sonoma_on_x86/?rdt=33428

And, edit, one after thought:

I did a web search for the term 'chunkcheck' and found it to be quite popular. If you just place the 'chunkcheck' script in /usr/bin as is, you could find that you are not the only one wanting to use a file with that name. You risk your file being overwritten, or you could end overwriting somebody else's work that hasn't been thought through properly.

The file 'chunkcheck' needs to be named more uniquely. Something like 'quickget-mac-chunkcheck' maybe.

[TuxVinyards]

This is starting to feel like a bit of a monologue but there are PROBLEMS that need fixing here.

The MacOS installer has never been good ....

CPU:

A web search: https://duckduckgo.com/?t=ffab&q=CPUID.07H%3AEBX.rtm+%5Bbit+11%5D&ia=web shows us that Intel links the current CPU warnings to TSX.

And qemu-system-x86_64 -cpu help tells us that we can use Broadwell-noTSX and Broadwell-noTSX-IBRS

My tests show me that -noTSX will cure the CPU problems.

Extensive guidance from Qemu docs also seems to recommend IBRS in order to mitigate against Spectre problems.

https://qemu.readthedocs.io/en/v8.1.0/system/i386/cpu.html#important-cpu-features-for-intel-x86-hosts

There is also guidance to suggest using tsx-ctrl as a parameter but I have yet to explore this. Currently I am using:

CPU="-cpu Broadwell-noTSX-IBRS,kvm=on,vendor=GenuineIntel,+sse,+sse4.1,+avx,+avx2,+hypervisor,+popcnt,+aes,+xsave,+xsavec,+xsaveopt,+xgetbv1,+bmi1,+bmi2,+smep,+fma,+movbe,+invtsc"

It also looks like it is time to drop Penryn as other references state we should avoid using core-duo, to which Penryn belongs.

https://qemu.readthedocs.io/en/v8.1.0/system/i386/cpu.html#other-non-recommended-x86-cpus

USB:

Unfortunately, the above will reignite the USB errors, and even the adjusted sse4.2 entry and tweaking the ssse3 entry, this doesn't seem to sort it. But, so far and subject to further tests, they can be cured by adding

mouse="virtio"
keyboard="usb"

to the .conf file. This works for both Ventura & Sonoma. For Mojave `mouse="usb" is required. I have yet to test other installs.

CHUNKCHECK:

Aside from my earlier comments about the choice of filename, the updated readme says that we need to install chunkcheck as an independent piece of software. Do these notes need updating? (I also can't find any windows server notes BTW. ??)

The given link only really seems to point to a gist where the author of our current system of 'macrecovery' praises the comment that forms the basis of the code that you have now supplied. https://gist.github.com/MCJack123/943eaca762730ca4b7ae460b731b68e7?permalink_comment_id=3920890#gistcomment-3920890

NOTE:

These tests were all completed with the 'check_cpu_flag' rewrite being used ....

[flexiondotorg]

Closed via #986