tracker icon indicating copy to clipboard operation
tracker copied to clipboard
verified publisher icon turnkeylinux

How to solve the problem that when using the mirror source, the mirror is still downloaded from the turnkey server?

Open dunkeng opened this issue 1 year ago • 4 comments

I use proxmox virtual environment. I need to use the mirror source. By modifying the specified URL in /usr/share/perl5/PVE/APLInfo.pm From: https://releases.turnkeylinux.org/pve To: https://mirrors.ustc.edu.cn/turnkeylinux/metadata/pve

After refreshing, it was found that when downloading the CT template, it was still downloaded from the turnkey server Then I checked the aplinfo.dat file and found the following content: Location: http://mirror.turnkeylinux.org/turnkeylinux/images/proxmox/debian-12-turnkey-ansible_18.0-1_amd64.tar.gz

The file path in aplinfo.dat is written as an absolute path, so modifying the mirror configuration is invalid Is it possible to modify the content of the Location item to a relative path? Or are there other methods that I don't know to solve this problem?

dunkeng avatar Jun 28 '24 04:06 dunkeng

TBH, my colleague handles the aplinfo.dat updates so I'm not super familiar with how it all works.

Although as you note, the urls in the dat file are to our mirror network, rather than a specific member.

I recommend that you try modifying the aplinfo.dat itself - to point to the specific url you need/want to use. Hopefully that will work. Although even if it does, it will get auto updated and overwrite your changes. IIRC it's updated daily.

As I say though, I'm not 100% sure how it all works, so there is a chance that will fail.

You could add an entry to your PVE hosts file to ensure that mirror.turnkeylinux.org always points to the mirrors.ustc.edu.cn IP. It's not ideal and may break if the IP changes (I'm not sure how they do their mirroring stuff either) but might be a good option?

E.g.:

mirror.turnkeylinux.org    202.141.160.110

Worst case scenario, you could just manually download the template(s) you want. I.e.:

cd /var/lib/vz/template/cache/
wget http://mirrors.ustc.edu.cn/turnkeylinux/images/proxmox/debian-12-turnkey-ansible_18.0-1_amd64.tar.gz

I hope that helps.

JedMeister avatar Jul 01 '24 06:07 JedMeister

This is the configuration of PVE. It can be seen that the location uses a relative path. Currently, I am directly downloading the file.

http://download.proxmox.com/images/aplinfo-pve-8.dat Package: alpine-3.19-default Version: 20240207 Type: lxc OS: alpine Section: system Maintainer: Proxmox Support Team [email protected] Architecture: amd64 Location: system/alpine-3.19-default_20240207_amd64.tar.xz md5sum: fe35133232231ed5918c3828e91c4069 sha512sum: dec171b608802827a2b47ae6c473f71fdea5fae0064a7928749fc5a854a7220a3244e405fb5ad14d09f52e5325a399c903169a075cd2f6787326972094561f0a Infopage: https://linuxcontainers.org Description: LXC default image for alpine 3.19 (20240207)

Some netizens have asked the mirror server administrator whether the URL can be modified by the mirror server after synchronization, but due to the existence of the asc file, it will result in verification failure. https://github.com/ustclug/discussions/issues/327

dunkeng avatar Jul 01 '24 14:07 dunkeng

Ok, so I've had a closer look at this.

Obviously what you want is possible and on face value it's a trivial change on our end. However there are implications to be considered. I have a number of reservations that I'd need to discuss with my colleagues, plus a lot on my plate right now. One of the main reasons for it being the way it is, is because we control the release.turnkeylinux.org domain. So the files are always downloaded from out server; but the images themselves are downloaded from the mirror network.

Regardless, I'll leave this open as a feature request, but I'm not willing to give it any priority ATM.

I have already provided a number of potential workarounds. Beyond those already noted, if you're happy to continue on your current path, then why not edit the PVE APLInfo module some more? I'm not particularly familiar with perl, but I'm certain that it wouldn't be that hard to update our PVE index file post download and update/replace the download URLs.

Stepping back though, IMO you are going about this completely the wrong way. What you want to achieve is not unreasonable, but your current approach is a bit of a hack. End users manually editing files in /usr (with the exception of /usr/local) is generally frowned on.

Files in /usr/share & are controlled by package management and future package updates will overwrite your changes - meaning that you will need to redo your changes any time the relevant PVE package is updated. I'm not sure how often the PVE perl modules are updated, but regardless, your changes will not be persistent. IMO you'd be best to reconsider your whole approach.

To ensure that your changes survive an update to Proxmox deb packages there is a much better and more robust way to do what you want. It'd be more work initially and may still eventually break at some point, if/when Proxmox make significant changes. But it would be persistent and much more reliable than your current approach. It also doesn't break existing Debian/Proxmox assumptions. If there is any chance someone else will administer the server, then it would be much easier for them to discover what you've done. Glossing over the actual perl changes required (because I'm not sure on details), here is the steps I suggest:

  • Create a new local perl module based on PVE::APLInfo - i.e. /usr/share/perl5/PVE/APLInfo.pm.
    • Create a dir for the replacement perl module - e.g. /usr/local/share/perl5/CustomPVE.
    • Copy /usr/share/perl5/PVE/APLInfo.pm into that dir
    • Modify /usr/local/share/perl5/CustomPVE/APLInfo.pm as required to achieve your ends.
  • Create a new custom pveupdate script. There are numerous ways you could go about that, but this is my suggestion. Note that beyond the first step I'm not 100% sure on details, but this is the general gist:
    • Copy /usr/bin/pveupdate to /usr/local/bin/.
    • Ensure that /usr/local/bin is ahead of /usr/bin in your $PATH.
    • Towards the top of /usr/local/bin/pveupdate - before the use PVE::APLInfo; line - ensure that your new CustomPVE::APLInfo is findable by perl. I.e. ensure that perl looks for modules in /usr/local/share/perl5. I think the File::FindLib module can do that?
    • Change the use PVE::APLInfo; line to use CustomPVE::APLInfo;.
  • Test and tweak until it does exactly what you want by running pveupdate and modifying /usr/local/share/perl5/CustomPVE/APLInfo.pm as required.
  • Override the default pve-daily-update service
    • Create a new dir; /etc/systemd/system/pve-daily-update.service.d/
    • Create a file in that dir with a .conf suffix - e.g. custom-mirror-url.conf
    • In that file put this contents:
[Service]
ExecStart=
ExecStart=/usr/local/bin/pveupdate-custom
  • Load modified service and do final confirmation like this:
systemctl daemon-reload
systemctl start pve-daily-update
systemctl status pve-daily-update

Hopefully that's of value?

JedMeister avatar Jul 15 '24 06:07 JedMeister

Thank you very much for your help. I tried hard to write code to implement the function, but I really don't know programming. After trying for a while, I gave up temporarily because considering the low frequency of my CT template download, I temporarily dealt with it in a simple and crude way for the time being.

sed -i 's|http://mirror.turnkeylinux.org|https://mirrors.ustc.edu.cn|' /var/lib/pve-manager/apl-info/releases.turnkeylinux.org

dunkeng avatar Jul 25 '24 22:07 dunkeng

Apologies that my suggested solution was a bit too complex and apologies too if you spent too much time on it.

I like the simplicity of your workaround and whilst many might consider it a bit hacky, I think it's actually quite elegant. TBH, I wish that I'd thought of it!

The only issue with your workaround is that it needs to be run manually. However, tweaking the pve-daily-update.service will resolve that. Somewhat similar to the last part of my previous suggestion, but instead run your sed command automatically after the pve-daily-update.service runs. Make sure that you have removed the previous service tweak that I suggested before (if you haven't already) and then add a new tweak:

mkdir -p /etc/systemd/system/pve-daily-update.service.d/
cat > /etc/systemd/system/pve-daily-update.service.d/update-turnkey-releases.conf <<EOF
[Service]
ExecStopPost=/bin/sed -i 's|http://mirror.turnkeylinux.org|https://mirrors.ustc.edu.cn|' /var/lib/pve-manager/apl-info/releases.turnkeylinux.org
EOF

Reload systemd services (required whenever you change a service):

systemctl daemon-reload

Finally, test it. First move the default turnkey release file and start the update service:

mv /var/lib/pve-manager/apl-info/releases.turnkeylinux.org /var/lib/pve-manager/apl-info/releases.turnkeylinux.org.old
systemctl start pve-daily-update.service

That will redownload the turnkey release file and run your sed command afterwards.

The final check is to count the number of times the turnkey mirror exists in the freshly downloaded releases.turnkeylinux.org file:

grep -c http://mirror.turnkeylinux.org /var/lib/pve-manager/apl-info/releases.turnkeylinux.org

I get 0 :grin:

And just to be 100% sure, check for your mirror url:

grep -c https://mirrors.ustc.edu.cn /var/lib/pve-manager/apl-info/releases.turnkeylinux.org

I get 110 for that one! :tada:

JedMeister avatar Jul 27 '24 03:07 JedMeister

I'm going to close this now. Good luck with it.

JedMeister avatar Jul 27 '24 03:07 JedMeister