zos
zos copied to clipboard
threefoldtech
Automatically cache some larger flists
When deploying large flists like full VM images on the Grid, we commonly see that it takes longer than five minutes for Zero OS to download and start up the image, resulting in timeouts in the clients. While the timeout itself should perhaps be extended in the case of full VM or in general, we also have an opportunity to improve user experience on the Grid by programming Zos to automatically cache a subset of the larger (official) images before they are deployed for the first time.
To keep things simple, the list could just be hard coded and only updated with new releases of Zos. In case of updates to the flists, some periodic querying and refreshing would be needed.
I'd suggest to start adding the official full VM images and maybe some of the larger official apps.
A low priority "nice to have", of course. Just opening this to log the idea and make a space for discussion about this.
Agree, having the same experience when deploying larger flists. Restarting a deploy attempt (Terraform or Play) after the timeout mostly solves the issue, but is not a good experience.
that's a nice suggestion actually although kinda tricky to implement correctly
- Usually an flist in itself is very small, even for huge full VM images the flist can be few kilos. Downloading the flist itself takes no time.
- Accessing the files inside the Flist is what trigger the "file" chunks download and cache. So basically when starting a VM and then we are trying to copy the image file from the flist to the VM virtual disk, is what trigger file chunks download from the hub.
- So if we gonna warm the cache we need to automatically read all the files from the flist and dump the content, may be something like
cat /path/to/flist/image > /dev/nullto force download
All this is not a problem the problem is what is coming next:
- ZOS is programmed to
clean upany cached files that are not used accessed for 3 days. So we can warm up the cache but then it gets flushed again in 3 days.
May be we can work around this by only trigger the cache cleaner if we running out of cache space.
We can do it with this one here https://github.com/threefoldtech/zos/issues/1781
Tbh I feel like preloading the cache is not the way forward. This goes against the principles of flist in the first place, and only works around the original issue, being the fact that we have only a single hub, and remote locations have huge latency which makes the download speed exponentially worse.
In my opinion, the proper solution is to make sure the hub is available in multiple geographic PoP's, so that latency is always small and thus download speed is improved.
Yes, you are absolutely right. But preloading the cache is a "fix" that can be done now. unlike distributed hub that has NO plan so far.
Also it's common practice to prepare a readonly copy of the image on the node (or near the node) to be ready to be deployed when needed. Since we can't have the (near the node solution) may be we can do this for now.
I don't mind postponing this for the better solution, if a plan is in place.
- ZOS is programmed to
clean upany cached files that are not used accessed for 3 days. So we can warm up the cache but then it gets flushed again in 3 days.
I wasn't aware of this part. Just disabling the cache cleaner, with priority given to the official images, would at least help in the case that the same flist is deployed on the same node after more than three days.
To me, an ideal solution would involve the nodes sharing the data peer to peer, along the lines of IPFS or bittorrent. I think some subset of farmers would even volunteer their nodes to be the local initial seeds and be willing to set up some port forwarding to help peers behind strict NAT.
The main issue I see with simply replicating the Hub across locales is that this currently can't be done on the Grid, since availability of public IPs and gateways is little to none outside of Europe. Somebody needs to provision this capacity and pay for it, thus creating additional external dependency.
Disabling the cache cleaner is not a good idea because it means old VMs, or flists (that can also be big) that are no longer used on this node are never cleaned.
pre-loading the cache with some files might be a good idea as i already stated, but this can work well for small farms but for big farms this gonna be a problem.
As @LeeSmet said the better solution is to have distributed and regional hub. But tbh I have no idea about the state of this and if there is a plan to do it in the near future. there has been talking about it for long time 🤷🏼♂️
there are also no plans for IPFS. But bittorrent sounds interesting, i will keep in mind to look into.
The question now is if it's worth it to do the work to pre-load the cache:
- Nodes after they are updated will all try to warm-up the cache causing a lot of load on the hub
- Farms that has lots of nodes will redownload the same files for each node.
I think the solution for this problem is to have a distributed hub + the cache pre-load.
best way is to have better CDN for all the files, and have them much more aggressively cached, also we need to know in flists which files need to be hotloaded (means needed to start a VM) and they get downloaded in parallel
I brought this up a few hours ago, not sure if its just chance that Kristoff commented or word got to him. But VMs taking too long to download is just one of the few last big useability issues with deploying. I won't pretend to have the solution, but even a temp fix if it doesn't align with being decentralized is going to be helpful.
The slow VMs is only issue with flists that uses raw disk images (the flist only contains a vm image) this style of VMs is deprecated and should not be used. Unfortunately we will have to keep the support around for a while because of some VMs that actually using this. We should plan phasing it out of the system.
Well whatever the default flist for a full VM ubuntu install caused this for me just a few days ago.