chimeraos
chimeraos copied to clipboard
ChimeraOS
each update is 1.1 GB or more in size
Updates are distributed as full system images which are currently about 1.1 GB in size, but this could grow in the future. So even if there is only one file changed in the update, the user must download the full image to get the update.
We could publish and use image diffs between each version increment so there wouldn't be a need to download the full update when updating by a single version.
I'm not exactly sure how you've implemented the update system, but if they're ISO images, you might be able to use something like zsync to sync diffs.
My initial attempts were using ostree, however I could never get it to work. Ostree also has distribution issues. Because it works like git, downloads are very slow. It was quite painful when working with it. I did read in a blog post that they are working on fixing this.
The solution, since we are using btrfs is to use that technology. I believe btrfs has the ability to apply subvolume differences.
However, I dont want to go down that path yet, because I want to explore alternatives to using btrfs first.
That makes sense. Hopefully they'll fix it, because it seems really promising.
btrfs send supports an "incremental stream", which references a parent. This could make smaller updates without introducing a new tech, but the build process and the version management becomes quite complex.
Something different I noticed is, that you create a stream with btrfs send, then use tar to package and compress it in another step. Why do you use tar at all? I did a small test: when you pipe the btrfs stream directly into xz (without tar), you'll get an image which is only 855MB instead of 1.2GB. Of cause this is only a small gain, but it's very easy to achieve.
Yes, I know about btrfs incremental update. I dont think it would be too bad to do, but I actually want to move away from using btrfs in the future.
Using tar makes it easy to use different compression algorithms by just changing a flag.
The reason for the size difference is because the images are actually compressed with bzip2 because I messed up the command line options. This will be fixed soon.
Some time ago i was experimenting with http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html.
My initial attempts were using ostree, however I could never get it to work. Ostree also has distribution issues. Because it works like git, downloads are very slow. It was quite painful when working with it. I did read in a blog post that they are working on fixing this.
The solution, since we are using btrfs is to use that technology. I believe btrfs has the ability to apply subvolume differences.
However, I dont want to go down that path yet, because I want to explore alternatives to using btrfs first.
There is a system similar to what Silverblue does and to what you do with gamerOS, which uses Btrfs, and it is microOS: https://en.opensuse.org/Portal:MicroOS https://kubic.opensuse.org/documentation/transactional-update-guide/transactional-update.html Why do you disable COW on Btrfs? in this way all advantages and functionalities of Btrfs are lost, mainly: compression, checksumming.
@emanuc COW is disabled because it is known to cause performance issues with games.
MicroOS seems interesting, thanks for sharing.
@alkazar Hi! So, still about COW: where do games "live"? If it's in /home and/or /var, which as far as I understand are already separate subvolumes, you can set (keep?) noCOW for them, and enable COW for / (that is pretty much what we are doing in MicroOS too)
@emanuc COW is disabled because it is known to cause performance issues with games.
MicroOS seems interesting, thanks for sharing.
This is my first time reading that Btrfs creates performance issues in games. I have several games on Steam, when I made the switch from ext4 to btrfs I did the comparison to see if there were any performance differences, I tried it with Counter-Strike: Global Offensive and some other games and I didn't notice any performance differences , and I switched to Btrfs. To date I have many games on a Btrfs disk, the Steam folder with about 80 games, on a subvolume with ZSTD level 3 compression option, I have no performance issues in any game. You can always create a subvolume and set "nocow", compress or nocompress etc. Hope this helps.
/dev/sdb1 on /media/emanu/dati/game type btrfs (rw,noatime,seclabel,compress=zstd:3,space_cache=v2,autodefrag,subvolid=1435,subvol=/@game)
@dfaggioli Games are indeed in a separate subvolume under /home. However, unless something has changed recently, BTRFS does not allow differing options on only some subvolumes. nocow on one subvolume means nocow on the entire BTRFS partition/disk.
@emanuc I have seen benchmarks showing that loading times degrade significantly over time with cow enabled, which I think makes sense especially if the game is updated a lot. I don't know where I saw this and can't seem to find it now though.
@alkazar COW is (let's say) per-directory. You can set nocow for a directory inside a subvolume, or for the directory inside which you mount a subvolume, which will basically make the subvolume "nocow". E.g., I just did:
# mkdir media ; mksubvolume --nocow /media
And I can now see with lsattr that the /media directory, where the subvolume is mounted, it's "nocow" ("C" attribute). And that applies to every file or directory that I will now create inside that dir. Which basically means it applies to the entire subvolume.
In MicroOS, we do exactly that. Root (i.e., / ) is read-only, and we have COW on for it (and we use snapshots heavily, for something that we call "transactional updates"). On the other hand, /home and /var, that are subvolumes, are "nocow".
@dfaggioli Games are indeed in a separate subvolume under /home. However, unless something has changed recently, BTRFS does not allow differing options on only some subvolumes. nocow on one subvolume means nocow on the entire BTRFS partition/disk.
It's simple to do, on a folder/subvolume or by file:
sudo btrfs subvolume create folderX
For no COW:
sudo chattr +C folderX
Compress:
sudo btrfs property set folderX compression zstd
No compress:
sudo btrfs property set folder X compression none
Other: man btrfs property
You can create a subvolume per user, without root permissions, which you can remove with a simple "rm", in this case it is a nested subvolume and you don't need to specify mounting on fstab:
btrfs subvolume create folderY
rm folderY
However, if you take snapshots on the parent subvolume, it is better to create the subvolume on the toplevel, therefore flat. This is because when you restore the subvolume, you would not find the contents of the nested subvolume.
I used to work at Canonical on Snapcraft and snaps. snapd uses xdelta to deliver incremental upgrades from one revision of a snap to another.
xdelta is simple to work with and there were concerns that it wouldn't offer sufficient efficiency in delta sizes, but some testing revealed it offered significant savings in most cases.
- https://github.com/jmacd/xdelta
I just tried xdelta between the 33 and 34 release. Unfortunately, the delta file was larger than the full file.
Been experimenting with zsync. The original wont work with https links (is THAT old) so a newer project ported parts of it to work with libcurl and AppImage picked it up to make a new version (highly experimental and non-funcitonal from their README). Aside for the disclaimer, the project works:
How to:
1- Install zsync2: https://aur.archlinux.org/packages/zsync2-git
2- "Server side" make a zsync file:
$ zsyncmake2 -u <url of the chimeraos.img.tar.xz file> <path chimeraos.img.ta.xz file> -b 32768
The -b is important, lower it and Github may trigger, higher and the chunks may be too big and it could be useless. this will produce a zsync file with the url and the precalculated hashs for each chunk.
3- "Client side" download the update for the file using the previous file as a seed:
$ zsync2 <zsync file i.e. chimeraos_v34.img.tar.xz.zsync> -i <seed file, i.e. chimeraos_v33.img.tar.xz>
The good part is the zsync files can be created after the fact or even in another server from the image file. It contains the hash for sync incrementally and the download url.
I think the image will continue to grow as long as we add features, we are currently at 1.4GiB give or take, is what most live distributions ISOs are nowadays. A potential solution is to make some kind of add-on system within the chimera app to download emulator cores on the user's home that can be downloaded on-demand as we do with custom Proton versions.
All efforts to make the download smaller instead of the image will depend on how often we build them and how different they are. The zsync method above is a solution that would work if we can do smaller windows of releases (say 1 week as an educated guess).
IS this still an issue? Since games are in the 10s of gigs or more I see this as a non-issue now.