Implement ramdisk support

[extrowerk]

It would be great to have a ramdisk support for the working folders to speed the things up a bit and to not risk the health of the BeFS.

Haikuporter should then:

• create a ramdisk (size?)
• check for available working folder
• if an working folder exist for the version, then copy it to the ramdisk
• start to build
• at exit (return 0 or else) copy back the ramdisk contents (maybe optional)
• destroy the ramdisk

Ramdisk size could be the problem here, as there is still some ports where the build-system downloads some 3rdparty tarballs during the BUILD() so one cannot easily say: lets allocate 4x tarballsize for ramdisk.

Any ideas?

[pulkomandy]

If your worry is BFS corruption, we should put effort in making BFS reliable, not trying to avoid using it.

Copy to/from the ramdisk is probably going to make things slower. You can trust the existing RAM caching to do this more efficiently. This would only work if you extract the archive and init the git repo directly in the ram disk. I think this can easily be done outside haikuporter (create ramdisk, checkout haikuports there, start building things).

I suggest getting an SSD for increased performance.

[fbrosson]

@extrowerk, I build everything on a ramdisk. My haikuports tree is in a partition mounted on /data and I have these on my haikuporter.conf:

OUTPUT_DIRECTORY="/ramwork"
DOWNLOAD_IN_PORT_DIRECTORY="yes"
PACKAGES_PATH="/data/packages-x86_gcc2"
#PACKAGES_PATH="/data/packages-x86_64"
REPOSITORY_PATH="/data/haikuports/repository-x86_gcc2"
#REPOSITORY_PATH="/data/haikuports/repository-x86_64"

Whenever I retart my system, I run:

ramdisk create -s 768mb
mkfs -q -t bfs /dev/disk/virtual/ram/0/raw ramwork
mountvolume ramwork

[extrowerk]

@pulkomandy : i already got SSD, and it is a long time since i last met with any BFS corruption, but it is still somewhat slow to build big packages. Ok, my ULV cpu doesnt rock the house anymore :)

@fbrosson : thanks for the info, mate!

[pulkomandy]

This is because haikuporter only moves the hpkgs if the source and destination directories are on the same partition/filesystem.

We should fix this.

Also, don't hesitate to make your ram disk larger: it will allocate RAM only when actually needed, and you can reclaim the RAM using the fstrim command if you need it for something else.

[mmlr]

Holding regularly and recently used parts of the filesystem/disk in memory while balancing that against the other memory users is pretty much the definition of a filesystem and/or block cache. As it can grow/shrink as needed it doesn't have the associated problems of a fixed size ramdisk. If it doesn't do a good job at speeding things up, then these caches should be improved instead of finding workarounds.

Also if you format your ramdisk with BFS anyway, you don't really reduce the FS overhead or avoid BFS bugs. It boils down to the same as making a dedicated partition on disk and then just regularly reformatting it. If the goal is to really avoid BFS and its overhead then you should instead look at using an actual RAM filesystem.

Have you actually measured the performance difference? To make a real comparison you'd probably want to run <haikuporter> --clean <port> and <haikuporter> --nopatch <port> first to make sure that the download speed does not play into the numbers. Then run time <haikuporter> <port> to gather the actual build time. Doing that for the ramdisk and the normal disk a couple of times should tell what kind of performance difference there is (or isn't). If the difference is significant, then that should be raised as a Haiku bug, as it'd show that the caches aren't as effective as they should be.

[fbrosson]

At the time I switched to building on a ramdisk I was using an old 8GB USB2 flash drive, so the performance increase was really great. Now I'm on a 64 GB SDRAM, which is faster, but I'm quite sure the performance increase is still significant. Yes, I should make some measurements...

BTW, FWIW, I use the same multiboot SDRAM with Haiku (x86_gcc2h, x86h, x86_64) alternatively:

• on real hardware
• on VirtualBox+KVM, QEMU+KVM or even QEMU+Xen. (I recommend QEMU+KVM with -vga=qxl and spicec as graphic client ) By the way, FWIW, the trick to use a real device (e.g. /dev/sdb) with VirtualBox is to:

ln -s /dev/sdb $HOME/myusbdisk.hdd
sudo chmod a+w /dev/sdb

(that's because VirtualBox does not allow to choose /dev/sdb directly, but if you use a symlink with an extension it likes, then it accepts it ;-)