Building on X86_64 and in a Virtual Machine

[openoms]

working branch: https://github.com/openoms/raspiblitz/tree/x86v1.2

Images for the Atomic Pi: https://www.digital-loggers.com/downloads/index.html#API_IMAGES

Ubuntu 18.04 barebone: http://www.dlifiles.com/atomicpi_ubuntu_bionic_bare_1.0.0.138.zip user and password: atomicpi

Build SD with: wget https://raw.githubusercontent.com/openoms/raspiblitz/x86v1.2/build_sdcard.sh && sudo bash build_sdcard.sh x86v1.2 openoms

Updated with v1.2 releas from @rootzoll and more: https://github.com/rootzoll/raspiblitz/pull/597

[openoms]

For VirtualBox: https://superuser.com/questions/554862/how-to-convert-img-to-usable-virtualbox-format VBoxManage convertfromraw --format VDI atomicpi_ubuntu_bionic_bare.img atomicpi_ubuntu_bionic_bare.vdi and mount as HDD

Did not manage to ssh to the AtomicPi VM yet despite setting up port forwarding in VirtualBox. https://stackoverflow.com/questions/5906441/how-to-ssh-to-a-virtualbox-guest-externally-through-a-host I does not seem to have an eth connection calling ifconfig.

UPDATE: ssh works well and correct IP is displayed if setting the Network Connection to Bridged in VirtualBox. (just make sure there is no openVPN active on the machine - connect to the VPN on the router if needed)

[openoms]

Testing Ubuntu Bionic mini.iso for VirtualBox: https://help.ubuntu.com/community/Installation/MinimalCD#A64-bit_PC_.28amd64.2C_x86_64.29_.28Recommended.29 http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/mini.iso

To make a .vdi hard disk from the iso: $ VBoxManage convertfromraw mini.iso mini.vdi $ VBoxManage modifyhd mini.vdi --resize 32000

start the VM with the mini.vdi Install the minimal image + openSSH (sudo apt-get install openssh-server -y)

• To spare the installation steps there are prebuilt VM images here: https://www.osboxes.org/virtualbox-images/

add an additional dynamicly allocated disk of at least 500 GB (this can be stored on an external HDD)

port forwarding for ssh (set up on the host machine): $ VBoxManage modifyvm Ubuntu-RB --natpf1 "ssh,tcp,,3022,,22"

Connect from a host terminal: $ ssh <user>@127.0.0.1 -p 3022

Build SD with: wget https://raw.githubusercontent.com/openoms/raspiblitz/x86v1.2/build_sdcard.sh && sudo bash build_sdcard.sh x86v1.2 openoms

login with admin: $ ssh [email protected] -p 3022

To copy the blockchain: Copy from the local machine works with: sudo rsync -avhW -e 'ssh -p 3022' --progress ./chainstate ./indexes ./blocks [email protected]:/mnt/hdd/bitcoin Blockchain sync works and is fast depending on the host machine. rtorrent does not work cloning another hdd does not work for now.

Tor does not work for now.

• The installed RaspiBlitz .vdi can be converted to a raw image and burnt to a USB/SDcard with Etcher to boot a computer directly: VBoxManage clonehd --format RAW Ubuntu18.04.01LTS.vdi RAW_Image_to_disk.img

• To use an external disk (for example a Raspiblitz HDD) in the VM use this command: $ VBoxManage internalcommands createrawvmdk -filename BLOCKCHAIN.vmdk -rawdisk /dev/sdb1 RAW host disk access VMDK file BLOCKCHAIN.vmdk created successfully.

• On Ubuntu 18.04 desktop got an error when creating .vmdk: Cannot open the raw disk '/dev/sdf': VERR_ACCESS_DENIED VBoxManage: error: The raw disk vmdk file was not created Solution: User should be in a disk group to access raw partitions https://unix.stackexchange.com/questions/284704/verr-access-denied-when-trying-to-add-a-raw-disk-to-virtual-box sudo usermod -a -G disk $USER log out and back again and .vmdk creation will succeed

[fluidvoice]

You can also boot VirtualBox from a connected USB drive. The same USB drive swappable between VBox and various x86 devices is helpful.

Burn using Etcher mini.iso onto the plugged in USB drive (device /dev/sdb in my case). To make .vmdk disk file from the USB drive: $ vboxmanage internalcommands createrawvmdk -filename ./miniblitz.vmdk -rawdisk /dev/sdb RAW host disk access VMDK file ./miniblitz.vmdk created successfully.

Create a new VM with the .vmdk file as drive, then start the VM. Run Ubuntu install. Choose mostly default settings. Allow drive partition expansion, etc.

[fluidvoice]

Testing Ubuntu Bionic mini.iso for VirtualBox:

@openoms what is your intention for using this mini.iso? You will use it to install Raspiblitz requirements sure, but what desktop if any and what minimal-base app set do you think is the best to include by default?
I was thinking Lubuntu. It would be useful I think, if we could (eventually) create a Grub menu for choosing to boot up as either a LN/Blitz node or Linux desktop.

[fluidvoice]

From within the booted VM...

$ wget https://raw.githubusercontent.com/openoms/raspiblitz/x86v1.2/build_sdcard.sh $ chmod +x build_sdcard.sh $ ./build_sdcard.sh This progressed for a while but failed before completing. Output Gist is here

[openoms]

From within the booted VM...

$ wget https://raw.githubusercontent.com/openoms/raspiblitz/x86v1.2/build_sdcard.sh $ chmod +x build_sdcard.sh $ ./build_sdcard.sh This progressed for a while but failed before completing. Output Gist is here

You need to run the build_sdcard.sh as root. Also need to use the parameters to download scripts from the correct branch.

The command is: $ wget https://raw.githubusercontent.com/openoms/raspiblitz/x86v1.2/build_sdcard.sh && sudo bash build_sdcard.sh x86v1.2 openoms or same effect: $ wget https://raw.githubusercontent.com/openoms/raspiblitz/x86v1.2/build_sdcard.sh $ chmod +x build_sdcard.sh $ sudo ./build_sdcard.sh x86v1.2 openoms

I have added logic to look for a hdd on sdb as well, because that is where the second disk is mounted in VirtualBox: https://github.com/openoms/raspiblitz/commit/7c98d939ba8a45efb8c7e3f99fbfa6de2fb720e4

[openoms]

Testing Ubuntu Bionic mini.iso for VirtualBox:

@openoms what is your intention for using this mini.iso? You will use it to install Raspiblitz requirements sure, but what desktop if any and what minimal-base app set do you think is the best to include by default? I was thinking Lubuntu. It would be useful I think, if we could (eventually) create a Grub menu for choosing to boot up as either a LN/Blitz node or Linux desktop.

I treat the AtomicPi and the VirtualBox as an SBC only. You either run a full node or a desktop. So I just keep the command line on either. The raspiblitz is a dedicated Bitcoin full node ad Lightning OS essentially with the screen only used for status display. If you want to use a desktop simultaneously try the Node Launcher: https://github.com/lightning-power-users/node-launcher.

I don`t think it would make sense to use GRUB to choose where to boot (apart from testing in your individual case). The RaspiBlitz is supposed to be always on.

[fluidvoice]

Testing (on LInux) the build with one of the AtomicPi images (Ubuntu Bionic Lxde) written to USB drive: **[EDIT:] tl;dr didn't work. build_sdcard script worked OK but node failed to come up and I don't want to try to debug it. Stick to CLI only, no desktop.

https://gist.githubusercontent.com/fluidvoice/2d081a86b52bc98b0be84f08de7b4d89/raw/bfbda354749a1b84a33a03aa1c1184e1769a8e65/api-fails.txt

[fluidvoice]

Testing Ubuntu Bionic mini.iso for VirtualBox:

@openoms I also did this now (CLI only) - with the same method I did above to use a plugged in 32GB USB drive as the destination drive. I had to change the VM network setting from NAT to Bridged to get an IP on my LAN (192.168.x.x vs 10.0.x.x) When booted up to command line I added the missing command ifconfig to show ip address with: sudo apt install net-tools I attached my HDD containing the blockchain to the VM using the same method and it's working fine: vboxmanage internalcommands createrawvmdk -filename ./blockchain.vmdk -rawdisk /dev/sdc Note: the device /dev/sdx may change between boots or plug/unplug so you need to verify that the .vmdk file has it set correct and if not edit the file: grep "/dev/" blockchain.vmdk

[fluidvoice]

• To spare the installation steps there are prebuilt VM images here: https://www.osboxes.org/virtualbox-images/

None of these images are "minimal" and almost all are over 1GB in size, including desktops and lots of programs not needed for Raspiblitz. This increases the time used to complete installation a lot.

[fluidvoice]

Debian 9.9 minimal net-install Download ~300MB net-install ISO

Install to VirtualBox .vdi destination drive: This took a while and installed a total of 3.9GB including the Debian desktop even though I chose to install nothing extra besides SSH server. Not a very minimal install.

Get IP Address of the VM: ip addr show SSH into it as as user you created during install: ssh user@ip_addr

Download build script: wget https://raw.githubusercontent.com/openoms/raspiblitz/x86v1.2/build_sdcard.sh

There is no sudo by default if you entered a root password during install. Login as root super-user (with root password): su - Run the build script: root@node:~#bash build_sdcard.sh x86v1.2 openoms

##exit root/su mode

[openoms]

https://github.com/openoms/raspiblitz/commit/e44308e4a92a13a5c4850bb4851da05a5b5db276

Debian 9.9 minimal net-install Download ~300MB net-install ISO Install to VirtualBox .vdi destination drive: This took a while and installed a total of 3.9GB including the Debian desktop even though I chose to install nothing extra besides SSH server. Not a very minimal install.

Yes, the net install images are small to download but bloat when installed. Same with the Ubuntu mini.iso, but that is still much smaller than the Debian. I am sticking with Ubuntu.

Also updated to the new lnd v0.6.1-beta (the RC2 is not available any more).

[fluidvoice]

I am sticking with Ubuntu

Good move. I'm finding Debian (not to mention other distro's) to be less straight forward or problematic. I'll prob return to focus on Ubuntu also.

[openoms]

FYI. The Ubuntu mini.iso once converted to a Virtualbox .vdi and installed and running Raspiblitz... I then tried to convert the .vdi to .img and using dd to write it to USB, which worked and looked good. All the files were seen by Linux file browser including in the /boot dir. But it would not boot; wasn't recognized by my notebook's BIOS. Also gparted did not recognize the partition details correctly. Some partition flags or other HDD config perhaps is not correct in the resulting drive.

/usr/bin/vbox-img convert --srcfilename miniblitz.vdi --dstfilename test.img --srcformat VDI --dstformat RAW sudo dd if=./test.img of=/dev/sdc bs=1024k oflag=dsync status=progress Using Etcher to write the .img also did not work

Can you try: VBoxManage clonehd --format RAW miniblitz.vdi test.img

And write that .img to the SD/USB with Etcher or dd.

[fluidvoice]

Can you try: VBoxManage clonehd --format RAW miniblitz.vdi test.img

And write that .img to the SD/USB with Etcher or dd.

There is no difference. vbox-img does the same thing as clonehd (now called clonemedium) command. It's just a separate binary.

[openoms]

These commands worked for me previously on my laptop converting a virtual Ubuntu 18.04.1 to a bootable USB. Does your USB boot if you write the original mini.iso (http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/mini.iso) to it?

[fluidvoice]

These commands worked for me previously on my laptop converting a virtual Ubuntu 18.04.1 to a bootable USB. Does your USB boot if you write the original mini.iso (http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/mini.iso) to it?

Yes, mini.iso would boot when burned using dd but only boots in BIOS (legacy) mode With the BIOS set to UEFI mode, the USB drive was not recognized.

VDI converted to IMG also booted, so not sure why it would not before: mini.iso install to "miniblitz" VM then shutdown => miniblitz.vdi convert to raw IMG file miniblitz.img => miniblitz.img dd to USB 31GB => Just one single /dev/sdb1 30GB Ext4 (w/no boot or other flags were set on the partition)

FYI, installing the rEFInd boot manager onto the system HDD or another USB drive allows choosing at boot-up time to boot either the miniblitz USB drive or any other distro's installed on other system partitions.

[fluidvoice]

VirtualBox Notes: Backing up the blockchain USB (.vmdk) drive and switching to it: Note: the VM /etc/fstab file must be edited to reflect the new drive's UUID

Create blockchain backup to another drive with rsync (source trailing backslash required): sudo rsync -rv -pogx --update --progress /mounted/path/BLOCKCHAIN/ /mounted/path/BLOCKCHAIN1

Optional - verify backup on host computer by syncing with bitcoind. sudo ./bitcoind -datadir=/mounted/path/BLOCKCHAIN1/bitcoin && sudo tail -f /mounted/path/BLOCKCHAIN1/bitcoin/debug.log

Create .vmdk shim for the backup drive and set it in the VM config and start the VM: vboxmanage internalcommands createrawvmdk -filename ./blockchain-sdb1.vmdk -rawdisk /dev/sdb1

VM startup fails. journalctl -xb | grep "failed for File System Check" (full log: https://pastebin.com/Hqi6iURt)

systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/8ece4d87-2aef-4566-b108-4cc517b89d1a.

The reason is /etc/fstab in the VM (.vdi) is set wrong - to old HDD:

UUID=8ece4d87-2aef-4566-b108-4cc517b89d1a /mnt/hdd ext4 noexec,defaults 0 2

but the new HDD has UUID: 154dd9a7-82f0-4176-8cd2-444bdb512b67

Solution: Boot into grub rescue and correct /dev/fstab blkid to show UUID's of all disks nano /etc/fstab to correct the file.

[fluidvoice]

ATOMIC PI notes: Inadequate power supply (low voltage) might show up as "No boot media found" during bootup as perhaps the eMMC is not powered up.

Run dd to backup the eMMC image but better wait to run it when booted up from an SD or USB drive. To enter BIOS settings hit the ESC (or DEL) key when bootup splash screen is seen.

Mini.ISO will boot in Legacy/BIOS mode only. AtomicPi is UEFI only - so SD card w/mini.iso is not detected/seen by the BIOS

Edit: there might be some USB burner that does it correct? well this is the manual way:

Ubuntu 18.10 "Cosmic" netboot mini.iso Ubuntu 19.04 "Disco" netboot mini.iso Make the Ubuntu 18.04 LTS "Bionic" netboot mini.iso UEFI bootable by the APi: (manual method) #. install 7z if not present sudo apt install p7zip-full

#. extract the ISO to a directory (./usbdisk) 7z x -ousbdisk mini.iso

#. extract efi.img as well 7z x -ousbdisk usbdisk/boot/grub/efi.img

#. format a USB drive MSDOS partition as FAT32; set partition flags ESP and Bootable #. copy all the files from ./usbdisk to the USB drive bootable partition #. this should work for a bootable SD card or USB drive

[fluidvoice]

VAGRANT notes:

Research - Vagrant is a better method of delivering easy to run VM "box" to non-technical users?

Vagrant - tool to easily build consistent VM development environments on Virtualbox, VMWare, Docker, etc.: 3 part, short YT vid intro

[openoms]

@fluidvoice I think it would be best to have a VM image (just like the SDcard image) and spin that up every time in a clean state when you want to test. I don't think that the Virtual Machine can be stable enough for multiple restarts and HDD changes. So would recommend to

• install a clean mini.iso (I store a copy of that as well)
• run the build_sdcard.sh and power off the VM.
• make a copy of the VM image and that will be your own clean state RaspiBlitz VM image. It will start up as a fresh install every time you boot a fresh copy.

[openoms]

Good news is the Armbian custom build setup uses the same Ubuntu Bionic mini.iso and also has some clues about Vagrant and VirtualBox: https://github.com/armbian/build

[fluidvoice]

@fluidvoice I think it would be best to have a VM image (just like the SDcard image) and spin that up every time in a clean state when you want to test. I don't think that the Virtual Machine can be stable enough for multiple restarts and HDD changes. So would recommend to

• install a clean mini.iso (I store a copy of that as well)
• run the build_sdcard.sh and power off the VM.
• make a copy of the VM image and that will be your own clean state RaspiBlitz VM image. It will start up as a fresh install every time you boot a fresh copy.

BTW my PR from this branch is now merged by rootzoll to the RaspiBlitz master, so can build the VM with that as well.

yes agreed. I thought just keeping VM snapshots would be enough, but no. The snapshots are linked and so they can make matters even worse when you want to correct something like external HD UUID or other config things. The snapshots are a complicated link of internal config states. Better to go back to a relativelly fresh installed image as you said.

[fluidvoice]

Raspiblitz v1.3 (for Intel/x86) on Debian 10 "Buster" in a VirtualBox VM NOTE: AS OF NOW THIS IS RELATIVELY UNTESTED, #RECKLESS, (have backups!)

• I chose to install only Debian Core, SSH, and the LXQt desktop.
• This was done on a Linux machine but very similar procedure should work on Windows.

Download and install VirtualBox for Windows / Linux SHA256SUMS (Youtube has numerous VBox tutorials of various operations which may be helpful)

QUICK INSTALL If you'd rather download and install a pre-made VirtualBox VM skip to the Import Appliance: section below. Otherwise continue for COMPLETE INSTALL

Download Buster mini.iso (net install) wget --continue http://ftp.nl.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/mini.iso SHA256SUM 20247e068b1edf5b3bfefff6f0cb7073ca48e56c4adba241429cf4f49dddd489 ./netboot/mini.iso

Create a new VirtualBox VM minimum 10GB disk, 1GB RAM, 1 CPU cores/threads. I used bridged network setting but NAT may also work.

• Mount the mini.iso file as a virtual CD-ROM drive in the VM and start the VM.
• Install Debian, then shut down the VM when install is complete.
• Remove the virtual CD-ROM or lower it's boot order, start the VM and boot into Debian.

Dowload the Raspiblitz v1.3 build script $ wget https://raw.githubusercontent.com/openoms/droidblitz/v1.3/build_sdcard.sh $ chmod +x build_sdcard.sh $ sudo ./build_sdcard.sh v1.3 openoms Note: sudo does not work by default on Debian so after logging in as [user] you need to:

• run su and enter the root password
• then you can run ./build_sdcard.sh v1.3 openoms

Import Appliance: (potentially helpful video tutorial) The complete pre-installed VM image (appliance) of Raspiblitz v1.3 Debian-Buster can be downloaded here: BusterBlitz.ova (3.2GB) To verify that the downloaded file is complete/uncorrupted verify the SHA256 hash is correct: $ shasum -a 256 BusterBlitz.ova 8b8908a0b4ce956c08cbc0e6e1419f275ac134e09111148c9dbd515f8da93a51 BusterBlitz.ova

• Import this appliance into VirtualBox, add a "raw" disk, then start the VM and install a new Raspiblitz node or upgrade an existing one via a USB-connected drive w/blockchain DB, LN, and wallet. (make sure you are backed up!)
• You need to add a second "raw" disk (empty or prior synced bitcoind/LND) as per below. This appliance/image contains only the boot disk with a fresh install of Debian and Raspiblitz.
• This should behave nearly identical to a Pi that's been booted from a freshly burned SD card.
• Oracle doc on importing appliance/VM

Before booting Raspiblitz VM Add a USB connected drive to the VM as a second "raw" (non-virtual) disk for your blockchain DB and wallet, then install Raspiblitz as normal. Login as admin password:raspiblitz to kick off the install script. No need to SSH

• To use an external USB disk (eg. Raspiblitz HDD) in the VM run: $ lsblk to see your HD device string "/dev/sd??" then use that in the following command $ VBoxManage internalcommands createrawvmdk -filename BLOCKCHAIN.vmdk -rawdisk /dev/sdb1 RAW host disk access VMDK file BLOCKCHAIN.vmdk created successfully.
• Then add the BLOCKCHAIN.vmdk to your VM as a second HD (the first one is your boot drive).
• Potentially helpful how-to add the .vmdk raw disk. In this case we are adding it as a second HD not the primary OS/install drive.

[Re-install from scratch]: You can change the CD in the appliance VM from 2nd to 1st in the boot order and it will bootup the Debian Buster mini.iso net-install disk that you can use to re-install from scratch... the Debian boot drive and reinstall Raspiblitz-x86 via the build_sdcard.sh script. This will take some time :)

I'm currently syncing/testing this VM.

[fluidvoice]

Side note: Stacer system optimizer/monitor has a nice GUI and features. Perhaps it's a good model for possible format of the Raspiblitz touchscreen GUI? [screenshots] Or maybe use this via VNC remote for admin purpose instead of SSH?

Install Stacer

• Debian x64

  Download stacer_1.1.0_amd64.deb from the Stacer releases page. Run sudo dpkg -i stacer*.deb on the downloaded package. Launch Stacer using the installed stacer command.

$ wget https://github.com/oguzhaninan/Stacer/releases/download/v1.1.0/stacer_1.1.0_amd64.deb $ sudo dpkg -i stacer_1.1.0_amd64.deb Selecting previously unselected package stacer. (Reading database ... 181495 files and directories currently installed.) Preparing to unpack stacer_1.1.0_amd64.deb ... Unpacking stacer (1.1.0-1) ... Setting up stacer (1.1.0-1) ... Processing triggers for hicolor-icon-theme (0.17-2) ... Processing triggers for desktop-file-utils (0.23-4) ... Processing triggers for mime-support (3.62) ... $ sudo stacer

• Build from Source