v86/docs/archlinux.md

(This document partly also applies to other Linuxes)

Choosing an installer ISO

Download Arch Linux 32 from https://archlinux32.org.

Basic installation using QEMU

Installing Arch Linux with these instructions will result in a raw disk image that can be booted by v86.

# fetch archlinux32 installer
wget https://mirror.archlinux32.org/archisos/archlinux32-2021.12.01-i686.iso

# Create a 10 gigabyte disk image. If you intend to pacstrap only 'base' then 1.5G should be fine also.
qemu-img create arch.img 10G

# Follow the normal installation process (you can add accel=kvm if your system supports it to speed up the installation)
qemu-system-x86_64 -m 256 -drive file=arch.img,format=raw -cdrom archlinux32-2021.12.01-i686.iso

For keyboard support, it is necessary to open /etc/mkinitcpio.conf and edit the following line:

MODULES="atkbd i8042"

For the changes to take effect, you need to regenerate the initial ramdisk with mkinitcpio -p linux

The resulting arch.img file is a bootable disk image for v86.

Scripting image creation for v86

Installing the ISO by hand takes a long time if you intend to recreate the image many times. There are various reasons why you might want to do this more than once. For example: because the emulator is slow you might want to compile any new software release in QEMU which is much faster and then use the resulting image in v86 instead of making the emulator compile the software. Another reason is that the build progress potentially takes long and if you want to do automated builds in parallel to find out what configurations do and don't work you can just throw more computing power at the problem in order to solve it. This example requires that you have packer, qemu and kpartx installed.

Creating a packer template

Packer is a tool that lets you boot an ISO in any of multiple emulators (so QEMU in our case) and send pre-scripted keystrokes to bootstrap and SSH server. Once the SSH connection is established a script can be started for further provisioning.

Create a template for automating the base installation:

mkdir -p packer
cat > packer/template.json << 'EOF'
{
  "provisioners": [
    {
      "type": "shell",
      "override": {
        "qemu": {
          "scripts": ["scripts/provision.sh"]
        }
      }
    }
  ],
  "builders": [
    {
      "accelerator": "kvm",
      "type": "qemu",
      "boot_command": [
        "<enter><wait30><enteropenssl passwd help<wait10>",
        "dhcpcd<enter><wait5>",
        "echo root:root | chpasswd<enter><wait5>",
        "systemctl start sshd<enter>"
      ],
      "headless": true,
      "boot_wait": "10s",
      "disk_size": 1500,
      "disk_interface": "ide",
      "iso_url": "https://mirror.archlinux32.org/archisos/archlinux32-2021.12.01-i686.iso",
      "iso_checksum": "90c6f5aecb095d5578f6c9970539da7c5e9324ec",
      "iso_checksum_type": "sha1",
      "ssh_wait_timeout": "120s",
      "ssh_pty": true,
      "ssh_username": "root",
      "ssh_password": "root",
      "ssh_port": 22,
      "format": "raw",
      "vm_name": "archlinux",
      "disk_detect_zeroes": "unmap",
      "memory": 2048,
      "vnc_bind_address": "0.0.0.0"
    }
  ]
}
EOF

You can tweak the options a bit to match your situation. For debugging, you can set headless to false. That will show you the VNC connection instead of running the boot_command in the background. For a base pacstrap, using a 2 GB disk image should be sufficient. The raw disk format is important. v86 does not read qcow2 images, only raw disk images. If your system does not support KVM (the default accelerator), you can change "accelerator": "none" to the settings, in macos you may use "accelerator": "hvf". Other accelerator options can be found here.

After gaining SSH connectivity to the VM, packer will run the scripts/provisioning.sh script in the guest.

Creating the Arch Linux installation script

Create a script for your Arch Linux installation. This runs in the live Arch Linux environment, so you need to partition the disk, do a pacstrap, and install a bootloader.

mkdir -p packer/scripts
### Write your own or copy paste the example below
vim packer/scripts/provision.sh

An example script to install Arch Linux with the root mounted using the 9p network filesystem:

#!/bin/bash
echo "Creating a GPT partition on /dev/sda1"
echo -e "g\nn\n\n\n\nw" | fdisk /dev/sda

# In case you might want to create a DOS partition instead. It doesn't really matter.
#echo "Creating a DOS partition on /dev/sda1"
#echo -e "o\nn\np\n1\n\n\nw" | fdisk /dev/sda

echo "Formatting /dev/sda1 to ext4"
mkfs -t ext4 /dev/sda1

echo "Mounting new filesystem"
mount -t ext4 /dev/sda1 /mnt

echo "Create pacman package cache dir"
mkdir -p /mnt/var/cache/pacman/pkg

# We don't want the pacman cache to fill up the image. After reboot whatever tarballs pacman has cached are gone.
echo "Mount the package cache dir in memory so it doesn't fill up the image"
mount -t tmpfs none /mnt/var/cache/pacman/pkg

echo "Updating archlinux-keyring"
pacman -Sy archlinux-keyring --noconfirm

# uncomment to remove signing if unable to resolve signing errors
sed -i 's/SigLevel.*/SigLevel = Never/g' /etc/pacman.conf

# Install the Arch Linux base system, feel free to add packages you need here
echo "Performing pacstrap"
pacstrap -i /mnt base linux dhcpcd curl openssh --noconfirm

echo "Writing fstab"
genfstab -p /mnt >> /mnt/etc/fstab

# When the Linux boots we want it to automatically log in on tty1 as root
echo "Ensuring root autologin on tty1"
mkdir -p /mnt/etc/systemd/system/getty@tty1.service.d
cat << 'EOF' > /mnt/etc/systemd/system/getty@tty1.service.d/override.conf
[Service]
ExecStart=
ExecStart=-/usr/bin/agetty --autologin root --noclear %I $TERM
EOF

# This is the tricky part. The current root will be mounted on /dev/sda1 but after we reboot
# it will try to mount root during boot using the 9p network filesystem. This means the emulator
# will request all files over the network using XMLHttpRequests from the server. This is great
# because then you only need to provide the client with a saved state (the memory) and the
# session will start instantly and load needed files on the fly. This is fast and it saves bandwidth.
echo "Ensuring root is remounted using 9p after reboot"
mkdir -p /mnt/etc/initcpio/hooks
cat << 'EOF' > /mnt/etc/initcpio/hooks/9p_root
run_hook() {
    mount_handler="mount_9p_root"
}

mount_9p_root() {
    msg ":: mounting '$root' on real root (9p)"
    # Note the host9p. We won't mount /dev/sda1 on root anymore,
    # instead we mount the network filesystem and the emulator will
    # retrieve the files on the fly.
    if ! mount -t 9p host9p "$1"; then
        echo "You are now being dropped into an emergency shell."
        launch_interactive_shell
        msg "Trying to continue (this will most likely fail) ..."
    fi
}
EOF

echo "Adding initcpio build hook for 9p root remount"
mkdir -p /mnt/etc/initcpio/install
cat << 'EOF' > /mnt/etc/initcpio/install/9p_root
#!/bin/bash
build() {
	add_runscript
}
EOF

# We need to load some modules into the kernel for it to play nice with the emulator
# The atkbd and i8042 modules are for keyboard input in the browser. If you do not
# want to use the network filesystem you only need these. The 9p, 9pnet and 9pnet_virtio
# modules are needed for being able to mount 9p network filesystems using the emulator.
echo "Configure mkinitcpio for 9p"
sed -i 's/MODULES=()/MODULES=(atkbd i8042 libps2 serio serio_raw psmouse virtio_pci virtio_pci_modern_dev 9p 9pnet 9pnet_virtio fscache netfs)/g' /mnt/etc/mkinitcpio.conf

# Because we want to mount the root filesystem over the network during boot, we need to
# hook into initcpio. If you do not want to mount the root filesystem during boot but
# only want to mount a 9p filesystem later, you can leave this out. Once the system
# has been booted you should be able to mount 9p filesystems with mount -t 9p host9p /blabla
# without this hook.
sed -i 's/fsck"/fsck 9p_root"/g' /mnt/etc/mkinitcpio.conf

# enable ssh password auth and root login
sed -i 's/#PermitRootLogin.*/PermitRootLogin yes/g' /etc/ssh/sshd_config
sed -i 's/#PasswordAuthentication.*/PasswordAuthentication yes/g' /etc/ssh/sshd_config

echo "Writing the installation script"
cat << 'EOF' > /mnt/bootstrap.sh
#!/usr/bin/bash
echo "Re-generate initial ramdisk environment"
mkinitcpio -p linux

# uncomment to remove signing if you are unable to resolve signing errors otherwise
sed -i 's/SigLevel.*/SigLevel = Never/g' /etc/pacman.conf

pacman -S --noconfirm syslinux gptfdisk
syslinux-install_update -i -a -m

# disabling ldconfig to speed up boot (to remove Rebuild dynamic linker cache...)
# you may want to comment this out
echo "Disabling ldconfig service"
systemctl mask ldconfig.service

sync
EOF

echo "Chrooting and bootstrapping the installation"
arch-chroot /mnt bash bootstrap.sh


cat << 'EOF' > /mnt/boot/syslinux/syslinux.cfg
# Config file for Syslinux -
# /boot/syslinux/syslinux.cfg
#
# Comboot modules:
#   * menu.c32 - provides a text menu
#   * vesamenu.c32 - provides a graphical menu
#   * chain.c32 - chainload MBRs, partition boot sectors, Windows bootloaders
#   * hdt.c32 - hardware detection tool
#   * reboot.c32 - reboots the system
#
# To Use: Copy the respective files from /usr/lib/syslinux to /boot/syslinux.
# If /usr and /boot are on the same file system, symlink the files instead
# of copying them.
#
# If you do not use a menu, a 'boot:' prompt will be shown and the system
# will boot automatically after 5 seconds.
#
# Please review the wiki: https://wiki.archlinux.org/index.php/Syslinux
# The wiki provides further configuration examples

DEFAULT arch
PROMPT 0        # Set to 1 if you always want to display the boot: prompt
TIMEOUT 100

# Menu Configuration
# Either menu.c32 or vesamenu32.c32 must be copied to /boot/syslinux
UI menu.c32
#UI vesamenu.c32

# Refer to http://syslinux.zytor.com/wiki/index.php/Doc/menu
MENU TITLE Arch Linux
#MENU BACKGROUND splash.png
MENU COLOR border       30;44   #40ffffff #a0000000 std
MENU COLOR title        1;36;44 #9033ccff #a0000000 std
MENU COLOR sel          7;37;40 #e0ffffff #20ffffff all
MENU COLOR unsel        37;44   #50ffffff #a0000000 std
MENU COLOR help         37;40   #c0ffffff #a0000000 std
MENU COLOR timeout_msg  37;40   #80ffffff #00000000 std
MENU COLOR timeout      1;37;40 #c0ffffff #00000000 std
MENU COLOR msg07        37;40   #90ffffff #a0000000 std
MENU COLOR tabmsg       31;40   #30ffffff #00000000 std

# boot sections follow
#
# TIP: If you want a 1024x768 framebuffer, add "vga=773" to your kernel line.
#
#-*

LABEL arch
    MENU LABEL Arch Linux 9p
    LINUX ../vmlinuz-linux
    APPEND root=/dev/sda1 rw quiet
    INITRD ../initramfs-linux.img

LABEL arch2
    MENU LABEL Arch Linux Disk
    LINUX ../vmlinuz-linux
    APPEND root=/dev/sda1 rw quiet disablehooks=9p_root
    INITRD ../initramfs-linux.img

LABEL hdt
        MENU LABEL HDT (Hardware Detection Tool)
        COM32 hdt.c32

LABEL reboot
        MENU LABEL Reboot
        COM32 reboot.c32

LABEL poweroff
        MENU LABEL Poweroff
        COM32 poweroff.c32
EOF
umount -R /mnt

With the packer template and the script you have enough to create an image that can be booted by v86. But because this example script installs an Arch Linux that wants to mount root over the network with 9p, we need to host that filesystem first. If you do not want to use 9p, you can just run (cd packer && packer build -force template.json) to build the image.

Creating the 9p filesystem

Now that we have an image that contains a filesystem, we can convert that filesystem into something we can host on the webserver together with the v86 library.

To do so, we need to mount the image once and create a json mapping of the filesystem. The following script shows how to map the filesystem in an automated fashion.

Create a script to builds the image and then creates v86 compatible artifacts:

vim build.sh

Example script:

#!/bin/sh

SRC=packer
TARGET=output

# build the boxfile from the iso
(cd $SRC && sudo PACKER_LOG=1 PACKER_LOG_PATH="./packer.log" packer build -force template.json)

# test if there is a boxfile where we expected it
if [ ! -f $SRC/output-qemu/archlinux ]; then
    echo "Looks like something went wrong building the image, maybe try again?"
    exit 1
fi;

# clean up any previous loops and mounts
echo "Making sure mountpoint is empty"
LOOP_DEV=$(sudo losetup -f)

sudo umount diskmount -f || /bin/true
sudo kpartx -d $LOOP_DEV || /bin/true
sudo losetup -d $LOOP_DEV || /bin/true

# mount the generated raw image, we do that so we can create
# a json mapping of it and copy it to host on the webserver
mkdir -p diskmount
echo "Mounting the created image so we can convert it to a p9 image"
sudo losetup $LOOP_DEV $SRC/output-qemu/archlinux
sudo kpartx -a $LOOP_DEV
sudo mount /dev/mapper/$(basename $LOOP_DEV)p1 diskmount

# make images dir
mkdir -p $TARGET
mkdir -p $TARGET/images
mkdir -p $TARGET/images/arch

# map the filesystem to json with fs2json
sudo ./tools/fs2json.py --out $TARGET/images/fs.json diskmount
sudo ./tools/copy-to-sha256.py diskmount $TARGET/images/arch

# copy the filesystem and chown to nonroot user
echo "Copying the filesystem to $TARGET/arch"
mkdir $TARGET/arch -p
sudo rsync -q -av diskmount/ $TARGET/arch
sudo chown -R $(whoami):$(whoami) $TARGET/arch

# clean up mount
echo "Cleaning up mounts"
sudo umount diskmount -f
sudo kpartx -d $LOOP_DEV
sudo losetup -d $LOOP_DEV

# Move the image to the images dir
sudo mv $SRC/output-qemu/archlinux $TARGET/images/arch.img

Given that the packer template and provision.sh is rooted at packer (adjust the value of $SRC otherwise), run the build.sh at root of your v86 repo:

chmod +x build.sh
./build.sh

Generated artifacts are now available for serving from output.

Using the created artifacts in v86

Now that we have everything we need to host a server that serves an Arch Linux environment over the network.

Create a checkout of v86 and run make build/libv86.js. We can then edit examples/arch.html, we have two options:

1. Boot Arch Linux from the 9p filesystem (generated .bin artifacts at /output/images/arch):

   filesystem: {
   baseurl: "../output/images/arch/",
   basefs: "../output/images/fs.json",
   },
   
   bzimage_initrd_from_filesystem: true,
   
   cmdline: [
   "rw",
   "root=host9p rootfstype=9p rootflags=trans=virtio,cache=loose",
   ].join(" "),
   
   acpi: false,
   autostart: true,
   

2. Boot the Arch Linux from the qemu raw disk image:

   hda: {
     url: "../output/images/arch.img",
     # set to true if you want to load it asynchrously during runtime (for this option we need to run a webserver that supports the Range header)
     # NOTE: async: false is slow but proved to be more realiable
     async: false,
   
     # This needs to be the size of the raw disk.
     size: 1.5 * 1024 * 1024 * 1024,
     # See the `disk_size` item in the packer template.
   },
   
   acpi: false,
   autostart: true,
   

Next, we need a webserver that supports the Range header. For example this extension of the SimpleHTTPServer. At your v86 root, run:

wget https://raw.githubusercontent.com/smgoller/rangehttpserver/master/RangeHTTPServer.py
python2 RangeHTTPServer.py

Now that the webserver is running, point your browser to http://localhost:8000/examples/arch.html. Wait for the Linux to boot. When the system is up, click 'Save state to file'. Your browser will download a v86state.bin file. Copy that file to /your/v86/dir/images. You can then edit examples/arch.html again and add a 'state' key to the V86 array.

initial_state: {
    "url": "http://localhost:8000/images/v86state.bin",
},

If you refresh http://localhost:8000/examples/arch.html you will see that the state is restored instantly and all required files are loaded over the network on the fly.

Networking

The emulator can emulate a network card. For more information look at the networking documentation. To set up networking in the VM, add the following item to the V86 array in the examples/arch.html file:

network_relay_url: "ws://localhost:8080/",

This will make the emulator try to connect to a WebSockets proxy. Running the proxy is very easy if you use the Docker container.

sudo docker run --privileged -p 8080:80 --name relay bennottelling/websockproxy

NOTE: original benjamincburns/jor1k-relay:latest has throttling built-in by default which will degrade the networking. bennottelling/websockproxy has this throttling removed via websockproxy/issues/4#issuecomment-317255890.

You can check if the relay is running correctly by going to http://localhost:8080/ in your browser. There you should see a message that reads Can "Upgrade" only to "Websocket"..

Now you should be able to get network connectivity in the virtual machine. If you are restoring from a saved state, you might need to first run:

ip link set enp0s5 down
rmmod ne2k-pci

To bring the network up, run:

modprobe ne2k-pci
dhcpcd -w4 enp0s5

It might take a while for a carrier to become available on the interface. If the dhcpcd command fails shortly after booting, wait a bit and try again a bit later. If you are using the 9p network filesystem you can use the developer tools networking tab (in chrome) to get a sense of what is going on by looking at the files that are being downloaded.

When the network is up you should be able to curl a website. To check, run curl icanhazip.com. There you should see the public IP of the machine running the proxy.

You can't do inbound traffic into the VM with the websockproxy Docker container because it uses a basic NAT. To SSH into the VM running in the browser, you can create a reverse SSH tunnel to expose the SSH port of the sshd in the VM to the outside world. You may need to start sshd first, it may also be reasonable to change root password:

passwd root
systemctl start sshd

then create a reverse SSH tunnel:

# This will create a port 1122 on the example.com server
# which forwards to the SSH in the VM
ssh root@example.com -R 1122:localhost:22

Now on the example.com server you should be able to SSH into your browser tab by running ssh root@localhost -p 1122.