123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461 |
- Raspberry Pi 3
- ==============
- The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four
- Arm Cortex-A53 cores.
- The following instructions explain how to use this port of the TF-A with the
- default distribution of `Raspbian`_ because that's the distribution officially
- supported by the Raspberry Pi Foundation. At the moment of writing this, the
- officially supported kernel is a AArch32 kernel. This doesn't mean that this
- port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by
- the Foundation can be compiled for AArch64 by following the steps in
- `AArch64 kernel build instructions`_.
- **IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM,
- which is available from both the Non-secure and Secure worlds. This port
- shouldn't be considered more than a prototype to play with and implement
- elements like PSCI to support the Linux kernel.
- Design
- ------
- The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a
- VideoCore IV that acts as primary processor (and loads everything from the SD
- card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi
- 3 documentation`_ for more information.
- This explains why it is possible to change the execution state (AArch64/AArch32)
- depending on a few files on the SD card. We only care about the cases in which
- the cores boot in AArch64 mode.
- The rules are simple:
- - If a file called ``kernel8.img`` is located on the ``boot`` partition of the
- SD card, it will load it and execute in EL2 in AArch64. Basically, it executes
- a `default AArch64 stub`_ at address **0x0** that jumps to the kernel.
- - If there is also a file called ``armstub8.bin``, it will load it at address
- **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All
- the cores are powered on at the same time and start at address **0x0**.
- This means that we can use the default AArch32 kernel provided in the official
- `Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and
- anything else we need is in ``armstub8.bin``. This way we can forget about the
- default bootstrap code. When using a AArch64 kernel, it is only needed to make
- sure that the name on the SD card is ``kernel8.img``.
- Ideally, we want to load the kernel and have all cores available, which means
- that we need to make the secondary cores work in the way the kernel expects, as
- explained in `Secondary cores`_. In practice, a small bootstrap is needed
- between TF-A and the kernel.
- To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode
- in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The
- architecture specifies that AArch32 Hypervisor mode isn't present when AArch64
- is used for EL2. When using a AArch64 kernel, it should simply start in EL2.
- Placement of images
- ~~~~~~~~~~~~~~~~~~~
- The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding
- between them so that the addresses they are loaded to match the ones specified
- when compiling TF-A. This is done automatically by the build system.
- The device tree block is loaded by the VideoCore loader from an appropriate
- file, but we can specify the address it is loaded to in ``config.txt``.
- The file ``kernel8.img`` contains a kernel image that is loaded to the address
- specified in ``config.txt``. The `Linux kernel tree`_ has information about how
- a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``:
- ::
- The zImage may also be placed in system RAM and called there. The
- kernel should be placed in the first 128MiB of RAM. It is recommended
- that it is loaded above 32MiB in order to avoid the need to relocate
- prior to decompression, which will make the boot process slightly
- faster.
- There are no similar restrictions for AArch64 kernels, as specified in the file
- ``Documentation/arm64/booting.txt``.
- This means that we need to avoid the first 128 MiB of RAM when placing the
- TF-A images (and specially the first 32 MiB, as they are directly used to
- place the uncompressed AArch32 kernel image. This way, both AArch32 and
- AArch64 kernels can be placed at the same address.
- In the end, the images look like the following diagram when placed in memory.
- All addresses are Physical Addresses from the point of view of the Arm cores.
- Again, note that this is all just part of the same DRAM that goes from
- **0x00000000** to **0x3F000000**, it just has different names to simulate a real
- secure platform!
- ::
- 0x00000000 +-----------------+
- | ROM | BL1
- 0x00020000 +-----------------+
- | FIP |
- 0x00200000 +-----------------+
- | |
- | ... |
- | |
- 0x01000000 +-----------------+
- | DTB | (Loaded by the VideoCore)
- +-----------------+
- | |
- | ... |
- | |
- 0x02000000 +-----------------+
- | Kernel | (Loaded by the VideoCore)
- +-----------------+
- | |
- | ... |
- | |
- 0x10000000 +-----------------+
- | Secure SRAM | BL2, BL31
- 0x10100000 +-----------------+
- | Secure DRAM | BL32 (Secure payload)
- 0x11000000 +-----------------+
- | Non-secure DRAM | BL33
- +-----------------+
- | |
- | ... |
- | |
- 0x3F000000 +-----------------+
- | I/O |
- 0x40000000 +-----------------+
- The area between **0x10000000** and **0x11000000** has to be manually protected
- so that the kernel doesn't use it. The current port tries to modify the live DTB
- to add a memreserve region that reserves the previously mentioned area.
- If this is not possible, the user may manually add ``memmap=16M$256M`` to the
- command line passed to the kernel in ``cmdline.txt``. See the `Setup SD card`_
- instructions to see how to do it. This system is strongly discouraged.
- The last 16 MiB of DRAM can only be accessed by the VideoCore, that has
- different mappings than the Arm cores in which the I/O addresses don't overlap
- the DRAM. The memory reserved to be used by the VideoCore is always placed at
- the end of the DRAM, so this space isn't wasted.
- Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for
- TF-A, there are 880 MiB available for Linux.
- Boot sequence
- ~~~~~~~~~~~~~
- The boot sequence of TF-A is the usual one except when booting an AArch32
- kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it
- can jump to the kernel in the same mode and let it take over that privilege
- level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of
- TF-A) it could only jump to the kernel in AArch32 in Supervisor mode.
- The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel
- in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The
- bootstrap should take care of this.
- This port support a direct boot of the Linux kernel from the firmware (as a BL33
- image). Alternatively, U-Boot or other bootloaders may be used.
- Secondary cores
- ~~~~~~~~~~~~~~~
- This port of the Trusted Firmware-A supports ``PSCI_CPU_ON``,
- ``PSCI_SYSTEM_RESET`` and ``PSCI_SYSTEM_OFF``. The last one doesn't really turn
- the system off, it simply reboots it and asks the VideoCore firmware to keep it
- in a low power mode permanently.
- The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to
- use mailboxes to trap the secondary cores until they are ready to jump to the
- kernel. This mailbox is located at a different address in the AArch32 default
- kernel than in the AArch64 kernel.
- Kernels with PSCI support can use the PSCI calls instead for a cleaner boot.
- Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During
- cold boot, all secondary cores wait in a loop until they are given given an
- address to jump to in this Mailbox (``bl31_warm_entrypoint``).
- Once BL31 has finished and the primary core has jumped to the BL33 payload, it
- has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop.
- The payload then makes them wait in another waitloop listening from messages
- from the kernel. When the primary CPU jumps into the kernel, it will send an
- address to the mailbox so that the secondary CPUs jump to it and are recognised
- by the kernel.
- Build Instructions
- ------------------
- To boot a AArch64 kernel, only the AArch64 toolchain is required.
- To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The
- AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit
- kernel.
- The build system concatenates BL1 and the FIP so that the addresses match the
- ones in the memory map. The resulting file is ``armstub8.bin``, located in the
- build folder (e.g. ``build/rpi3/debug/armstub8.bin``). To know how to use this
- file, follow the instructions in `Setup SD card`_.
- The following build options are supported:
- - ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image.
- By default this option is 0, which means that TF-A will jump to BL33 in EL2
- in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32
- mode.
- - ``PRELOADED_BL33_BASE``: Used to specify the address of a BL33 binary that has
- been preloaded by any other system than using the firmware. ``BL33`` isn't
- needed in the build command line if this option is used. Specially useful
- because the file ``kernel8.img`` can be loaded anywhere by modifying the file
- ``config.txt``. It doesn't have to contain a kernel, it could have any
- arbitrary payload.
- - ``RPI3_DIRECT_LINUX_BOOT``: Disabled by default. Set to 1 to enable the direct
- boot of the Linux kernel from the firmware. Option ``RPI3_PRELOADED_DTB_BASE``
- is mandatory when the direct Linux kernel boot is used. Options
- ``PRELOADED_BL33_BASE`` will most likely be needed as well because it is
- unlikely that the kernel image will fit in the space reserved for BL33 images.
- This option can be combined with ``RPI3_BL33_IN_AARCH32`` in order to boot a
- 32-bit kernel. The only thing this option does is to set the arguments in
- registers x0-x3 or r0-r2 as expected by the kernel.
- - ``RPI3_PRELOADED_DTB_BASE``: Auxiliary build option needed when using
- ``RPI3_DIRECT_LINUX_BOOT=1``. This option allows to specify the location of a
- DTB in memory.
- - ``RPI3_RUNTIME_UART``: Indicates whether the UART should be used at runtime
- or disabled. ``-1`` (default) disables the runtime UART. Any other value
- enables the default UART (currently UART1) for runtime messages.
- - ``RPI3_USE_UEFI_MAP``: Set to 1 to build ATF with the altername memory
- mapping required for an UEFI firmware payload. These changes are needed
- to be able to run Windows on ARM64. This option, which is disabled by
- default, results in the following memory mappings:
- ::
- 0x00000000 +-----------------+
- | ROM | BL1
- 0x00010000 +-----------------+
- | DTB | (Loaded by the VideoCore)
- 0x00020000 +-----------------+
- | FIP |
- 0x00030000 +-----------------+
- | |
- | UEFI PAYLOAD |
- | |
- 0x00200000 +-----------------+
- | Secure SRAM | BL2, BL31
- 0x00300000 +-----------------+
- | Secure DRAM | BL32 (Secure payload)
- 0x00400000 +-----------------+
- | |
- | |
- | Non-secure DRAM | BL33
- | |
- | |
- 0x01000000 +-----------------+
- | |
- | ... |
- | |
- 0x3F000000 +-----------------+
- | I/O |
- - ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional.
- Please use the code from `here <https://github.com/OP-TEE/optee_os>`__.
- Build the Trusted Firmware with option ``BL32=tee-header_v2.bin
- BL32_EXTRA1=tee-pager_v2.bin BL32_EXTRA2=tee-pageable_v2.bin``
- to put the binaries into the FIP.
- .. warning::
- If OP-TEE is used it may be needed to add the following options to the
- Linux command line so that the USB driver doesn't use FIQs:
- ``dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0``.
- This will unfortunately reduce the performance of the USB driver. It is
- needed when using Raspbian, for example.
- - ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option to 1 to enable
- it. In order to use TBB, you might want to set ``GENERATE_COT=1`` to let the
- contents of the FIP automatically signed by the build process. The ROT key
- will be generated and output to ``rot_key.pem`` in the build directory. It is
- able to set ROT_KEY to your own key in PEM format. Also in order to build,
- you need to clone mbed TLS from `here <https://github.com/ARMmbed/mbedtls>`__.
- ``MBEDTLS_DIR`` must point at the mbed TLS source directory.
- - ``ENABLE_STACK_PROTECTOR``: Disabled by default. It uses the hardware RNG of
- the board.
- The following is not currently supported:
- - AArch32 for TF-A itself.
- - ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any
- address by changing the file ``armstub8.bin``, so there's no point in using
- TF-A in this case.
- Building the firmware for kernels that don't support PSCI
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- This is the case for the 32-bit image of Raspbian, for example. 64-bit kernels
- always support PSCI, but they may not know that the system understands PSCI due
- to an incorrect DTB file.
- First, clone and compile the 32-bit version of the `Raspberry Pi 3 TF-A
- bootstrap`_. Choose the one needed for the architecture of your kernel.
- Then compile TF-A. For a 32-bit kernel, use the following command line:
- .. code:: shell
- CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \
- RPI3_BL33_IN_AARCH32=1 \
- BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin
- For a 64-bit kernel, use this other command line:
- .. code:: shell
- CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \
- BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin
- However, enabling PSCI support in a 64-bit kernel is really easy. In the
- repository `Raspberry Pi 3 TF-A bootstrap`_ there is a patch that can be applied
- to the Linux kernel tree maintained by the Raspberry Pi foundation. It modifes
- the DTS to tell the kernel to use PSCI. Once this patch is applied, follow the
- instructions in `AArch64 kernel build instructions`_ to get a working 64-bit
- kernel image and supporting files.
- Building the firmware for kernels that support PSCI
- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- For a 64-bit kernel:
- .. code:: shell
- CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \
- PRELOADED_BL33_BASE=0x02000000 \
- RPI3_PRELOADED_DTB_BASE=0x01000000 \
- RPI3_DIRECT_LINUX_BOOT=1
- For a 32-bit kernel:
- .. code:: shell
- CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \
- PRELOADED_BL33_BASE=0x02000000 \
- RPI3_PRELOADED_DTB_BASE=0x01000000 \
- RPI3_DIRECT_LINUX_BOOT=1 \
- RPI3_BL33_IN_AARCH32=1
- AArch64 kernel build instructions
- ---------------------------------
- The following instructions show how to install and run a AArch64 kernel by
- using a SD card with the default `Raspbian`_ install as base. Skip them if you
- want to use the default 32-bit kernel.
- Note that this system won't be fully 64-bit because all the tools in the
- filesystem are 32-bit binaries, but it's a quick way to get it working, and it
- allows the user to run 64-bit binaries in addition to 32-bit binaries.
- 1. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To
- speed things up, do a shallow clone of the desired branch.
- .. code:: shell
- git clone --depth=1 -b rpi-4.18.y https://github.com/raspberrypi/linux
- cd linux
- 2. Configure and compile the kernel. Adapt the number after ``-j`` so that it is
- 1.5 times the number of CPUs in your computer. This may take some time to
- finish.
- .. code:: shell
- make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig
- make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
- 3. Copy the kernel image and the device tree to the SD card. Replace the path
- by the corresponding path in your computers to the ``boot`` partition of the
- SD card.
- .. code:: shell
- cp arch/arm64/boot/Image /path/to/boot/kernel8.img
- cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/
- cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b-plus.dtb /path/to/boot/
- 4. Install the kernel modules. Replace the path by the corresponding path to the
- filesystem partition of the SD card on your computer.
- .. code:: shell
- make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
- INSTALL_MOD_PATH=/path/to/filesystem modules_install
- 5. Follow the instructions in `Setup SD card`_ except for the step of renaming
- the existing ``kernel7.img`` (we have already copied a AArch64 kernel).
- Setup SD card
- -------------
- The instructions assume that you have an SD card with a fresh install of
- `Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly
- untouched). They have been tested with the image available in 2018-03-13.
- 1. Insert the SD card and open the ``boot`` partition.
- 2. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore
- bootloader into booting the Arm cores in AArch64 mode, like TF-A needs,
- even though the kernel is not compiled for AArch64.
- 3. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore
- bootloader will look for a file called ``armstub8.bin`` and load it at
- address **0x0** instead of a predefined one.
- 4. To enable the serial port "Mini UART" in Linux, open ``cmdline.txt`` and add
- ``console=serial0,115200 console=tty1``.
- 5. Open ``config.txt`` and add the following lines at the end (``enable_uart=1``
- is only needed to enable debugging through the Mini UART):
- ::
- enable_uart=1
- kernel_address=0x02000000
- device_tree_address=0x01000000
- If you connect a serial cable to the Mini UART and your computer, and connect
- to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some
- text. In the case of an AArch32 kernel, you should see something like this:
- ::
- NOTICE: Booting Trusted Firmware
- NOTICE: BL1: v1.4(release):v1.4-329-g61e94684-dirty
- NOTICE: BL1: Built : 00:09:25, Nov 6 2017
- NOTICE: BL1: Booting BL2
- NOTICE: BL2: v1.4(release):v1.4-329-g61e94684-dirty
- NOTICE: BL2: Built : 00:09:25, Nov 6 2017
- NOTICE: BL1: Booting BL31
- NOTICE: BL31: v1.4(release):v1.4-329-g61e94684-dirty
- NOTICE: BL31: Built : 00:09:25, Nov 6 2017
- [ 0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517
- Raspbian GNU/Linux 9 raspberrypi ttyS0
- raspberrypi login:
- Just enter your credentials, everything should work as expected. Note that the
- HDMI output won't show any text during boot.
- .. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S
- .. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S
- .. _Linux kernel tree: https://github.com/torvalds/linux
- .. _Linux tree fork: https://github.com/raspberrypi/linux
- .. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/
- .. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap
- .. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/
- .. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/
|