AWS Graviton processors expand options for embedded Linux developers

One of the big changes underway is the migration of the Arm architecture to the desktop and to the cloud.

Docker Desktop and QEMU can be used to build and run Linux applications targeted for the Arm architecture on x86 machines, but the developer experience is not the same as developing on an Arm computer. Using AWS Graviton processors, embedded Linux developers can perform many tasks in a shorter time with more flexibility.

Embedded Linux systems are constructed from two main elements, the Linux kernel and the root file system. The traditional development process involves cross-compiling the software on an x86 machine and then programming a storage device such as an SD card or eMMC module. Once the programming process is complete the embedded system boots and runs Linux. The Raspberry Pi is a popular example of the process, but there are many systems that operate in a similar way.

Embedded systems typically have some software that is tightly coupled to the hardware. This means that developers run the software directly on the target hardware. This changes with AWS Graviton processors. Let’s look at two examples to understand the benefits of using AWS Graviton processors for embedded Linux development.

Using Arm file systems on EC2 instances powered by AWS Graviton processors

The NXP S32V automotive development platform targets vision and sensor fusion applications. The software from NXP comes in the form of a board support package (BSP) for the Arm Cortex-A53 based device. The download contains the Linux kernel images, the file system, and other artifacts such as the Linux device tree and the boot loader.

Using Graviton-based EC2 instances it’s possible to extract the file system and use the chroot command to inspect and modify the contents of the file system. Because the S32V and AWS Graviton processors share a common Arm architecture, the applications in the file system can be executed on either system. Let’s see how to determine which version of gcc is installed in the S32V file system and to compile and run a C language example using an EC2 instance.

First, create a new EC2 instance. The instance can be any of the instance types powered by Graviton processors including A1, T4g, M6g, C6g, or R6g. There are numerous AWS tutorials about how to create an AWS account and use the AWS Console to configure and launch a new EC2 instance. Make sure to substitute one of the above instance types in any tutorial you use.

For this example, I created a t4g.medium running Ubuntu 18.04 with a public IP address and accessible from my desktop via ssh.

I downloaded the S32V BSP using the link above and extracted the tar file from the package which contains the root file system. I ignored all of the other Linux kernel files and SD card images.

Copy the file system tar.gz file to the created EC2 instance and extract it. Use your key file and EC2 IP address in the commands.

$ scp -i key.pem fsl-image-auto-s32v234evb.tar.gz ubuntu@<ec2-ip-address>:~/
$ ssh -i key.pem ubuntu@<ec2-ip-address>

After connecting to the EC2 instance setup the root file system.

$ mkdir fs ; cd fs
$ tar xvf ../fsl-image-auto-s32v234evb.tar.gz ; cd ..
$ sudo chroot fs /bin/bash 

Now, I’m in the S32V file system and the bash prompt changes.

Check the version of gcc installed.

bash-4.4# gcc --version
gcc (Linaro GCC 6.3-2017.06~dev) 6.3.1 20170509

Here is a simple test.c program which can be compiled and run in the chroot. Edit a new file test.c and add the contents. This can be done on the EC2 machine and copied into the root fs directory or you can run vi right in the chroot and create the file.

#include <stdio.h>
#include <stdlib.h>

int main()
{
    srand(1);

    double a1[1000] __attribute__((__aligned__(16)));
    double a2[1000] __attribute__((__aligned__(16)));
    double a3[1000] __attribute__((__aligned__(16)));

    double sum = 0;

    for (int i = 0; i < 1000; i++) {
        a1[i] = (rand() % 2000) - 1000;
        a2[i] = (rand() % 2000) - 1000;
    }


    for (int i = 0; i < 1000; i++) {
        a3[i] = a1[i] * a2[i];
    }

    for (int i = 0; i < 1000; i++) {
        sum += a3[i];
    }

    printf("Sum: %f\n", sum);

    return 0;
}

Exit the chroot and copy test.c into the fs/ directory and re-enter the chroot.

$ cp test.c fs/home/root
$ sudo chroot fs /bin/bash
bash-4.4# cd /home/root

Even vi works inside of the chroot to edit test.c

bash-4.4# gcc -O3 -g test.c -o test
bash-4.4# ./test
Sum: 258936.000000

Disassemble the program using objdump. Partial output is shown here.

bash-4.4# objdump -S test | more
    double sum = 0;

    for (int i = 0; i < 1000; i++) {
        a1[i] = (rand() % 2000) - 1000;
  4004fc:       5289ba74        mov     w20, #0x4dd3                    // #19923
{
  400500:       a9025bf5        stp     x21, x22, [sp, #32]
        a1[i] = (rand() % 2000) - 1000;
  400504:       72a20c54        movk    w20, #0x1062, lsl #16
{
  400508:       a90363f7        stp     x23, x24, [sp, #48]
  40050c:       910103b5        add     x21, x29, #0x40
  400510:       8b0303b6        add     x22, x29, x3
    srand(1);
  400514:       d2800018        mov     x24, #0x0                       // #0
        a1[i] = (rand() % 2000) - 1000;
  400518:       5280fa13        mov     w19, #0x7d0                     // #2000
    for (int i = 0; i < 1000; i++) {
  40051c:       d283e817        mov     x23, #0x1f40                    // #8000
    srand(1);
  400520:       97ffffe8        bl      4004c0 <srand@plt>
        a1[i] = (rand() % 2000) - 1000;
  400524:       97ffffdb        bl      400490 <rand@plt>
  400528:       9b347c01        smull   x1, w0, w20
  40052c:       9367fc21        asr     x1, x1, #39
  400530:       4b807c21        sub     w1, w1, w0, asr #31
  400534:       1b138020        msub    w0, w1, w19, w0
  400538:       510fa000        sub     w0, w0, #0x3e8
  40053c:       1e620000        scvtf   d0, w0
  400540:       fc386aa0        str     d0, [x21, x24]

It looks and feels just like a native Arm Linux computer. There is no cross-compiling and the exact tools from the S32V file system are used.

Some applications will require specific hardware and will not run correctly on a general-purpose machine, but many run as expected. It is easy to see how applications can be built and tested using the same tools and libraries provided by the BSP. Everything can be inspected before taking the time to program a board and boot it up only to find out that something was not included or does not function properly.

Docker from scratch is another way to utilize Graviton-based EC2 instances for embedded Linux development.

Exit the chroot and let's try Docker from scratch.

Using Docker with an embedded file system

Docker provides another way to work with file systems for embedded Linux.

Install Docker on the same t4g.medium instance and make sure it can run hello-world.

$ sudo apt update
$ sudo apt upgrade -y
$ curl -fsSL get.docker.com -o get-docker.sh && sh get-docker.sh
$ sudo usermod -aG docker ubuntu ; newgrp docker
$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

To build a docker image from scratch using the S32V file system create a Dockerfile with your favorite editor.

FROM scratch
ADD fsl-image-auto-s32v234evb.tar.gz /
ADD test.c /home/root
CMD /bin/bash

Build the docker image with the Dockerfile and the root file system in the current directory.

$ docker build -t s32v -f Dockerfile .

Run the docker image and compile the software.

$ docker run -it  s32v
bash-4.4# cd /home/root
bash-4.4# gcc -O3 -g test.c -o test
bash-4.4# ./test
Sum: 258936.000000
bash-4.4# exit

Docker does not save the modified image automatically when a container exits. To save the modified container get the container id and use the commit command to write a new image.

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                      PORTS               NAMES
fb2481cfe668        s32v                "/bin/sh -c /bin/bash"   23 seconds ago       Exited (0) 2 seconds ago                        elastic_liskov
$ docker commit fb2481cfe668 s32v

Images can be saved in Docker Hub or AWS ECR. It's also possible to save a tar file of the image and manually copy it between machines.

$ docker save s32v | gzip > s32v-fs.tgz

The tar file from docker save is a tar file for each image layer and could be put back into the file system we started from with the modifications made.

Docker also makes it easy to transport images between machines and offers some ease of use compared to chroot.

Docker and chroot combined with EC2 instances powered by Graviton processors are a powerful tool for embedded Linux developers and make the development experience for Arm easier. The many EC2 configurations make it possible to right size the number of CPUs and the size of memory for a task. Compiling large projects can be done much faster and easier than on the development board or by cross-compiling.

The techniques used with the NXP S32V root file system can be applied to most any embedded Linux development board.

Summary

AWS Graviton processors improve the developer experience for embedded Linux systems. Commands like chroot and Docker from scratch improve the developer experience by not requiring everything to be done on a development board. The cross-compile, copy, run, debug loop is more time consuming when the build machine is x86 and the target system is Arm. The variety of A1, T4g, M6g, C6g, and R6g instances make it easy to select the right hardware for best performance.

It’s worth thinking about how AWS Graviton processors fit into your workflow for Linux on Arm development. Now is a perfect time to experiment with the new T4g EC2 instances and take advantage of the free trial available until the end of 2020.