Install Data Plane Development Kit (DPDK) and Build NDN-DPDK

My Workflow

Data Plane Development Kit (DPDK) is a set of libraries to accelerate packet processing workloads, often used on 10Gbps and faster networks. Storage Performance Development Kit (SPDK) is another set of libraries for writing high performance storage applications, and is often used together with DPDK.

In past few years I've been developing NDN-DPDK, a high speed packet forwarder for Named Data Networking (NDN), a future Internet architecture.

As a stallion coder, I understand the importance of continuous integration. Since the public release of NDN-DPDK in 2019, I was using Travis CI, but I'm facing a limitation: I want to compile the project both in Ubuntu directly and as a Docker container, but Travis CI job syntax does not have good support for that.

In this hackathon, I learned that GitHub Actions supports multiple jobs with different steps. This allows me to build a workflow with two separate jobs.
The first job builds and tests the project in Ubuntu:

1. Install DPDK and SPDK, using the setup-dpdk Action I created.
2. Install other dependencies of NDN-DPDK.
3. Compile NDN-DPDK.
4. Run unit tests.

The second job builds the Docker container, using the docker/build-push-action@v1 Action.

Submission Category:

DIY Deployments

Yaml File or Link to Code

• setup-dpdk action: https://github.com/yoursunny/setup-dpdk
• NDN-DPDK repository: https://github.com/usnistgov/ndn-dpdk

Considering that installing DPDK and SPDK is a reusable building block, I decide to create a custom Action to install DPDK and SPDK. I choose to use "composite run steps" type because it is the simplest.
DPDK is a fairly large project and takes about 10 minutes to compile, and it would be nice to have caching. However, "composite run steps" Action type does not support importing other existing Actions with uses: keyword. Thus, users of my Action have to configure caching in the top-level workflow.

During my attempt, I made many mistakes and learned from them:

• $GITHUB_WORKSPACE is not a suitable place for downloading and compiling dependencies such as DPDK, because actions/checkout@v2 step would use this directory to checkout the codebase. Instead, I should download and compile DPDK in a directory under $HOME.
• DPDK uses -march=native compiler flag for maximum performance, but this flag also makes the compiled library only work on a particular CPU. Although GitHub-hosted runners use the same Azure machine type, there are minor differences in the CPU model. Therefore, if I use actions/cache on compiled DPDK and SPDK libraries, it's necessary to include CPU flags (obtained from lscpu -J command) as part of the cache key, to prevent libraries compiled for one CPU from being used on a different CPU.

Additional Resources / Info