gRPC on dotnet core 3.1 on RaspberryPi 3

gRPC on dotnet core 3.1 on RaspberryPi 3 (originally written for preview3, the nuget packages have been updated for the latest version)

tl;dr

It's not officially supported, but by compiling the native libgrpc_csharp_ext library for ARM, it is possible. I've made it easier with a set of scripts to help you compile it on your own. I've also created a nuget package too, so you don't have to compile it and a template dotnet new grpcpi to connect all the dots and supercharge your grpc pi life. Also, if your goal is to put your code on github to share (please do!), try gitstub, a dotnet cli global tool that encourages github-first development.

• To compile libgrpc_csharp_ext on your own
  • https://github.com/erikest/libgrpc_csharp_ext
• To get an already compiled libgrpc_csharp_ext.x86.so (compiled for arm) that you can add to your project right now
  • https://www.nuget.org/packages/libgrpc_csharp_ext.arm7
• To start a fresh grpc project ready to publish to linux-arm for the pi.
  • https://github.com/erikest/grpcpi
  • https://www.nuget.org/packages/grpcpi/
  • dotnet new -i grpcpi
  • note: as of .net core 3.1, there is an issue with the publish command selecting the wrong version of libgrpc_csharp_ext.x86.so when publishing for linux-arm.
• To get gitstub, so you can get all your projects up to GitHub with minimal fuss, see my other post about it or just install and discover on your own
  • dotnet tool install --global gitstub

Intro

It's a great day when you sit down to get something done with a new magic bauble, the excitement visible in your eyes, only to discover the coolness is not yet supported for 'your architecture/platform/environment'. Thus was the case when I decided to start a new client/server project with RaspberryPis. Being eager to use dotnet core, but wanting something a little more terse than json over the wire, I discovered the efficient client/server protocol gRPC, was now available for my favorite cross-platform framework - dotnet core. The tooling is new and shiny, with a canonical 'hello world' template ready to go in Visual Studio 2019 and dotnet core 3 preview 6 from the command line (dotnet new grpc). Unfortunately, the officially supported processor architectures are x64 and x86 - arm has been left out for now. However, we'll see that with a little elbow grease and hackery, we can get the new gRPC bits working with the RaspberryPi, then get the demo running and let out a satisfied chortle when the pi talks back.

Overview

We're going to look at a couple of aspects of getting this working:

1. First we'll discuss the set up - devices, development environments, tooling, etc.
2. Then we'll walk through the process of compiling the native libgrpc_csharp_ext for the arm7 processor on the RaspberryPi. We'll look at the automated build/test script and an experimental script in the repository that is used to compile the library for android as our inspiration. Don't worry, it turns out to not be too complicated and I've got you covered with scripts to make it straight forward if you want to follow along.
3. With a native library in hand, we'll examine the canned gRPC template and what we need to do to get it working on the pi
4. After crawling through that, we'll bring in a nuget package that will save us some tedium
5. Finally, we'll look at the new grpcpi template, install it and generate a new project that ties it all together. We'll work through this example to send a message from the server to the pi via gRPC, which the pi will then use to generate a series of blinks on an led. At the end we'll have all the pieces we need to start developing more meaningful applications on this platform.
6. As a bonus, we'll install gitstub, a dotnet cli global tool that connects up a new or existing project to a github repository to jump start your dev cycle and get your source open in record time.

Let's Get Started!

Tools, devices, environments, oh my!

We'll need some tools. First let's talk about our main development machine. In my case an old x64 based dell laptop. On this box, be it physical or virtual, we'll need the dotnet preview6 sdk and runtime. Those are available here. Since we're going to examine the new templates in visual studio, we'll need Visual Studio 2019

Next, for the RaspberryPi, we'll be using raspbian as the linux distro, though others will likely work fine. We'll also need a few other bits to compile the library. You can step through these individually below or you can use the following commands to download a script that should work to automate this and compile the library all in one step.

As always, don't trust me and my script until you read it and verify that it isn't a nefarious evil doer masquerading as an innocent attempt to get you compiling this obscure library.

On the pi:

wget https://github.com/erikest/libgrpc_csharp_ext/setupAndCompile.sh
chmod +x ./setupAndCompile.sh
./setupAndCompile.sh

To compile this library, we'll need the build tools, the source code and as we'll discuss later, we'll need to determine how to build just the library we're interested in, as the grpc source code contains lots of other bits that we don't need to recompile for this task.

1. Get git - if you're following along on raspbian, then you've already got git, good.
2. Install the build tools (script).
   • In the README.md for the csharp submodule, we're told to look at the BUILD.md.
   • BUILD.md tells us for linux compilation, the tools we'll need are:

$ [sudo] apt-get install build-essential autoconf libtool pkg-config
$ [sudo] apt-get install libgflags-dev libgtest-dev
$ [sudo] apt-get install clang libc++-dev

• What they don't tell you, probably because it's not supported, is that we'll also need cmake, which is listed under Windows as a build dependency and we'll see why when we go over the build process.

$ [sudo] apt-get install cmake

1. Git the source (script)

 $ mkdir ~/libgrpc_csharp_ext/
 $ cd ~/libgrpc_cshar_ext/
 $ git clone -b $(curl -L https://grpc.io/release) https://github.com/grpc/grpc
 $ cd grpc
 $ git submodule update --init

• This is also from BUILD.md
• If you want to deploy your app to the pi as a framework-dependent deployment or a framework-dependent executable, you'll need the dotnet runtime. Otherwise, publishing for a particular runtime will create a self-contained deployment), meaning all of the necessary libraries are published with the executable. You can read more about the three styles here. For this walkthrough, we'll being doing a framework-dependent executable, so we'll need the runtime.
  • As of this writing, the latest version, 3.1, can be had thus (script):

#Dependencies
sudo apt-get install curl libunwind8 gettext

#SDK needed to compile
wget https://download.visualstudio.microsoft.com/download/pr/d52fa156-1555-41d5-a5eb-234305fbd470/173cddb039d613c8f007c9f74371f8bb/dotnet-sdk-3.1.101-linux-arm.tar.gz

#Runtime needed to run apps
wget https://download.visualstudio.microsoft.com/download/pr/da60c9fc-c329-42d6-afaf-b8ef2bbadcf3/14655b5928319349e78da3327874592a/aspnetcore-runtime-3.1.1-linux-arm.tar.gz

#Make a home and extract the tars
mkdir -p $HOME/dotnet
tar zxf dotnet-sdk-3.1.101-linux-arm.tar.gz -C $HOME/dotnet
tar zxf aspnetcore-runtime-3.1.1-linux-arm.tar.gz -C $HOME/dotnet

#Set environment variables, including the path so the dotnet cli can be found
export DOTNET_ROOT=$HOME/dotnet
export PATH=$PATH:$HOME/dotnet

#push the path to the profile so it 'sticks' after restarts
sudo echo "export DOTNET_ROOT=$HOME/dotnet" >> /etc/profile
sudo echo "export PATH=$PATH:/$HOME/dotnet" >> /etc/profile

OK! We've got tools, we've got source, now we just need to put them to work

Compiling libgrpc_csharp_ext

If we again inspect the csharp submodule README, we'll see the recommended way to get compiling is:

# from the gRPC repository root
$ python tools/run_tests/run_tests.py -l csharp -c dbg --build_only

If you tried to run that now, you'd get a fair number of errors. The problem is that this run_tests.py is trying to compile all the stuff in the csharp submodule - we only want the native library. What we need to do is examine run_tests.py to see what it is doing and if we can peel away the other build steps and just get the native library out.

In run_tests.py, each language has a class which is switched using the -l flag, as in the command above -l csharp. When we do a find for csharp, we see some goodies. Specifically, let's note:

• a method called pre_build_steps. Hmm, that sounds interesting and it's pointing to another script
• a method called make_targets. Since make is typically orchestrating gcc/g++ builds and we're trying to compile a native c++ library, this is also intriguing. And it references a target called 'grpc_csharp_ext' - that sounds like just what we need!

As a bit of confirmation, let's look at the csharp/experimental folder, where there are scripts for compiling the native library for android, ios and unity. In the android script we see, after a call to cmake to set up the build environment: make -j4 grpc_csharp_ext AHA! the -j4 switch tells the compiler how many threads to spawn and then we have our grpc_csharp_ext target. Okay, we're feeling good now that we're on the right track. We don't actually have a makefile yet though, so we need to get that sorted.

Before we can make, we have to cmake a makefile, so let's make way back to that pre_build_script and take a look under the hood.

set -ex

# cd to repository root
cd "$(dirname "$0")/../../.."

mkdir -p cmake/build
cd cmake/build

cmake -DgRPC_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE="${MSBUILD_CONFIG}" ../..

cd ../../src/csharp

dotnet restore Grpc.sln

Looks promising, we've got a little directory shuffling, then a call to cmake (which should now make clear why we needed that tool installed as well), and finally a restore on the grpc solution, to get ready for the dotnet part of the build. Well, we don't want the dotnet part though, we just want the cmake call to set up the build environment and then to build our native target.

By combining what we learned from the make target (grpc_csharp_ext), with the prebuild step here, we can arrive at the commands to perform the compilation. These are executed relative to the root of our work, which if you're following along should be at ~/libgrpc_csharp_ext/.

here's our script, also located here

set -ex

#intended to be run from the repository root
cd grpc/

#create location for build files
mkdir -p cmake/build
cd cmake/build

#setup the build files
cmake -DgRPC_BUILD_TESTS=OFF -DCMAKE_BUILD_TYPE="${MSBUILD_CONFIG}" ../..

#compile libgrpc_csharp_ext.so
make -j4 grpc_csharp_ext

cp libgrpc_csharp_ext.so ../../../libgrpc_csharp_ext.x86.so

Everything there looks solid, but, why are we copying libgrpc_csharp_ext.so to libgrpc_csharp_ext.x86.so? Well, that's a perfect segue into the next topic!

gRPC template

On our dev box, we can create a new grpc project using the included template

mkdir grpcDemo
cd grpcDemo
dotnet new grpc

If we then publish the client for the linux-arm runtime identifier (RID)

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm --self-contained false

If we don't include the --self-contained false, then we'll get a whole bunch of framework libraries as well. This could be good for portability, since you can just push the whole publish folder to any pi and get going, but for iterative development, doing the one time set up of the framework to avoid the extra file copy seems like the way to go.

we'll find in the /bin/debug/netcoreapp3.0/linux-arm/publish folder two copies of the libgrpc_csharp_ext, one .x86.so and one .x64.so

If we deploy this to an existing directory on the pi, for example by using scp

scp ./grpcDemo.Client/bin/debug/netcoreapp3.0/linux-arm/publish pi@<PI IP>:~/grpc.Client

as noted here, the scp command no longer supports the handy /. which would copy the files without including the directory they are in. So we get a 'publish' folder on the pi when we don't want one. So it goes.

and run it from the pi

cd ~/grpc.Client/publish
chmod +x grpcDemo.Client
./grpcDemo.Client

we'll get an error like this person did on SO

Which basically says it can't load the libgrpc_csharp_ext.x86.so library. Here's where our problem and the beginning of all this work started - this library isn't compiled for the arm processor, but it is still being included in the publish output and is being referenced by the NativeExtension class. When we dig into the repository and look at the source, specifically here we notice that the file name it looks for depends on the OS, linux in this case which needs a static library, .so, and the architecture. Despite the pi having a 64 bit processor, the reality is that the OS is ARM32, thus the architecture string gets set as x86. All that is to say, on a raspberrypi running raspbian, the file name it tries to load is libgrpc_csharp_ext.x86.so - so, let us give it what it wants!

On the pi

cp ~/libgrpc_csharp_ext/libgrpc_csharp_ext.x86.so ~/grpc.Client/publish
./grpcDemo.Client

This time it should pause for about 15-20 seconds before we see

Unhandled Exception: Grpc.Core.RpcException: Status(StatusCode=Unavailable, Detail="Connect Failed")
   at gRPCpi.Program.Main(String[] args) in C:\Users\Geocoder\Source\Repos\gRPC Template\content\gRPCpi\src\gRPCpi.Clien
t\Program.cs:line 26
   at gRPCpi.Program.<Main>(String[] args)
Aborted

But, hey, that means it is running! It just couldn't connect to the server. We're getting closer to shaving our yak! Well, it turns out it's trying to connect to the server on the localhost by default. So we need to make a couple of modifications, since our server will be running over a network on our dev machine.

• In the Client's program.cs, we need to aim the grpc client at the server's (dev box) ip address. If you don't know yours, ipconfig on windows or ifconfig on linux will get you there.

 //CHANGE THIS TO YOUR SERVER ADDRESS
var serverIpAddress = "YOUR IP";

var channel = new Channel($"{serverIpAddress}:" + port, ChannelCredentials.Insecure);

• We also need the server to be listening on that ip address and port and by default it is not, it listens only on localhost. To open this up to our network IP, we need to make a change to the kestrel server configuration:

Host.CreateDefaultBuilder(args)
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
        webBuilder.UseUrls("http://*:50051");
    });

Here we've said listen on all (*) IP addresses bound to the network interface.

Now, let's republish the client to the pi. So on the dev box again, at the project root

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm
scp ./grpcDemo.Client/bin/debug/netcoreapp3.0/linux-arm/publish pi@<PI IP>:~/grpc.Client

Before we run the client again, let's get the server started and ready. In Visual Studio you can just run a new debug instance, or from the command line

dotnet run --project grpcDemo/grpcDemo.Server/grpcDemo.Server.csproj

Then back on the pi, let's run the client, after we overwrite the libgrpc_csharp_ext library again

cp ~/libgrpc_csharp_ext/libgrpc_csharp_ext.x86.so ~/grpc.Client/publish
cd ~/grpc.Client/publish
chmod +x grpcDemo.Client
./grpcDemo.Client

If all has gone to plan, we should get this

Greeting: Hello GreeterClient

Chortle! woo! we did it - time to hang up our hats, jobs done... Well not exactly, while this shows that we can 'get it working', what we really want is to have the correct library come across with all the other published artifacts, so we can continue development without so many steps. Sure, we could write more scripts to make this less painful, but isn't there a better way?

Libgrpc_csharp_ext.arm7 Nuget Package

So far, we've manually overwritten the 'wrong' native library with our arm compiled version. What we really want is for the correct library to be brought in during publish - so whichever runtime we publish for, it will have the correct version. If you publish for windows or linux or osx, you'll notice the publish command correctly copies over the version needed for each runtime.

What's going on here?

In the Grpc.Core nuget package, which is referenced in the template, are copies of that native library for each of win, linux, and osx runtimes. You can verify this by examining the package cache at YourUserName.nuget\packages\grpc.tools<version>\runtimes folder. The publish command uses this convention to find additional runtime specific components to copy to the build and publish output. This same mechanism can be used to provide the library for the linux-arm runtime. Let's go ahead and add a reference to this package

dotnet add grpcDemo\grpcDemo.Client package libgrpc_csharp_ext.arm7

Then publish again

dotnet publish grpcDemo/grpcDemo.Client/grpcDemo.Client.csproj -r linux-arm --self-contained false

We can verify (by file size) that our version of the library has been copied over and is ready for deployment. Hey, now this project has legs and quick iterations are at hand!

Next up

In the next post, we'll finish all this work up by making it even easier to get started by adding a new project template which puts it all together. Then we'll use gitstub to jumpstart a new open source client/server blinking led project that everybody will surely want!