DEV Community

jeikabu
jeikabu

Posted on • Originally published at rendered-obsolete.github.io on

Loading Native Libraries in C#

#csharp #pinvoke #osx #native

While possible for C# to call functions in native libraries, you must avoid mixing 32 and 64-bit binaries at runtime lest you invite the wrath of BadImageFormatException. Woe unto you.

Different hardware and OSes have different caveats. This is mostly talking about 64-bit Windows 10 and occasionally OSX (64 bit).

“Any CPU”

.Net assemblies can be compiled with a platform target of “Any CPU”. This SO answer covers it nicely:

  • Binary targetting “Any CPU” will JIT to “any” architecture (x86, x64, ARM, etc.)- but only one at a time.
  • On 64-bit Windows 10, an “Any CPU” binary will JIT to x64, and it can only load x64 native DLLs.

What happens if you try to load a 32-bit assembly into a 64-bit process (or vice-versa)? BadImageFormatException.

Windows “Hack”

Pinvoke is one approach to call functions in native DLLs from C#.

For several years I’ve used a well-known trick to selectively load 32/64-bit native libraries in Windows desktop applications:

class ADLWrapper
{
    [DllImport("LibADLs")]
    static extern int LibADLs_GetAdapterIndex(IntPtr ptr);

    static ADLWrapper()
    {
        // If 64-bit process, need to load 64-bit native dll. Otherwise, 32-bit dll.
        // Both dlls need to have same filename and be dllexport'ing the same functions.
        if (System.Environment.Is64BitProcess)
        {
            var handle = LoadLibraryEx(pathTo64bitLibs + "LibADLs.dll", IntPtr.Zero, 0);
            if (handle != IntPtr.Zero)
            {
                //...
            }
        }
        else
        {
            // Load 32-bit dll
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

This assumes you have two identically named shared libraries with different paths. For example, x86/libadls.dll and x64/libadls.dll. It’s slight abuse of the way dynamic libraries are found. In this case a snippet of C# wrapper around a support library for AMD’s Display Library (ADL).

Prior to the introduction of Is64BitProcess in .Net 4.0, the value of (IntPtr.Size == 8) could be used instead.

When dealing with dynamic libraries with different names (as is the case with ADL), because the argument to DllImportAttribute must be a constant we have the unflattering:

interface ILibADL
{
    Int64 GetSerialNumber(int adapter);
}

class LibADL64 : ILibADL
{
    [DllImport("atiadlxx.dll")] // 64-bit dll
    static extern int ADL_Adapter_SerialNumber_Get(int adapter, out Int64 serial);

    public Int64 GetSerialNumber(int adapter)
    {
        Int64 serial;
        if (ADL_Adapter_SerialNumber_Get(adapter, out serial) == 0)
        {
            return serial;
        }
        return -1;
    }
}

class LibADL32 : ILibADL
{
    [DllImport("atiadlxy.dll")] // 32-bit dll
    static extern int ADL_Adapter_SerialNumber_Get(int adapter, out Int64 serial);

    //...
Enter fullscreen mode Exit fullscreen mode

And then somewhere else:

if (Environment.Is64BitProcess)
{
    adl = new LibADL64();
}
else
{
    adl = new LibADL32();
}
Enter fullscreen mode Exit fullscreen mode

Simple and effective. But doesn’t work on OSX (or Linux).

GetDelegateForFunctionPointer

Another approach I first came across when experimenting with NNanomsg uses GetDelegateForFunctionPointer():

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate int nn_socket_delegate(int domain, int protocol);
public static nn_socket_delegate nn_socket;

static void InitializeDelegates(IntPtr nanomsgLibAddr, NanomsgLibraryLoader.SymbolLookupDelegate lookup)
{
    nn_socket = (nn_socket_delegate)Marshal.GetDelegateForFunctionPointer(lookup(nanomsgLibAddr, "nn_socket"), typeof(nn_socket_delegate));
}
Enter fullscreen mode Exit fullscreen mode

Where lookup is GetProcAddress() on Windows (and dlsym() on posix platforms).

A similar approach is used by gRPC.

Ok, but unwieldly for large APIs like nng (which is where we’re headed).

SWIG et al.

It’s hard to talk about interfacing with native code and not come across SWIG (or other similar technologies).

Over the last 15 years we’ve crossed paths a few times. The first time being when tasked with creating a python 1.5 wrapper for a Linux virtual device driver. Most recently, a half-hearted attempt to use it with AllJoyn.

But it always ends the same way: frustration trying to debug my (miss-)use of a non-trivial tool, and befuddlement with the resulting robo-code.

“Modern” Nupkg

Necessity being the mother of invention, supporting multiple platforms begat advances in nuget packaging to address the issue.

We’ve been trying to get a csnng nupkg containing a native library (nng) working on OSX:

  1. Build the dynamic library (libnng.dylib): mkdir build && cd build && cmake -G Ninja -DBUILD_SHARED_LIBS=ON .. && ninja
  2. Copy into runtimes/osx-x64/native of the nupkg

In RepSocket.cs:

[DllImport("nng", EntryPoint = "nng_rep0_open", CallingConvention = Cdecl)]
[return: MarshalAs(I4)]
private static extern int __Open(ref uint sid);
Enter fullscreen mode Exit fullscreen mode

Which works fine on Windows, but on OSX:

Mono: DllImport error loading library 'libnng': 'dlopen(libnng, 9): image not found'.
Mono: DllImport error loading library 'libnng.dylib': 'dlopen(libnng.dylib, 9): image not found'.
Mono: DllImport error loading library 'libnng.so': 'dlopen(libnng.so, 9): image not found'.
Mono: DllImport error loading library 'libnng.bundle': 'dlopen(libnng.bundle, 9): image not found'.
Mono: DllImport error loading library 'nng': 'dlopen(nng, 9): image not found'.
Mono: DllImport error loading library '/Users/jake/test/bin/Debug/libnng': 'dlopen(/Users/jake/test/bin/Debug/libnng, 9): image not found'.
Mono: DllImport error loading library '/Users/jake/test/bin/Debug/libnng.dylib': 'dlopen(/Users/jake/test/bin/Debug/libnng.dylib, 9): image not found'.
Mono: DllImport error loading library '/Users/jake/test/bin/Debug/libnng.so': 'dlopen(/Users/jake/test/bin/Debug/libnng.so, 9): image not found'.
...
Mono: DllImport error loading library '/Library/Frameworks/Mono.framework/Versions/5.12.0/lib/libnng.dylib': 'dlopen(/Library/Frameworks/Mono.framework/Versions/5.12.0/lib/libnng.dylib, 9): image not found'.
...
Enter fullscreen mode Exit fullscreen mode

The additional library loading information is enabled by setting environment variables:

# .NET Framework
MONO_LOG_MASK=dll
MONO_LOG_LEVEL=info
# .NET Core
DYLD_PRINT_LIBRARIES=YES
Enter fullscreen mode Exit fullscreen mode
  • In Visual Studio for Mac: Right-click project Options->Run.Configurations.Default under “Environment Variables”
  • Visual Studio Code: in "configurations" section of .vscode/launch.json add "env": { "DYLD_PRINT_LIBRARIES":"YES" }

One solution comes from Using Native Libraries in ASP.NET 5 blog:

  1. Preload the dylib (similar to Windows)
  2. Use DllImport("__Internal")

Code initially based off Nnanomsg:

static LibraryLoader()
{
    // Figure out which OS we're on. Windows or "other".
    if (Environment.OSVersion.Platform == PlatformID.Unix ||
                Environment.OSVersion.Platform == PlatformID.MacOSX ||
                // Legacy mono value. See https://www.mono-project.com/docs/faq/technical/
                (int)Environment.OSVersion.Platform == 128)
    {
        LoadPosixLibrary();
    }
    else
    {
        LoadWindowsLibrary();
    }
}

static void LoadPosixLibrary()
{
    const int RTLD_NOW = 2;
    string rootDirectory = AppDomain.CurrentDomain.BaseDirectory;
    string assemblyDirectory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);

    // Environment.OSVersion.Platform returns "Unix" for Unix or OSX, so use RuntimeInformation here
    var isOsx = System.Runtime.InteropServices.RuntimeInformation.IsOSPlatform(OSPlatform.OSX);
    string libFile = isOsx ? "libnng.dylib" : "libnng.so";
    // x86 variants aren't in https://docs.microsoft.com/en-us/dotnet/core/rid-catalog
    string arch = (isOsx ? "osx" : "linux") + "-" + (Environment.Is64BitProcess ? "x64" : "x86");

    // Search a few different locations for our native assembly
    var paths = new[]
        {
            // This is where native libraries in our nupkg should end up
            Path.Combine(rootDirectory, "runtimes", arch, "native", libFile),
            // The build output folder
            Path.Combine(rootDirectory, libFile),
            Path.Combine("/usr/local/lib", libFile),
            Path.Combine("/usr/lib", libFile)
        };

    foreach (var path in paths)
    {
        if (path == null)
        {
            continue;
        }

        if (File.Exists(path))
        {
            var addr = dlopen(path, RTLD_NOW);
            if (addr == IntPtr.Zero)
            {
                // Not using NanosmgException because it depends on nn_errno.
                var error = Marshal.PtrToStringAnsi(dlerror());
                throw new Exception("dlopen failed: " + path + " : " + error);
            }
            NativeLibraryPath = path;
            return;
        }
    }

    throw new Exception("dlopen failed: unable to locate library " + libFile + ". Searched: " + paths.Aggregate((a, b) => a + "; " + b));
}

[DllImport("libdl")]
static extern IntPtr dlopen(String fileName, int flags);

[DllImport("libdl")]
static extern IntPtr dlerror();

[DllImport("libdl")]
static extern IntPtr dlsym(IntPtr handle, String symbol);
Enter fullscreen mode Exit fullscreen mode

The use of System.Runtime.InteropServices.RuntimeInformation came from this blog.

We construct a path based on the host OS (Linux vs OSX, 32 vs 64-bit), and pass it to dlopen() to pre-load the shared library.

Note the absence of file extension with [DllImport("libdl")]. This will load libdl.dylib on OSX and libdl.so on Linux.

Change the imported function to:

[DllImport("__Internal", EntryPoint = "nng_rep0_open", CallingConvention = Cdecl)]
[return: MarshalAs(I4)]
private static extern int __Open(ref uint sid);
Enter fullscreen mode Exit fullscreen mode

Debug output:

Native library: /Users/jake/test/bin/Debug/runtimes/osx-x64/native/libnng.dylib
Mono: DllImport attempting to load: '__Internal'.
Mono: DllImport loaded library '(null)'.
Mono: DllImport searching in: '__Internal' ('(null)').
Mono: Searching for 'nng_rep0_open'.
Mono: Probing 'nng_rep0_open'.
Mono: Found as 'nng_rep0_open'.
Enter fullscreen mode Exit fullscreen mode

Works, but requiring every DllImport use "__Internal" leaves a lot to be desired. There’s a few alternatives:

  • Use T4 template (or other script) to juggle the pinvoke imports needed by each platform
  • Setting DYLD_xxx_PATH
  • .config file with <dllmap>
  • Just copying the dylib to the output path as part of the build (arbitrary scripts can be included as part of a nupkg)

Again, this approach works, but has the following drawbacks:

  • Over-reliance on build/package tooling that can be a hassle to debug and get working correctly
  • On Windows, requires setting Platform target to something other than Any CPU
  • Argument to DllImport must be compile-time constant and requires massaging to get “magic” working on all platforms

One Load Context to Rule Them All

Came across “Best Practices for Assembly Loading”.

Started looking for information on these “load contexts” and found an interesting document:

Custom LoadContext can override the AssemblyLoadContext.LoadUnmanagedDll method to intercept PInvokes from within the LoadContext instance so that can be resolved from custom binaries

Ho ho.

Also came across this post of someone that works on ASP.NET Core. He’s using AssemblyLoadContext to wrangle plugins, but mentions LoadUnmanagedDll is “the only good way to load unmanaged binaries dynamically”.

To get started, need System.Runtime.Loader package: dotnet add nng.NETCore package system.runtime.loader

First attempt hard-coding paths and filenames:

public class ALC : System.Runtime.Loader.AssemblyLoadContext
{
    protected override Assembly Load(AssemblyName assemblyName)
    {
        if (assemblyName.Name == "nng.NETCore")
            return LoadFromAssemblyPath("/Users/jake/nng.NETCore/bin/Debug/netstandard2.0/nng.NETCore.dll");
        // Return null to fallback on default load context
        return null;
    }
    protected override IntPtr LoadUnmanagedDll(string unmanagedDllName)
    {
        // Native nng shared library
        return LoadUnmanagedDllFromPath("/Users/jake/nng/build/libnng.dylib");
    }
}
Enter fullscreen mode Exit fullscreen mode

DllImported methods must be static so can’t use Activator.CreateInstance() to easily get at them. Could probably use reflection to extract them all, but that would be unwieldy.

I think the key is from that LoadContext design doc:

If an assembly A1 triggers the load of an assembly C1, the latter’s load is attempted within the LoadContext instance of the former

Basically, once a load context loads an assembly, subsequent dependent loads go through the same context. So, I moved a factory I use to create objects for my tests into the assembly with the pinvoke methods:

TestFactory factory;

[Fact]
public async Task PushPull()
{
    var alc = new ALC();
    var assem = alc.LoadFromAssemblyName(new System.Reflection.AssemblyName("nng.NETCore"));
    var type = assem.GetType("nng.Tests.TestFactory");
    factory = (TestFactory)Activator.CreateInstance(type);
    //...
Enter fullscreen mode Exit fullscreen mode

We can’t easily call the static pinvoke methods directly, but we can use a custom load context to instantiate a type which then calls the pinvokes.

I rarely find exceptions exciting, but this one is:

Exception thrown: 'System.InvalidCastException' in tests.dll: 
'[A]nng.Tests.TestFactory cannot be cast to [B]nng.Tests.TestFactory. 
Type A originates from 'nng.NETCore, Version=0.0.1.0, Culture=neutral, PublicKeyToken=null' in the context 'Default' at location '/Users/jake/nng.NETCore/tests/bin/Debug/netcoreapp2.1/nng.NETCore.dll'. 
Type B originates from 'nng.NETCore, Version=0.0.1.0, Culture=neutral, PublicKeyToken=null' in the context 'Default' at location '/Users/jake/nng.NETCore/tests/bin/Debug/netcoreapp2.1/nng.NETCore.dll'.'
Enter fullscreen mode Exit fullscreen mode

Different load contexts, different types.

I’m referencing the nng.NETCore assembly (which contains the pinvokes) in my test project and also trying to load it here. How am I supposed to use a type I don’t know about? This is an opportunity for a C# feature I never use, dynamic:

dynamic factory = Activator.CreateInstance(type);
//...
var pushSocket = factory.CreatePusher(url, true);
Enter fullscreen mode Exit fullscreen mode

Test passes, hit breakpoints most of the places I expect (neither VS Code nor VS for Mac can hit breakpoints through dynamic), but if I set DYLD_PRINT_LIBRARIES my assemblies are conspiculously absent:

dyld: loaded: /usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/System.Globalization.Native.dylib
dyld: loaded: /usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/System.Native.dylib
Microsoft (R) Test Execution Command Line Tool Version 15.7.0
Copyright (c) Microsoft Corporation. All rights reserved.

dyld: loaded: /usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/System.Security.Cryptography.Native.Apple.dylib
Starting test execution, please wait...

Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.3259 Seconds
dyld: unloaded: /usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/libhostpolicy.dylib
dyld: unloaded: /usr/local/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/libhostpolicy.dylib
Enter fullscreen mode Exit fullscreen mode

It would seem AssemblyLoadContext.LoadFrom*() doesn’t use dyld? Hmm… not sure about that.

Obviously, I don’t want to use dynamic all over the place. I refactored things; remove the test assembly reference to the pinvoke assembly, and introduce a “middle-man”/glue assembly containing interfaces both use:

Assembly Project References Dynamically Loads Notes
“tests” interfaces pinvoke Unit/integration tests
“interfaces” interfaces of high-level types using P/Invoke
“pinvoke” interfaces P/Invoke methods and wrapper types that use them

That enables me to write the very sane:

[Fact]
public async Task PushPull()
{
    var alc = new ALC();
    var assem = alc.LoadFromAssemblyName(new System.Reflection.AssemblyName("nng.NETCore"));
    var type = assem.GetType("nng.Tests.TestFactory");
    IFactory<NngMessage> factory = (IFactory<NngMessage>)Activator.CreateInstance(type);
    var pushSocket = factory.CreatePusher("ipc://test", true);
Enter fullscreen mode Exit fullscreen mode

And now I can load native binaries from anywhere I like.

Out of curiousity, I wondered if I could add a reference back to the pinvoke, and after the native library had been successfully called use it directly:

Native.Msg.UnsafeNativeMethods.nng_msg_alloc(out var msg, UIntPtr.Zero);
Enter fullscreen mode Exit fullscreen mode

Nope:

nng.Tests.PushPullTests.PushPull [FAIL]
[xUnit.net 00:00:00.5527750] System.DllNotFoundException : Unable to load shared library 'nng' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libnng, 1): image not found
Enter fullscreen mode Exit fullscreen mode

The “default” load context knows not about the native library- it’s only in my custom context.

I get the feeling there may be a simpler way to achieve what I want. Need to investigate this a bit more.

Performance

None of this is “zero-cost”.

The NNanomsg source code references this blog which mentions the SuppressUnmanagedCodeSecurity attribute may improve performance significantly. The security implications aren’t immediately clear from the documentation, but it sounds like it may only pertain to operating system resources; it can’t be any less safe than calling the library from native code…

There’s numerous methods to manipulate messages and I’ll write them in pure C# to avoid doing pinvoke for trivial string operations.

Top comments (6)

Collapse
 
isatindersingh profile image
Satinder Singh •

Hi,
I must say this is nice detailed article, well I just came to know about a broken link in this post which pointing to one of my article. Can you update it with working link
i.e codepedia.info/dotnet-core-to-dete...
instead of
json-formatter.codepedia.info/dotn...

Thanks & Happy Coding

Collapse
 
jeikabu profile image
jeikabu •

Done

Collapse
 
paulcc profile image
paulcc • • Edited

Hello, I found your article while searching for solutions to the DllImport style difference in calling native code ( C/C++ ) between Windows and iOS.
I am trying to get common .netstandard2.0 implementation of native interface that needs to work both on Windows and iOS.
From reading your text, I got the impression that it's possible on Windows to preload a dll, which somehow makes the symbols from the DLL available, and then use the P/Invoke DllImport("__Internal") style. My thinking is that on the Windows I'd preload the DLL, and then proceed to using the same DllImport("__Internal") style, thus getting a single C# implementation working on both targets.
In my tests on Windows, the DllImport("__Internal") throws an exception complaining that the DLL does not exist - so it seems the "__Internal" is taken as DLL file name.

Could you, please, tell me if I understood your text correctly, and if so, suggest the simplest test code.
I am a C++ programmer just trying to figure this out....
Thanks, Paul.

Collapse
 
eri0o profile image
eri0 •

This was a great post, thanks for writing it. Running native code, with cross platform builds on C# and .NET is something that I couldn't find much material on the internet.

Collapse
 
jeikabu profile image
jeikabu •

Glad you enjoyed it. Wish I had more time to investigate it further.

Collapse
 
borrrden profile image
Jim Borden •

Glad somebody else is worry about this and not just me. In the end the "copy the files with nupkg script and then load based on IntPtr size" has served me pretty well!

Read next

moh_moh701 profile image

c# Clean Code: Best Practices for new, Operators, and using

mohamed Tayel -

sebastiandevelops profile image

Introducing PinataNET: A Simple C# Wrapper for the Pinata API

Sebastian Van Rooyen -

canro91 profile image

TIL: How To Use the Specification Pattern in C# To Simplify Repositories

Cesar Aguirre -

juarezasjunior profile image

Microservices Integration and Message Processing with MassTransit and RabbitMQ

Juarez Júnior -