By: Arpad Borsos

Like mechanics who restore their own cars or plastic surgeons who self-rhinoplasty, our developers put their skills to interesting uses during their free time. Here, Native Platform Engineer Arpad Borsos breaks down how memory mappings and dynamic library loading works and how it relates to native Android libraries loaded from APKs.

Libraries are key to modular programming, as they offer functionality in a single unit which can be shared with other developers. As you’re no doubt aware, there are two types of libraries: static and dynamic. Static libraries are directly embedded into your application at build time, while dynamic libraries are linked when starting the application, or at any later point.

This concept is important because dynamic libraries can be updated without modifying the application itself for things like fixing security issues or improving performance. Applications can also be split into multiple, dynamically-loaded libraries for organizational reasons, or when an application consists of a User Interface and a separate service running in the background.

When working with the Android NDK, native libraries written in C, Rust, or similar low-level languages are loaded dynamically by the Java layer. That is how sentry-native gets loaded into Android Apps that use the NDK.

Usually, these libraries are individual files on disk. You might have seen a few of these “.dll” files next to Windows Applications, which are similar on other operating systems. The dynamic loader on Android, which is itself a system library, has the ability to load libraries directly from Android “.apk” packages without needing to first extract them to disk. This is quite beneficial, as it saves precious disk space on your mobile device.

So far, sentry-native, and in turn, NDK support for our Android SDK, relied on these files being extracted to disk. This has created a lot of friction — especially for new customers — as newer Android Versions no longer extract “.apk” packages by default. To get around this, application developers had to set some explicit configuration flags, many of which were frequently overlooked when setting up Sentry for Android.

Starting with version 5 of our Android SDK, we added support for libraries loaded from “.apk”, which will remove the need to change configuration flags while also improving disk space usage of Apps that use the Sentry SDK.

Let us take a deeper dive into how all of this works, and what we need to change in order to support this use case.

Which libraries are loaded?

On most platforms, you can query the list of loaded libraries directly from the dynamic loader via an API. For example, Windows has the Tool Help Library, and on Apple platforms there are some dyld functions available. Unfortunately, Linux has no standardized userspace tools. While GNU/Linux has the dl_iterate_phdr function, it is notably not available on older Android systems (Bionic Status lists the API as available starting with API 21, aka Android 5, released at the end of 2014). That means in order to support ancient Android versions, you will need to get the list of loaded libraries from somewhere else.

Standard practice here is to find the mapped ELF files by parsing the memory map info from /proc/XXX/maps. This is what Breakpad (in two places), Crashpad, Android’s libunwindstack, and LLDB do. In my opinion, they take this approach because they’re outside observers in the sense that they just can’t query the dynamic loader from inside the process.

This approach makes sense: it’s how I approached loading libraries for sentry-native. That said, you have to be careful to cover all cases — notably .so files loaded directly from inside Android .apk packages. And so, I went about finding ways to support these .apk cases.

The /proc/X/maps format

In Linux, all the executables and libraries are ELF files. Cloudflare has a great tutorial on how a loader parses and processes these ELF files. The documentation for the /proc/X/maps output format is described in this manpage. The format includes the start/end of the virtual address space, permission information, information about the inode (file), and the offset inside that file.

While there are cases when a library needs just one mapping, most of the time, it’s split into two or more mappings. Usually that consists of a read-only mapping that includes the ELF headers and metadata, and an executable mapping that holds the actual program code. On my Linux system, I saw up to six mappings for a single file:

7f8cd3467000-7f8cd3475000 r--p 00000000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd3475000-7f8cd34da000 r-xp 0000e000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd34da000-7f8cd34f6000 r--p 00073000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd34f6000-7f8cd34f7000 ---p 0008f000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd34f7000-7f8cd34fa000 r--p 0008f000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0
7f8cd34fa000-7f8cd34fc000 rw-p 00092000 00:1c 7597971                    /usr/lib/libcurl.so.4.7.0

The interesting case here is that the fourth mapping is not readable, and basically creates a gap in the address space.

These two mappings load the exact same libraries, once extracted to disk, once directly from the apk:

```77a85dbda000-77a85dbdd000 r-xp 00000000 fd:05 40992 /data/app/x/y/lib/x86_64/libsentry-android.so
77a85dbdd000-77a85dbde000 ---p 00000000 00:00 0
77a85dbde000-77a85dbdf000 r--p 00003000 fd:05 40992 /data/app/x/y/lib/x86_64/libsentry-android.so
77a85dc15000-77a85dd6c000 r-xp 00000000 fd:05 40991 /data/app/x/y/lib/x86_64/libsentry.so
77a85dd6c000-77a85dd6d000 ---p 00000000 00:00 0
77a85dd6d000-77a85dd79000 r--p 00157000 fd:05 40991 /data/app/x/y/lib/x86_64/libsentry.so
77a85dd79000-77a85dd7a000 rw-p 00163000 fd:05 40991 /data/app/x/y/lib/x86_64/libsentry.so

```77a85dbf0000-77a85dbf3000 r-xp 00001000 fd:05 40977                      /data/app/x/y/base.apk
77a85dbf3000-77a85dbf4000 ---p 00000000 00:00 0
77a85dbf4000-77a85dbf5000 r--p 00004000 fd:05 40977                      /data/app/x/y/base.apk
77a85dc15000-77a85dd6c000 r-xp 00006000 fd:05 40977                      /data/app/x/y/base.apk
77a85dd6c000-77a85dd6d000 ---p 00000000 00:00 0
77a85dd6d000-77a85dd79000 r--p 0015d000 fd:05 40977                      /data/app/x/y/base.apk
77a85dd79000-77a85dd7a000 rw-p 00169000 fd:05 40977                      /data/app/x/y/base.apk

The mappings are basically the same — it’s just that in the case of the base .apk, the file offsets are different. And the Android loader, again, inserts a non-readable gap in between.

So how do we get the library list from there?

So far, the sentry-native modulefinder implementation has been a bit too conservative. Because of concerns reading arbitrary memory, we mmap-ed the file into memory and tried to extract ELF headers. Unfortunately, that approach doesn’t work with those libraries that are loaded directly from apk files, as the ELF headers are at a certain offset in that file. Plus, as we demonstrated above, some issues related to non-contiguous mappings and double mappings caused problems in the old implementation, as it worked based on the filename that it saw.

So my new approach is to keep track of readable mappings, their file offsets, and the gaps between them. For each readable mapping, I am looking for the magic ELF signature. If I find one, I process the previously saved mappings, while also taking care of possible duplicates.

This approach still has unanswered questions. One is trying to read arbitrary memory. I think I’m pretty safe as I’m only considering readable mappings, but one improvement would be to use process_vm_readv here. That said, I have also seen problems with using that mapping on Android. Another potential issue is how to correctly deal with mappings which have gaps in them, or even ones that appear multiple times. The ELF file might instruct the loader to load the executable code at a different offset to the ELF header in RAM than the offset on disk — or it might not. It very much depends on how well we use this information to post-process crash reports.

This problem is not unique to the way that sentry-native used to read the list of libraries. We also saw some breakpad tools get this wrong by creating minidumps with invalid mappings that fail later on in the post-processing pipeline.

Loading libraries is a non-trivial problem, and I am confident that I’m not the only one struggling with it. And make no mistake: it’s some work to investigate failures and patch the relevant code for such a specific use case. But with Android adoption picking up more and more traction, it’s necessary work that will save space for your user - and stress on yourself.