Amazon Lumberyard, like CryEngine, is mostly C++. Some tools are done in Python, and Lua can additionally be used for scripting in-game. But, it being primarily C++ opens up the possibility of using languages that support native code bindings- like Rust.
As an experiment, I wanted to look at how feasible it would be to use Rust, starting with the simplest case- integrating a Rust static library.
New to Amazon Lumberyard? Check out my series on Lumberyard basics.
Bindgen
bindgen uses clang/LLVM to generate Rust FFI bindings from C/C++ header files. This makes it easy to call functions defined in native libraries and work with native types and data. The user guide provides a good introduction.
In Cargo.toml
:
[package]
name = "lmbr_sys"
version = "0.1.0"
edition = "2018"
[build-dependencies]
bindgen = "0.51"
In the build script, build.rs
:
use std::{env, path::PathBuf};
fn main() {
let builder = bindgen::Builder::default()
.header("wrapper.hpp")
.clang_arg("-I<Lumberyard root>/dev/Code/Framework/AzCore")
.enable_cxx_namespaces()
.generate_inline_functions(true)
.whitelist_type("AZ::Debug::.*")
;
let bindings = builder.generate().expect("Unable to generate bindings");
let out_path = PathBuf::from(env::var("OUT_DIR").unwrap());
bindings
.write_to_file(out_path.join("bindings.rs"))
.expect("Couldn't write bindings");
}
Builder
provides a number of methods to control what and how bindings are generated. clang_arg()
can pass options used when building a C++ program.
OUT_DIR
is an environment variable set in build scripts to the path in the target/
directory containing intermediate files.
wrapper.hpp
has declarations we want to generate bindings for. For starters, it only contains:
#include <AzCore/Debug/Trace.h>
AZ::Debug::Trace
contains static methods used by the tracing macros to output text to the Lumberyard console/log.
Unfortunately, cargo build
resulted in a Segmentation Fault.
Debugging Bindgen
The end of CONTRIBUTING.md has some good information on working with and debugging bindgen.
creduce is used to help produce minimal repro cases. It’s pretty neat, what starts as a 3.4MB pre-processed C++ source file ends up 4 lines that fail the same way.
Key to this is a “predicate script” that determines if the behavior you’re trying to isolate has occurred. My first predicate.sh
:
#!/usr/bin/env bash
# Exit the script with a nonzero exit code if:
# * any individual command finishes with a nonzero exit code, or
# * we access any undefined variable.
set -eu
~/projects/rust-bindgen/csmith-fuzzing/predicate.py \
--expect-bindgen-fail \
--bindgen-args "-- -std=c++14" \
./wrapper.hpp
Running creduce on our header file produces:
creduce ./predicate.sh ./wrapper.hpp
# Output
}
Indeed, an invalid source file is the shortest possible way to get bindgen to fail. I need bindgen to partially work, but not output bindings:
~/projects/rust-bindgen/csmith-fuzzing/predicate.py \
--expect-bindgen-fail \
--bindgen-args "-- -std=c++14" \
--bindgen-grep "Unhandled cursor kind 24" \
./wrapper.hpp
Produces:
template <class d> struct g {
d e;
static const long f = __alignof__ (e);
};
Using a Custom LLVM
Turns out this problem was the same as an already reported issue and requires a fix in llvm.
Follow the “Getting Started” guide to build llvm trunk. The requirements don’t seem to mention this, but make sure you’ve got plenty of memory. A Linux VM with 8GB RAM and 1.5GB of swap space runs out of memory linking.
See if there’s any build options that interest you:
cmake -DLLVM_TARGETS_TO_BUILD="X86" -DLLVM_ENABLE_PROJECTS=clang -Thost=x64 ../llvm
Once the build succeeds, using this custom version of llvm/clang is as simple as setting the LIBCLANG_PATH
environment variable:
LIBCLANG_PATH=<llvm-project>/build/lib cargo build
Or, if using Visual Studio and PowerShell:
$env:LIBCLANG_PATH="<llvm-project>/build/Debug/bin"
cargo build
This gets us past the seg fault and generates the first version of our Rust FFI bindings. However, now Rust compilation fails.
Conflicting Types
The initial generated bindings fail to compile with rustc:
error[E0391]: cycle detected when processing `root::AZ::u32`
--> /XXX/lmbr/target/debug/build/lmbr_sys-4d14e371f0340e58/out/bindings.rs:91:24
|
91 | pub type u32 = u32;
| ^^^
|
= note: ...which again requires processing `root::AZ::u32`, completing the cycle
note: cycle used when processing `root::AZ::Debug::ProfilerRegister::m_systemId`
--> /XXX/lmbr/target/debug/build/lmbr_sys-4d14e371f0340e58/out/bindings.rs:547:33
|
547| pub m_systemId: root::AZ::u32,
| ^^^^^^^^^^^^^
The issue here being Lumberyard defines u32
and others that conflict with Rust’s built-in types. In AzCore/base.h
:
namespace AZ
{
#if AZ_TRAIT_COMPILER_INCLUDE_CSTDINT // Defined on Windows/Mac/Linux/Android
typedef int8_t s8;
typedef uint8_t u8;
typedef int16_t s16;
typedef uint16_t u16;
typedef int32_t s32;
typedef uint32_t u32;
# if AZ_TRAIT_COMPILER_INT64_T_IS_LONG // int64_t is long
typedef signed long long s64;
typedef unsigned long long u64;
# else
typedef int64_t s64;
typedef uint64_t u64;
# endif
//...
Causes bindgen to generate:
pub mod AZ {
//...
pub type s8 = i8;
pub type u8 = u8; // error[E0391]: cycle detected
pub type s16 = i16;
pub type u16 = u16; // error[E0391]: cycle detected
pub type s32 = i32;
pub type u32 = u32; // error[E0391]: cycle detected
pub type s64 = i64;
pub type u64 = u64; // error[E0391]: cycle detected
One possible work-around:
.blacklist_type(r"AZ::u\d{2,3}") // Ban AZ::u32, etc. generated by C typedefs
.raw_line("type U32 = u32;") // Create top-level type aliases
.raw_line("type U64 = u64;")
// Define AZ::u32 in terms of our top-level aliases
.module_raw_lines("root::AZ", ["pub type u32 = crate::U32;", "pub type u64 = crate::U64;"].iter().map(|s| *s))
In short, for u32
(or another conflicting type):
- Blacklist bindgen generated
AZ::u32
- Create
type U32 = u32
alias in crate root - Output
mod AZ { pub type u32 = crate::U32 }
in place of blacklistedAZ::u32
The resulting generated bindings.rs
becomes:
/* automatically generated by rust-bindgen */
type U32 = u32; // Our top-level type aliases
type U64 = u64;
#[allow(non_snake_case, non_camel_case_types, non_upper_case_globals)]
pub mod root {
//...
#[allow(unused_imports)]
use self::super::root;
pub mod AZ {
#[allow(unused_imports)]
use self::super::super::root;
pub type u32 = crate::U32; // C typedefs defined via top-level aliases
pub type u64 = crate::U64;
//...
When primitive types are added to std
, bindgen could use those and everything would be fine (without the work-around):
pub mod AZ {
//...
pub type u32 = std::primitive::u32;
If you get “expected syntax” or “unknown type” errors, make sure you pass -x c++
to clang or name the header file *.hpp
(instead of *.h
- see this issue).
As usual, Macs may require a bit more love. Depending on the version of macOS and Xcode installed, you may need some more header files (see this forum post and the relevant release notes):
open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg
Likewise, to use rust-lldb:
ps aux | grep build
# Get PID
sudo env "PATH=$PATH" rust-lldb -p <PID>
Static Library
With our bindings generated, we can put them to use and create our static library as a package example called “staticlib”:
[[example]]
name = "staticlib"
crate-type = ["staticlib"] # Examples are executables by default
[dev-dependencies]
log = "0.4"
We’ll just create a simple function to call from C/C++:
use log::{info};
#[no_mangle]
pub extern fn example_static_lib() {
lmbr_logger::init().unwrap();
info!("RUST!!!!");
}
lmbr_logger
is an implementation of the venerable log
crate for Lumberyard:
[package]
name = "lmbr_logger"
version = "0.1.0"
edition = "2018"
[dependencies]
lmbr_sys = { version = "0.1", path = "../lmbr_sys" }
log = "0.4"
Following the example from the log
docs:
use log::{Level, LevelFilter, Metadata, Record, SetLoggerError};
use std::{ffi::CString, os::raw::c_char};
struct LmbrLogger;
static LOGGER: LmbrLogger = LmbrLogger;
pub fn init() -> Result<(), SetLoggerError> {
log::set_logger(&LOGGER)
.map(|()| log::set_max_level(LevelFilter::Info))
}
impl log::Log for LmbrLogger {
fn enabled(&self, metadata: &Metadata) -> bool {
metadata.level() <= Level::Info
}
fn log(&self, record: &Record) {
if self.enabled(record.metadata()) {
let message = format!("{}", record.args());
log("RUST", &message);
}
}
fn flush(&self) {}
}
pub fn log(window: &str, message: &str) {
let window = CString::new(window).unwrap();
let window = window.as_bytes_with_nul().as_ptr() as *const c_char;
let message = CString::new(message).unwrap();
let message = message.as_bytes_with_nul().as_ptr() as *const c_char;
unsafe {
lmbr_sys::root::AZ::Debug::Trace::Output(window, message);
}
}
Rust strings aren’t null-terminated, so we use CString
to convert them as expected by native code.
cargo build --examples
should produce our library in target/debug/examples/
. We can verify our test function will be callable from C/C++:
dumpbin /symbols C:\XXX\lmbr\target\debug\examples\staticlib.lib | Select-String -Pattern example_static_lib
# Output
008 00000000 SECT4 notype () External | example_static_lib
Per dumpbin /symbols
docs, the third column value SECTx
shows it is defined in the object file, and the fifth column value External
shows it is externally visible.
Adding a library to Waf module
We previously introduced Lumberyard’s Waf-based build system.
We might look at integrating this as a 3rd-party library when we turn this into a Gem, for now we’ll just hard-code the path. In Sandbox/Editor/wscript
:
# ...
hw = dict(
# ...
# OLD: win_lib = ['version'],
win_lib = ['version', 'staticlib', 'Ws2_32', 'userenv'],
libpath = ['c:/XXX/lmbr/target/debug/examples/'],
If you get “undefined symbol” link errors check out this issue.
Our example_static_lib()
method is then usable from C/C++:
extern "C" void example_static_lib();
//...
example_static_lib();
Top comments (0)