A web app built with Rust and Python

When I started learning Rust I took a few of my Python projects and rewrote the code with this new language I was checking but then I found out that there wasn't a real alternative for some of the libraries I was using.

After searching on crates.io, two crates got my attention, CPython and PyO3, you can use both for writing Python native modules and running and interacting with Python.

While you can find some examples in their corresponding GitHub repository on how to use Python within your Rust projects, I didn't find so much information on how to call a Python function (in an external library, on a .py file) from Rust, that returns a value or a set of values to be processed.

An answer on a StackOverflow's post helped me preparing a practical example I presented at a local Google I/O Extended event, a Rust web app that connects to Firebase through Python, retrieve data and print it on a HTML document.

By the time I proposed the talk "Web Apps with Rust & Firebase" for Google I/O Extended last year, I didn't know that there isn't a crate to connect to Firebase that actually works. So I started to think about the possibility to build the web app with Rust and only use Python for connecting to Firebase.

Let me explain how I built the project.

Configuration

First of all I created a new Rust project using Cargo:

cargo new rust-python-demo

The above command will create a new directory with the name rust-python-demo, inside this directory you will find a src dir that contains a main.rs file that will later be modified for writing the code of the app, and the file Cargo.toml that is the manifest of the project.

Don't forget to change to the new directory:

cd rust-python-demo

Cargo.toml

Cargo.toml must be modified to look as follows:

[package]
name = "rust-python-demo"
version = "0.1.0"
authors = ["mattdark"]
edition = "2018"

[dependencies]
serde = "1.0.99"
serde_derive = "1.0.99"
serde_json = "1.0.40"
rocket = "0.4.2"

[dependencies.cpython]
version = "0.4"
features = ["python-3-7"]

[dependencies.rocket_contrib]
version = "0.4"
features = ["handlebars_templates"]

[[bin]]
name = "rust-python-demo"
path = "src/main.rs"

The section [package] contains name and version of the app, information of the developer and edition of Rust being used, 2015 or 2018.

[dependencies] section has the list of dependencies required for the project, I'm using Rocket, a web framework for Rust, and Serde, a framework for serializing and deserializing Rust data structures. For CPython you can specify the version of Python you'll be using, and as Rocket has support for both Handlebars and Tera (template engines), you must indicate the one being used for the project, in this case Handlebars.

To assign a specific name to the binary of the app, you can do it in the [[bin]] section, indicating the name of the source code file.

pyproject.toml

pyproject.toml is the file that Poetry uses to manage a Python project and its dependencies. It contains information about the project like name, version, description and authors, in the [tool.poetry] section. The version of Python and list of dependencies must be specified in the [tool.poetry.dependencies] section, for this project I'm using any Python version greater than or equal to 3.7.3 and less than 3.8. To connect to Firebase I'm using the firebase library and the rest of packages are dependencies of it.

The file should look as follows:

[tool.poetry]
name = "rust-python-demo"
version = "0.1.0"
description = ""
authors = ["Mario Garcia <iscmariog@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.7.3"
firebase = "*"
python-jwt = "*"
gcloud = "*"
sseclient = "*"
pycrypto = "*"
requests-toolbelt = "*"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

Python

Why using a specific Python version other than the one available on your system or the one that you installed from the repositories? Some Linux distributions, specially those rolling release ones, will offer the latest stable version of Python available, 3.8.2 at the moment of writing this article.

Other distros will have the latest release of Python 3.6 or Python 3.7 but you will probably need a different version depending on the compatibility and support of the libraries you're using for the project.

So this is the moment when pyenv can help, as it makes possible to have other versions of Python installed on your system in a friendly way.

To list the versions of Python available through pyenv, run the following command:

pyenv install --list

For this project I'm using Python 3.7.7. Before installing the version of Python you'll be using for the project bear in mind that if you want to embed Python in Rust, shared libraries must be built for Python. Run the following command to install Python properly:

env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.7.7

After running the above command add the shared libraries directory to the LD_LIBRARY_PATH environment variable as follows:

export LD_LIBRARY_PATH=~/.pyenv/versions/3.7.7/lib/

Poetry

Before installing the dependencies of the project you must specify the version of Python that will be used for Poetry to create the virtual environment of the project, run the following command:

poetry env use 3.7.7

Then install the dependencies of the project by running:

poetry install

This command will also create a virtual environment.

Note: If you don't require a virtual environment for your project, you can run the following command before poetry install:

poetry config virtualenvs.create false

Rust

CPython and PyO3 works with the Nightly version of Rust, so before writing the code of your app change the toolchain of the project by running:

rustup override set nightly

Understanding the code

Directory structure

After configuring the app you will have Cargo.toml, pyproject.toml and the main.rs file in the src directory that it will be modified later to put the code of the app.

In the src directory a python dir must be created to put the Python module that will make the connection to Firebase and retrieve the data from it, named pyrebase.py.

In the root directory the templates and static dirs should be added. The Handlebars templates and the static files (e.g. CSS, JavaScript, Pictures) will be stored in those dirs.

Firebase

Go to console.firebase.google.com and add a new project, I will name it Rust Python Demo.

Then add Firebase to a web app, you will get the following configuration details that you will need to configure the Python module:

var firebaseConfig = {
    apiKey: "APIKEY",
    authDomain: "rust-python-demo.firebaseapp.com",
    databaseURL: "https://rust-python-demo.firebaseio.com",
    projectId: "rust-python-demo",
    storageBucket: "rust-python-demo.appspot.com",
    messagingSenderId: "SENDERID",
    appId: "APPID"
  };

And finally go to Database from the left sidebar and Create database. For this demo I will store a list of random names generated through listofrandomnames.com.

The structure of the database will look as follows, you can import the following JSON to the database for testing:

firebase.json

{
  "speakers" : {
    "speaker1" : {
      "id" : "1",
      "last_name" : "Moore",
      "name" : "Madie"
    },
    "speaker2" : {
      "id" : "2",
      "last_name" : "Nathanson",
      "name" : "Norbert"
    },
    "speaker3" : {
      "id" : "3",
      "last_name" : "Mcconelle",
      "name" : "Major"
    },
    "speaker4" : {
      "id" : "4",
      "last_name" : "Schalk",
      "name" : "Sophia"
    },
    "speaker5" : {
      "id" : "5",
      "last_name" : "Bertrand",
      "name" : "Benny"
    }
  }
}

The name of the database is speakers as this demo is part of a web app that I'm working on.

Don't forget to configure the rules of your database as follows:

{
  "rules": {
    ".read": true,
    ".write": false
  }
}

Python

src/python/pyrebase.py

import json
from firebase import Firebase
def read_data(self):
    config = {
        "apiKey": "APIKEY",
        "authDomain": "rust-python-demo.firebaseapp.com",
        "databaseURL": "https://rust-python-demo.firebaseio.com",
        "projectId": "rust-python-demo",
        "storageBucket": "rust-python-demo.appspot.com",
        "messagingSenderId": "MESSAGINGSENDERID"
    }
    firebase = Firebase(config)

    speaker = list()
    db = firebase.database()
    all_speakers = db.child("speakers").get()
    for x in all_speakers.each():
        speaker.append(x.val())
    s = json.dumps(speaker)
    return s

For this module I'm using the json and firebase libraries. It only has a function named read_data() that will connect to Firebase and retrieve the data from the database.

Inside the function I defined the config variable that has the configuration details required to connect to Firebase.

After connecting to Firebase I get all the registries stored in the database and append it to the speaker list and convert that list to a JSON structure for Rust to import this data properly.

Rust

src/main.rs

#![feature(proc_macro_hygiene, decl_macro)]

#[macro_use] extern crate rocket;

extern crate rocket_contrib;
extern crate cpython;

use cpython::{Python, PyResult, PyModule};

#[macro_use]
extern crate serde_derive;
extern crate serde_json;

use std::collections::HashMap;
use std::path::{Path, PathBuf};
use crate::handlebars::{to_json};

use rocket::response::NamedFile;
use rocket_contrib::templates::{Template, handlebars};

#[derive(Serialize, Deserialize, Debug)]
pub struct Speakers {
    pub id: String,
    pub name: String,
    pub last_name: String,
}

const FIRE_PY: &'static str = include_str!("./python/pyrebase.py");

#[get("/")]
fn index() -> Template {
    let gil = Python::acquire_gil();
    let py = gil.python();
    let s = run_python(py).unwrap();
    let mut data = HashMap::new();
    data.insert("speakers".to_string(), to_json(&s));
    Template::render("index", &data)
}

fn run_python(py: Python<'_>) -> PyResult<Vec<Speakers>> {
    let m = module_from_str(py, "pyrebase", FIRE_PY)?;
    let out: String = m.call(py, "read_data", (2,), None)?.extract(py)?;
    let speakers: Vec<Speakers> = serde_json::from_str(&out).unwrap();
    Ok(speakers)
}

fn module_from_str(py: Python<'_>, name: &str, source: &str) -> PyResult<PyModule> {
    let m = PyModule::new(py, name)?;
    m.add(py, "__builtins__", py.import("builtins")?)?;

    let m_locals = m.get(py, "__dict__")?.extract(py)?;
    py.run(source, Some(&m_locals), None)?;
    Ok(m)
}

#[get("/<file..>", rank=3)]
fn files(file: PathBuf) -> Option<NamedFile> {
    NamedFile::open(Path::new("static/").join(file)).ok()
}

fn rocket() -> rocket::Rocket {
    rocket::ignite().mount("/", routes![index, files])
    .attach(Template::fairing())
}

fn main() {
    rocket().launch();
}

At first, in the main() function the Rocket app is run through the rocket() function where the routes of the app are mounted and the template Fairing is attached to the app.

As it only loads a HTML document, it has two routes, index and files, the first one points to the index() function where the root of the app is loaded and the last one pointing to the files() function where the static files in the static directory is loaded.

In the index() function, the Python interpreter is embedded and the reference to the Python binary is assigned to the py variable, in the first two lines.

Then the function run_python() is called. In the first line the module_from_str() function is called, where the pyrebase module is loaded.

After that the read_data() Python method is called and the value returned by it is assign to the out variable. Then the out variable is deserialize and the values are converted into a vector using the struct Speakers.

The vector speakers is returned to the index() function where it was called and converted into a HashMap to pass the values to the index template.

Template

templates/index.html.hbs

<body>
    <div class="limiter">
        <div class="container-table100">
            <div class="wrap-table100">
                <div class="table100">
                    <table>
                        <thead>
                            <tr class="table100-head">
                                <th class="column1">ID</th>
                                <th class="column2">Name</th>
                                <th class="column3">Last Name</th>
                            </tr>
                        </thead>
                        <tbody>
                            {{ #each speakers }}
                                <tr>
                                    <td class="column1">{{ id }}</td>
                                    <td class="column2">{{ name }}</td>
                                    <td class="column3">{{ last_name }}</td>
                                </tr>
                            {{ /each }}
                        </tbody>
                    </table>
                </div>
            </div>
        </div>
    </div>
...
</body>

The index HTML document will only display the data retrieved from the database in a table.

Running the app

The code of the app is written so it's time to run the demo built.

The project must be built first, run the following command from the terminal:

poetry run cargo build --release

As the app is being run through a virtual env created by poetry, the poetry run command must be executed first and then the project is built for production with cargo build --release.

Cargo created a binary name rust-python-demo in the target/release directory. To run the app write the following command in the terminal:

poetry run ./target/release/rust-python-demo

Now go to localhost:8000 in your browser. You will see the following screen.

The demo of the web app built with Rust and Python is up and running, to get the code go to gitlab.com/mattdark/rust-python-demo.