Modelling with NDB

Python ORMs(Object Relationship Mappers) for SQL databases, such as Django ORM and SQLAlchemy, ship with properties that explicitly model database relationships between different database tables.
However, ndb - the python ORM built for Google Cloud Datastore does not provide such properties. Fortunately, using existing ndb properties, database relationships can still be modelled with ndb.
There are three types of relationships in a database, namely: one-to-one, one-to-many, and many-to-many. Let's dive into each relationship's implementation using ndb as well as their pitfalls.

Assumptions:
1) Readers are familiar with Database Relationships.
2) Readers have basic familiarity with python.
3) Readers are familiar with various column properties of ndb.

One-to-One

Famous examples of one-to-one relationships include:

• Western Countries and their presidents.
• Personal bank account and its owners.
• Monogamous marriage between two partners.

To implement a one-to-one relationship, ndb provides the KeyProperty for storing datastore keys. The KeyProperty also accepts an optional Kind argument to validate keys assigned to this property.
One-To-One relationships between two ndb Kinds can be implemented by creating a KeyProperty on one of the Kinds and setting its optional kind argument to be the other Kind.
Let's write some code to illustrate how KeyProperty can be used to implement a one-to-one relationship:

Class Person(ndb.Model):
    name = ndb.StringProperty()
    nationality = ndb.StringProperty(repeated=true)
    email = ndb.StringProperty()
    age = ndb.IntegerProperty()

Class Country(ndb.Model):
    name = ndb.StringProperty()
    location = ndb.GeoProperty()
    citizen_count = ndb.IntegerProperty()
    currency_name = ndb.StringProperty()
    president = ndb.KeyProperty(Person)

# Creating
obama = Person(name='Barack Obama', 
               nationality=['American',  'Kenya'], 
               email='darasimiajewole@gmail.com', 
               age=61)
obama_key = obama.put()

america = Country(name='America', 
                  citizen_count=1*10**6, 
                  currency_name='dollars', 
                  president=obama_key)

america_key = america.put()
america_id = america_key.id()

# Reading
america = Country.query(Country.president=obama_key).get()
obama = america.president.get()

Pitfalls

• KeyProperty does not enforce uniqueness like the one-to-one fields of traditional SQL ORMs. Unless a uniqueness check is being made before the entities of the Two Kinds are created/updated, a one-to-many relationship can be inadvertently created instead of one-to-one.

One-to-Many

Famous examples of one-to-many relationships include:

• Polygamous marriages.
• Father and his Children.
• Planets and their Moons
• Companies and their products
• Countries and their States

You have two ways to model a one-to-many relationship with ndb:

1) Using KeyProperty

This is quite similar to modeling a one-to-one relationship except the uniqueness check is omitted.
To implement this relationship between two Kinds using KeyProperty, simply create a field on one of the Kinds with the property type as the KeyProperty type and pass the other Kind class as the optional type argument. Let's dive into some code to illustrate the modeling:

Class Country(ndb.Model):
    name = ndb.StringProperty()
    location = ndb.GeoProperty()
    citizen_count = ndb.IntegerProperty()
    currency_name = ndb.StringProperty()

Class State(ndb.Model):
    name = ndb.StringProperty
    location = ndb.GeoProperty()
    population = ndb.IntegerProperty()
    country = ndb.KeyProperty(Country)

# Creating
## parent entity
nigeria = Country(name='Nigeria',       
                  citizen_count=2*10**7,   
                  currency_name='Naira')
nigeria_key = country.put()

## child entities
lagos = State(name='Lagos', 
              population=30*10**6, 
              country=nigeria_key)
lagos.put()

abj = State(name='Abuja',  
            population=10*10**6, 
            country=nigeria_key)
abj.put()

# Reading all states in a country
all_state = State.query(State.country == nigeria_key).fetch()
#[lagos, abj]


# Updating
china = Country(name='China',        
                citizen_count=2*10**9,   
                currency_name='yen')
china_key = china.put()

lagos.country = china_key
lagos.put()

Pitfalls

• Querying for the child entities is not supported within a datastore transaction because they don't have the same "datastore ancestor".

2) Modelling using Ancestor Key

Just a bit of background on datastore and ancestor keys culled from the official docs:
Datastore entities are identified by a key, an identifier unique within the application's datastore. The key can have an ancestor, which is another key. This parent can have an ancestor, and so on; at the top of this chain of parents is a key with no ancestor, called the root. Entities whose keys have the same root form an entity group or group.
Leveraging on the ancestor key relationship between root entities and entity groups, one-to-many relationships can be established between two Kinds by making the Key of the parent Kind the ancestor to the Key of the child Kind. Let's show some code to illustrate this:

Class Company(ndb.Model):
    name = ndb.StringProperty()
    staff_num = ndb.StringProperty()
    headquarter = ndb.StringProperty()

Class Product(ndb.Model):
    name = ndb.StringProperty()
    market_share = ndb.IntegerProperty()
    users_num = ndb.StringProperty()


# Creating
## parent
google = Company(name='Google', 
                 staff_num=20000, 
                 headquarter='Yabacon Valley')
## parent key
google_key = google.put()

## child key will have parent as google_key
gmail_key = ndb.Key(Product, 'Gmail Service', parent=google_key)
chrome_key = ndb.Key(Product, 'Chrome Browser', parent=google_key)
## child
gmail = Product(key=gmail_key, 
                name='Gmail', 
                market_share=80, 
                users_num=200)
gmail.put()
chrome = Product(key=chrome_key, 
                 name='Chrome Browser',
                 market_share=70,
                 users_num=34)
chrome.put()

# Reading
## reading parent entity
google_key = chrome.key.parent()

## reading child entities
all_product = Product.query(ancestor=google_key).fetch()
print(all_product) # [chrome, gmail]

To change the parent entity in a one-to-many relationship modeled using an Ancestor key, the child entity is deleted and recreated with a new key formed from the new parent key. These extra steps are needed because entity keys are immutable in the datastore after been created. Let's show some code:

# updating
## new parent entity
microsoft = Company(name='Microsoft', 
                    staff_num=5000, 
                    headquarter='Palo Alto')
microsoft_key = microsoft.put()

# child entity
chrome = chrome_key.get()
chrome_key.delete()

## creating new child entity key from new parent key
new_chrome_key = ndb.Key(Product,
                         'Chrome Explorer', 
                          parent=microsoft_key)
new_chrome = Product(key=new_chrome_key)
new_chrome.name = chrome.name
new_chrome.market_share = chrome.market_share
new_chrome.users_num = chrome.users_num
new_chrome.put()

Pitfalls:

• Datastore writes take longer, due to replication of changes across the entity groups stored on many Datastore servers. This could lead to inconsistent reads seconds after a datastore write.

• It's highly advisable not to use the keys of the child entities as parent keys of entities of another Kind. This is due to the volatility of the child entity keys during the update of its parent.

• The relationship is not explicit enough from the model definition.

• Multiple One-to-Many relationships cannot be formed by a child entity to multiple parents of a different kind. So the relationship between a User and their home country, and the same User and their Job, cannot be modeled using ndb ancestor keys.

Many-to-Many relationships

Famous examples of this relationship include:
1) Pizza and their toppings. Pizzas can have multiple toppings and a topping can be on multiple pizzas.
2) Citizens with multiple countries.

The KeyProperty type from NDB has earlier been used to implement the one-to-one relationship and also one-to-many relationship,  it should not be surprising that it can also be used to model the many-to-many relationship.
That is possible because the KeyProperty type supports the repeated argument, which means a list of Datastore keys can be attached to the KeyProperty fields.
Leveraging on that, many-to-many relationships between two NDB kinds such as Pizza and Toppings could be modeled using a repeated KeyProperty on the Pizza kind with the Kind argument as the Toppings kind. Below is some code to illustrate how the modeling works:

class Pizza(ndb.Model):
    name = ndb.StringProperty()
    steps = ndb.TextProperty(repeated=true)
    toppings = ndb.KeyProperty(Toppings, repeated=true)

Class Toppings(ndb.Model):
    name = ndb.StringProperty()
    ingredients= ndb.StringProperty(repeated=true)
    date_created = ndb.DatetimeProperty(auto_now=true)

# Creating
cheese = Toppings(name='cheese', 
                  ingredients= ['cheese', 'random seasoning'])
onions = Toppings(name='onion', 
                  ingredients= ['onions', 'chili seasonings'])
bacon = Toppings(name='bacon', 
                 ingredients= ['bacon', 'garlic seasoning'])
ndb.put_multi(cheese, onions, bacon)

macaroni_cheese = Pizza(name='Macaroni Cheese', 
                        steps=['A', 'B', 'C'], 
                        toppings=[cheese.key, bacon.key])
barbeque = Pizza(name='Barbeque', 
                 steps=['H', 'L', 'O'], 
                 toppings=[cheese.key, onion.key])
ndb.put_multi(macaroni_cheese, barbeque)

# Reading
barbeque = Pizza.get_by_id(barbeque_id)
toppings = barbeque.toppings
### [cheese.key, onion.key]

cheese_pizzas = Pizza.query(Pizza.toppings == cheese.key).fetch()
### [macaroni_cheese_key, barbeque_key]


# Updating
sweet_corn = Toppings(name='sweet corn', 
                      ingredients=['sweet corn', 'honey'])
sweet_corn.put()


macaroni_cheese.toppings.append(sweet_corn.key)
macaroni_cheese.put()

Pitfalls

• Having more than one indexed repeated properties on a Kind can lead to exploding index. Due to this behavior, it's advisable for ndb kinds not to have more than a single Many-to-Many relationships.