Python Recursion Errors & Serializer

I recently completed my first Flask-React application, a Spotify clone app called 'myTunes' that allows users to create unique playlists, add songs from my PostgreSQL database, then play those songs through linking to the Spotify web-browser. I felt like I had finally gotten the hang of one-to-many and many-to-many database model relationships - my database was seeded and rendering on the React front-end, but I noticed my initial fetch was taking longer and longer to complete. I figured it was a byproduct of pulling more data in from my backend routes and that I would continue to build my app and then research ways to make the fetch more efficient. I added one more route - a post route to add a Song instance to a Playlist - and suddenly my environment crashed and I saw a giant 'RECURSION ERROR' in my terminal.

Defining Recursion

Welp. This was the first time I'd ever experienced a recursion error, and it looked terrifying. I tried to scroll to the top of the error, but it seemed unending. Turns out, that's exactly what a recursion is - a function that calls itself from within the function. The important part of writing a recursion correctly is that the function must terminate at some point, otherwise it cannot run because it uses too much memory, processing power, or simply is just infinite.

Writing recursions can be very helpful - for example, you may want to write a function that counts backward from some starting point and stops at 0. As long as the function can reach a termination point, recursive functions are perfectly acceptable to write and can be extremely efficient.

Recursion in Model Relationships

Rendering relational data is a prime environment for experiencing recursion errors. For example, in the myTunes application I built, I designed a many-to-many relationship between my Playlist and Song models. I have simplified the models here to only include relevant information to solving the recursion error.

Playlist Model:

class Playlist(db.Model):
    __tablename__ = "playlists"

    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String, nullable=False)

    songs = db.relationship('Song', secondary=playlist_song, back_populates='playlists')

    user_id = db.Column(db.Integer, db.ForeignKey('users.id'))

    def __repr__(self):
        return f'<Playlist ID: {self.id} | Name: {self.name}>'

Song Model:

class Song(db.Model):
    __tablename__ = "songs"

    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String, nullable=False)
    artist_name = db.Column(db.String, nullable=False)

    playlists = db.relationship('Playlist', secondary=playlist_song, back_populates='songs')

    def __repr__(self):
        return f'<Song ID: {self.id} | Name: {self.name} | Artist: {self.artist_name}>'

Join Table:

playlist_song = db.Table('playlist_songs',
                          db.Column('playlist_id', db.Integer, db.ForeignKey('playlists.id'), primary_key=True),
                          db.Column('song_id', db.Integer, db.ForeignKey('songs.id'), primary_key=True))

The Playlist and Song models are related through a join table, Playlist_Song. I pulled all of this data into my application through the /artists endpoint, invoking the Python method .to_dict() on each instance on the Flask side of app, then invoking .json() when rendering the data in a fetch request through my React front-end. This process of converting data objects into other forms to be transmitted is called serialization.

As I continued to build out Flask endpoints, I noticed my fetch request on the React end was taking longer and longer to complete. Eventually, the dreaded recursion error hit and I frantically Googled and asked ChatGPT for help before taking a breath and remembering I could test instances of my models in the Flask shell.

flask shell
hello = Song(name="Hello", artist_name="Adele")
jams = Playlist(name="Jams")

After creating instances of a Song and Playlist, I tested the relationship and the .to_dict() method to ensure the .to_dict() method could successfully be invoked and return a dictionary object.

hello.to_dict()
{...} / Successful response /
jams.to_dict() 
{...} / Successful response /
hello.playlists
[] / Successful response as no playlists were assigned /
jams.songs
[] / Successful response as no songs were assigned

Single instances with no relationships were able to successfully convert to dictionary objects through the .to_dict() method. I next moved to testing multiple instances with relationships established on my seeded database.

flask shell
playlists = Playlist.query.all()
songs = Song.query.all()
playlists[0].songs / Grabbed first playlist and listed all songs
[...] / Successful response with the songs I had related to the playlist
playlists[0].songs.to_dict()
RECURSION ERROR

AHA! I scrolled back in my terminal to check the output from playlists[0].songs again and saw that in addition to outputting the 5 songs I had assigned to the playlist, each of those songs had a nested 'playlists' key that included the top level playlist, and that each of those nested 'playlists' keys had the songs nested within them again... When invoking .to_dict() on this structure, it produced a recursion error because it caused a NEVER ENDING nesting of data.

I also tested this data the other way around - as songs.playlists instead of playlists.songs, and found another recursion error. I needed to find a way to exclude my database relating the data from each model endlessly to one another. Enter - SQLalchemy serializer mixin.

Serializer Mixin

Serializer Mixin is a mixin that adds the .to_dict() method to model instances. This eliminates the need to define an explicit .to_dict() class method. It also allows the user to set rules for serialization so that certain relationships and columns can be EXCLUDED from the .to_dict() method, ideally avoiding the recursion error cause by Python trying to nest this relational data endlessly when serializing the data objects to another type.

Serializer Mixin can be used by importing it from sql-alchemy. Here are the updated Playlist and Song models utilizing Serializer Mixin.

from sqlalchemy_serializer import SerializerMixin

class Playlist(db.Model, SerializerMixin):
    __tablename__ = "playlists"

    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String, nullable=False)

    songs = db.relationship('Song', secondary=playlist_song, back_populates='playlists')

    user_id = db.Column(db.Integer, db.ForeignKey('users.id'))

    def __repr__(self):
        return f'<Playlist ID: {self.id} | Name: {self.name}>'

class Song(db.Model, SerializerMixin):
    __tablename__ = "songs"

    serialize_rules = ('-playlists', )

    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String, nullable=False)
    artist_name = db.Column(db.String, nullable=False)

    playlists = db.relationship('Playlist', secondary=playlist_song, back_populates='songs')

    def __repr__(self):
        return f'<Song ID: {self.id} | Name: {self.name} | Artist: {self.artist_name}>'

playlist_song = db.Table('playlist_songs',
                          db.Column('playlist_id', db.Integer, db.ForeignKey('playlists.id'), primary_key=True),
                          db.Column('song_id', db.Integer, db.ForeignKey('songs.id'), primary_key=True))

In the Song model, I specified rules to exclude '-playlists' from serialization. After upgrading my Flask database and reseeding, I opened the Flask shell and tested the same instances as before. This time, songs[0].playlists.to_dict() and playlists[0].songs.to_dict() successfully returned dictionary objects with no recursion errors.

Testing for Recursive Functions

After finding the first recursion error in my code through testing instances of my seeded database in the Flask shell, I continued to test out each of the relationships I had built between my additional models: User, Artist, Song, and Playlist. In each case, when invoking .to_dict() on a model related to another, I ran into a recursion error. I added serialize rules to exclude each of the areas resulting in an error. Check out my GitHub repo for myTunes to see the full models.py code to see all instances of serialize rules I needed to add.

Conclusion

After adding the necessary serialize rules to each model, I started up my application using honcho (if you don't know about honcho, check it out to run both front and back-end development environments with one command!) and not only did I NOT run into a recursion error, but my fetch data loaded quickly. I realized in hindsight that the other recursion errors I had built in through my models had not quite been problematic enough to bypass Python's recursion limit, but by eliminating the accidental-recursive functions my database instances were able to be serialized much more quickly and without unwanted nested data.

While following through recursion errors can be daunting, I hope this guide helps take some of the mystery out of this error. Since solving my own recursion errors with Serializer Mixin, I have also found the Marshmallow (de)Serialization library from SQL-Alchemy. I have yet to use this library, but it looks like another solution to serialization errors.

myTunes Info

If you're interested in checking out myTunes, here is my full GitHub repository! For the TLDR version of myTunes, check out my walkthrough video.