Learning from over-engineering

We can gain incredible insights into from our success as well as our failures in software engineering. Yet it is often overlooked how sharing our failures can help us understand why our specific solution failed and help others avoid the same pitfalls.

I’d like to take a moment to explore a time where I over-engineered myself in to a solution that, while novel, ended up being far too difficult to maintain in comparison to the value the code itself delivered.

The Challenge

I was tasked with creating a python package that would serve as the middleware between a client application and an existing REST API. The package was expected to, at the bare minimum, be able to do the following:

• Make async REST requests against a given API endpoint
• Unmarshal the responses from the API and be able to process that the response
• Return the response according to the expectations of the client application
• Handle any errors that may occur

Proposed Solution

I knew from the very beginning that my solution needed to be as DRY as possible. In this case I knew that the endpoints of the API I was working with were all constructed very similar to one another; for example:

.../resource/
.../users/
.../blogs/
...

The same can be said for the responses, again, for example:

{
    "uuid": "94cccdc9-3564-434f-9b07-a02dc1edb353",
    "name": "object_name",
    "tags": ["python", "code_example"],
    "meta": {"state": "IA", "details": ""},
}

I also wanted my solution to produce responses that could be introspected. This meant that would need to create a unique Class for reach response that I could receive so that consumers would be able to know or find out members of those classes.

As you might imagine that could be difficult to do with N number of possible classes and only one function making the requests. Instead of opting for conditionals or flags I chose to create a generic Python Decorator that would decorate member methods of a given Class and make the requests on that class’s behalf.

The Classes ended up taking the following structure.

class ExampleResponse(ProcessResponse):
    def __init__(
        self, 
        id: str,
        some_field: str,
    ) -> None:
        self.id = id
        self.some_field = some_field

class Example:
    def __init__(self, session) -> None:
        # async request setup stuff

    @process(ExampleResponse)
    async def get(self, id: str):
        async with self.session.get(...) as resp:
            return await resp.text()

Each class would have to have its own response class it could unmarshal into and each member method was decorated with @process() that takes the destination object as its only parameter. Each endpoint needed its own class and member methods all doing very similar things.

That’s not very DRY.

The DRYness of my code was achieved in the decorator itself. All the unmarshalling and processing happened here.

class ProcessResponse(object):
    def __init__(self, *args, **kwargs):
        ...

def parse_error_message(error):
    ... # Read the error string in and turn it into a python error obj

def process(return_type: Type[ProcessResponse]):
    def inner(func):
        async def wrapper(*args, **kwargs):
            res = await func(*args, **kwargs)
            try:
                parsed = json.loads(res)
                if isinstance(parsed, list):
                    return None, [return_type(**par) for par in parsed]
                return None, return_type(**parsed)
            except (JSONDecodeError, TypeError):
                return parse_error_message(res), None
        return wrapper
    return inner

The decorator module defined a parent class that all responses had to inherit from and returned either an object of type return_type or a Python error.

Challenges Faced

This approach had two very clear weaknesses that became challenges in their own right:

1. It was overly complicated
2. It was far too rigid to be maintainable

First, the excessive complexity. It was hard to trace errors though this code when something broke. In just these few lines of code we:

• Decode a JSON string
• Attempt to unpack that decoded string (twice!)
• Assume that if the decoded string could not be unpacked that meant it was an error and that we should try to force it into an error of some form

Not to mention that decorators are not the most inherently easy to understand structures by themselves. So I mixed in a little async for fun. While it made sense to me, I found it difficult to explain to others if they didn’t already have a strong understanding of decorators and other Python behavioral traits.

Secondly, what happens when the response from the API changes? For example, our ExampleResponse class expects a JSON response that looks like

{
    "id": "123",
    "some_field": "some cool string",
}

If we added, renamed, or otherwise modified this response without also modifying our ExampleResponse class we would face JSON parsing errors as we attempt to unpack the decoded response into our classes.

This was a large challenge for this library as it was being developed against an API that would change with some high level of frequency.

Potential Improvements

The initial design, while well-intentioned, fell short of its potential due to its complexity and rigidity. To prevent falling into similar traps in future projects, the following considerations are essential:

1. Decorators:
   • Carefully assess the necessity of using decorators. While they can be powerful, their excessive use can lead to convoluted code.
2. Generalization vs. Clarity:
   • Strive for generalized logic, but not at the expense of code clarity. Code should be easy to understand and troubleshoot. Keep complexity in check.
3. Leverage Python's Standard Library:
   • Explore the capabilities of Python's Standard Library thoroughly before diving into complex solutions. Often, you'll find that Python provides built-in tools to simplify tasks.
     • For example, Python offers SimpleNamespace, a versatile data structure that, when combined with json.loads, allows you to load JSON into an object that supports dot notation, making it easy to access fields like example.some_field.

Conclusion

While this solution initially served its purpose, it soon became apparent that it wasn't a sustainable, long-term answer. In retrospect, a more effective strategy would have involved thorough pre-planning and research, leading to the discovery of potentially simpler and more maintainable solutions. This approach would have spared me the need for frequent rewrites and the complexities of debugging sessions.