Superpower REST API DX with Serverless ⚡ and DevOps Best Practices on AWS (🐍 Python Version)

💃 Ops I did it again

This is going to be interesting!

In the first article of this series, we've seen how to have an effective and awesome DX using serverless and devops best practices using the stack I commonly use for my project, based on Node.js.
In the second article of this series, we've replaced Node.js with PHP (which is neither a native engine on AWS Lambda), to demonstrate that our assumptions are language and stack agnostic.

But those are stacks and languages which I know very well, and for which I have experience using Swiss Army knife tools like Jest or PHPUnit.

What about a language that I don't usually use for my projects?

🐍 Here comes Python

Python has become one of the first choices as a language in the stack of many dev: it ranked 3rd on the annual StackOverflow survery for those willing to learn to code, and 4th on the same survey for professional developers.

I have very basic experience using Python, as I usually use it for very specific tasks in which Python is the language of choice. That's why I'm betting with myself that I can achieve the same results for my serverless stack even if I don't master the language.

⚡ Serverless Framework and Python

Confirming Serverless Framework was a good choice and AWS Lambda is a very good service to go serverless, both support natively Python.
It's a matter of changing a line in my provider section:

runtime: python3.9

Also, we'll install and use serverless-python-requirements plugin to handle deployment of python requirements.

sls plugin install -n serverless-python-requirements

And activate under plugin section

- serverless-python-requirements #install python requirements

We have now Lambdas running on Python which will work in cloud when deployed packed with their requirements.

🚀 Local development

Since Serverless and AWS Lambda both support Python natively, the same applies to Serverless Offline.
We don't need to do anything unusual; we simply run our API with:

sls offline

📄 IaC, OpenAPI and doc as code

All our efforts on document with code our infrastructure is still safe: we don't have to change anything else than the handler to point our python function, as other definitions are language-agnostic.

hello:
  handler: src/function/hello/handler.hello #function handler
  package: #package patterns
    include:
      - "!**/*"
      - src/function/hello/**
  events: #events
    #keep warm event
    - schedule:
        rate: rate(5 minutes)
        enabled: ${strToBool(${self:custom.scheduleEnabled.${env:STAGE_NAME}})}
        input:
          warmer: true
    #api gateway event
    - http:
        path: /hello #api endpoint path
        method: 'GET' #api endpoint method
        cors: true
        caching: #cache
          enabled: false
        documentation:
          summary: "/hello"
          description: "Just a sample GET to say hello"
          methodResponses:
            - statusCode: 200
              responseBody:
                description: "A successful response"
              responseModels:
                application/json: "HelloResponse"
            - statusCode: 500
              responseBody:
                description: "Internal Server Error"
              responseModels:
                application/json: "ErrorResponse"
            - statusCode: 400
              responseBody:
                description: "Request error"
              responseModels:
                application/json: "BadRequestResponse"

✅ TDD with Pyunit

I don't know if Pyunit is the best framework available for testing projects in Python, but it comes shipped with the language, which is a significant advantage for me.

As we have seen before in other versions of our skeleton, we need OpenAPI validators to validate our API behavior:

• openapi-spec-validator is useful to validate our spec is valid
• openapi-schema-validator is useful to validate our response against a schema of our spec

We can then use our validators in our test to see that response is compliant with our OpenAPI spec.

import unittest
import json
from openapi_spec_validator import validate
from openapi_spec_validator.readers import read_from_filename
from openapi_schema_validator import validate as validate_schema
from src.function.hello.handler import hello

class TestHello(unittest.TestCase):

    def test_hello_response_against_spec(self):

        # Get spec from file name
        spec_dict, base_uri = read_from_filename('doc/build/openapi.yaml')

        # If no exception is raised by validate(), the whole spec is valid as OpenApi.
        validate(spec_dict)

        # Get specific schema for this response
        hello_response_schema = spec_dict['components']['schemas']['HelloResponse']

        # Call your function
        hello_response = hello({},{})

        # Get the response body
        hello_response_body = json.loads(hello_response['body'])

        # If no exception is raised by validate_schema(), the response is valid against the spec
        validate_schema(hello_response_body, hello_response_schema)

if __name__ == '__main__':
    unittest.main()

Let's run our test with:

python -m unittest discover -s tests -p '*_test.py'

And get our output

🔐📈 Security by design, monitoring and observability

As discussed in my first article and in the second one of this series, we shouldn't miss any of this topic.
Let's recap again and again what we should not forget:

• use a VPC and subnets in your templates
• use AWS WAF in front of your API
• use AWS Cognito as a user pool / identity pool to protect your API usage
• use AWS CloudWatch for dashboards and alarms (don't forget SlicWatch, this awesome serverless plug-in which automate for you those resource providing)

Also I suggest to have a look at AWS Lambda PowerTools, which simplifies adopting best practices with Lambda

🏁 Final Thoughts

Once again, we've observed that going serverless and adopting a DevOps culture isn't about the language but rather a mindset for executing tasks correctly and improving developer experience. On the contrary, it's an excellent approach for delving deeper into a language we want to explore further. By focusing solely on the code, we can delegate other non-business logic tasks to our architectural components, thus accelerating language learning.

🌐 Resources

You can find a skeleton of this architecture with Python support open sourced by Eleva here.
It has a hello function, which you can use to start developing your own serverless REST API in Python.

🏆 Credits

A heartfelt thank you to:

• Moneo team, which is using this skeleton as base for their APIs
• A. Fraccarollo and, again, A. Pagani, as the co-authors of CF files and watchful eyes on the networking and security aspect.
• C. Belloli and L. Formenti to have pushed me to going out from my nerd cave.
• L. De Filippi for enabling us to make this repo Open Source and explain how we develop Serverless APIs at Eleva.

We all believe in sharing as a tool to improve our work; therefore, every PR will be welcomed.

⏭️ Next steps

I have a pipeline set up to reproduce these concepts also with Java. If you'd like me to prioritize work on these or other languages, please comment on this article to request it.

📖 Further Readings

If you want to write serverless apps with Python without using Serverless Framework, you can ship them with Chalice.

🙋 Who am I

I'm D. De Sio and I work as a Solution Architect and Dev Tech Lead in Eleva.
I'm currently (Apr 2024) an AWS Certified Solution Architect Professional, but also a User Group Leader (in Pavia) and, last but not least, a #serverless enthusiast.
My work in this field is to advocate about serverless and help as more dev teams to adopt it, as well as customers break their monolith into API and micro-services using it.