Over the last year I have been given the chance to work with some amazing people and we all have been developing micro-services that expose RESTful APIs.
Through out all this year we had to integrate with other APIs, such as RESTful, NVP - Name-Value Pair, etc, and both internal or external 3rd parties.
Making sure the project runs without any major incident is hard work. It involves a lot of people from a lot of departments and their ability to share information is the most vital part before developing the API.
In no way am I stating that these are the right approaches to take when designing an API. I am just sharing what I have learned the hard way.
So here are some points I find important to keep in mind when given a new task, aka, build a new micro-service.
- Understand the API dependencies;
- a dependency can be an internal API, external API or an AWS service you need to integrate with, etc...
- Understand if those dependencies provide all the functionality the API requires (both tech and business);
- does it fit the purpose;
- how well it is documented/supported;
- is it too strict;
- how flexible regarding unforeseen/future changes.
- Understand what the API MUST deliver;
- current deliver;
- future deliveries (although in an agile environment this is a fuzzy topic).
- An API contract;
- a contract is an agreement between two or more parties that develop/consume the API.
- API documentation;
- API testing.
All above topics can be grouped into 3 main categories:
- API dependencies
- API specification
- API development
API dependencies
As I stated above, this is where typically you will have the business and technical requirements and you or your team start brainstorming/spiking on the best course of action.
This means, understanding what are the API dependencies and their added value.
From personal experience, you always miss some edge cases so it is a good practice to use UML sequence diagrams to help you structure how your endpoints will behave, such as request headers, payloads, responses, etc.
What is an UML sequence diagram?
An UML sequence diagram describes how operations are carried out in an interactive format. They are time based and they show the order of those interactions. They also specify all the participants in the workflow.
A visual interpretation helps you keep track of the workflow and define happy paths as well as errors from any 3rd party API and how your own API should deal with that information.
How can we as a developer take advantage of sequence diagrams? By using tools such as PlantUML or mermaidJS which allows us to generate diagrams from textual representations.
A simple example with PlantUML (taken from the official website):
@startuml
Alice -> Bob: Authentication Request
Bob --> Alice: Authentication Response
Alice -> Bob: Another authentication Request
Alice <-- Bob: Another authentication Response
@enduml
which will generate the next image:
This is a cool feature because it can be control versioned.
I find that mermaidJS is still a little behind of PlantUML in terms of integrations and functionalities but they are both powerful tools and I have used both in different contexts. You should use the one that best fits your needs.
If you use Confluence, there is a nice plugin for PlantUML.
API specification
After you have defined the diagrams, the next step is to start drafting the contract.
This contract should be "signed off" in a standard specification that most developers can be familiarised with. Luckily the OpenAPI Specification has been here for a while.
The specification is written in yaml and it can also be control versioned.
Once again and from personal experience, the drafted contract may suffer small to medium changes. Which is normal, it's a contract where more than one team is involved and feedback is always a good thing.
Always be open to suggestions but don't forget that your team owns the API.
Discussion is healthy and allows us to see different angles to achieve the same goal.
Keep in mind that your API may also suffer changes in the future which may impact production environments. So think wisely on versioning your API being that through path versioning such as /v1, etc. Or by headers such as GitHub's example Accept: application/vnd.github.v3+json.
If you are like me and your OCD kicks in when the topic "API versioning" comes to the table then read this interesting post about Evolvable APIs from Fagner Brack - To Create An Evolvable API, Stop Thinking About URLs.
API development
It's time to take all the value we gained before to start implementing it into code. Just make sure to follow the contract and protect the micro-service against unexpected 5xx popping into production.
But depending on the language you choose to code, a big part of the development is testing - unit, functional, etc...
With the right tools you can prepare functional test scenarios by using Postman or Insomnia.
Postman has a neat feature which is called Newman where you can run a collection against a file to check if your endpoints follow the contract.
At this point I had shared tools that can be version controlled along with the current code. Making it easy to keep all of them synced.
Demoing with an example
Nothing is better than an "almost real" example to demonstrate everything described above.
This example is based on making capture, void and refund transactions, given an authorization identifier.
an authorization identifier means that we locked some amount from the payment method used by a customer.
Fictional requirements
- capture the authorization;
- charge the account an amount lower or equal than the locked funds in a specified currency;
- returns a transaction identifier for possible refund.
- void the authorization;
- release the locked funds.
- refund the account;
- providing a valid transaction identifier;
- returns a refund transaction identifier.
- all above actions MUST be validated against another fictional internal API;
- validate that accountId is linked to the authorizationId.
- all above actions MUST have a required X-Api-Key header;
- for security reasons.
- all above actions SHOULD have an X-Correlation-Id header.
- for keeping track of workflows.
Let's name the micro-service as process-transactions.
Planning with sequence diagrams
From the previous fictional requirements we can define 3 participants:
- USER - The API consumers;
- MS - The API micro-service;
- API - The API consumed.
A draft of the diagram should resemble as the following image:
Tooling for PlantUML
Consider the following file structure.
./images ./plantuml ├── capture.puml
Where capture.puml has the following content.
@startuml
participant "USER" as A
participant "MS" as B
participant "API" as C
title //process-transactions// micro-service capture workflow
rnote left A
**headers**
X-Api-Key<font color="red">*</font> //<string>//
X-Correlation-Id //<string>//
end note
activate A
A -> B: **POST** ""/capture/:authorizationId""
rnote left A
**payload**
accountId:<font color="red">*</font> //<string>//
amount:<font color="red">*</font> //<number>//
currency:<font color="red">*</font> //<string>//
end note
rnote left B
**headers**
X-Api-Key<font color="red">*</font> //<string>//
X-Correlation-Id //<string>//
end note
activate B
B -> C: **POST** ""/validate""
rnote left B
**payload**
accountId:<font color="red">*</font> //<string>//
authorizationId:<font color="red">*</font> //<string>//
end note
alt success request
rnote right B
**headers**
X-Correlation-Id //<string>//
end note
activate C
B <-- C: ""**200** OK""
rnote right B
**payload**
success: //true//
end note
|||
B -> B: capture amount
activate B
deactivate B
rnote right A
**headers**
X-Correlation-Id //<string>//
end note
A <-- B: ""**200** OK""
rnote right A
**payload**
transactionId: //<string>//
end note
|||
else failure request
rnote right B
**headers**
X-Correlation-Id //<string>//
end note
B <-- C: ""**200** OK""
deactivate C
rnote right B
**payload**
success: //false//
end note
rnote right A
**headers**
X-Correlation-Id //<string>//
end note
A <-- B: ""**422** UNPROCESSABLE ENTITY""
deactivate B
rnote right A
**payload**
error: //true//
reason: Conditions could not be met
end note
|||
end
deactivate A
@enduml
We can use the package node-plantuml for generating the sequence diagram as an image.
npm install node-plantumlpuml generate -s -o ./images/capture.svg ./plantuml/capture.puml
Now we have a version controlled file that describes our /capture endpoint.
Writing the contract in the OpenAPI Specification
PlantUML gives us a pretty good view of what the capture endpoint expects as a request and responses.
Remember that at this point the micro-service logic is still a black-box, and it should remain that way for now.
We are trying to achieve a contract that does what business is expecting.
It's also expected that all the dependencies of the micro-service regarding 3rd/internal parties APIs are clear on their purposes and which of their endpoints suits our needs.
In our capture endpoint we assume some generic response. But we could be calling x number of endpoints if needed before the capture replies with anything.
Anyways, the OpenAPI is defined as a yaml file with all the specifications.
But if we have a few endpoints and a lot of responses, it might be useful to have separate files for each section of the Specification.
Ultimately this will ease the burden of maintaining the specification.
Organising the contract structure
Updating the above file structure.
./images ├── capture.svg ./plantuml ├── capture.puml ./open-api ├── components │ ├── headers │ │ └── x-correlation-id.yaml │ ├── headers.yaml │ ├── parameters │ │ ├── authorization-id.yaml │ │ └── x-correlation-id.yaml │ ├── responses │ │ ├── capture-200.yaml │ │ └── capture-422.yaml │ ├── responses.yaml │ ├── schemas │ │ ├── capture-200.yaml │ │ ├── capture-422.yaml │ │ └── capture.yaml │ └── schemas.yaml ├── components.yaml ├── index.yaml ├── info.yaml ├── paths │ └── capture.yaml └── paths.yaml
Instead of using the normal
'#/components/...', ref is a relative link to the file, which after the compile step will be properly OAS "reffed".
Content of ./open-api/index.yaml:
openapi: 3.0.2
tags:
- name: capture
info:
$ref: './info.yaml'
paths:
$ref: './paths.yaml'
components:
$ref: './components.yaml'
security:
- X-Api-Key: []
Content of ./open-api/paths.yaml:
/capture/{authorizationId}:
post:
$ref: './paths/capture.yaml'
Content of ./open-api/paths/capture.yaml:
summary: Capture an amount
tags:
- capture
operationId: capturePost
parameters:
- $ref: '../components/parameters/authorization-id.yaml'
- $ref: '../components/parameters/x-correlation-id.yaml'
requestBody:
content:
application/json:
schema:
$ref: '../components/schemas/capture.yaml'
responses:
'200':
$ref: '../components/responses/capture-200.yaml'
'422':
$ref: '../components/responses/capture-422.yaml'
Tooling for OAS
We can use the package swagger-cli for generating the compiled Specification file.
npm install swagger-cli-
swagger-cli bundle -o open-api.yaml --type yaml open-api/index.yaml.
And the full specification:
openapi: 3.0.2
tags:
- name: capture
info:
version: 1.0.0
title: Process Transactions Micro-service
description: 'Capture, void and refund an account.'
paths:
'/capture/{authorizationId}':
post:
summary: Capture an amount
tags:
- capture
operationId: capturePost
parameters:
- name: authorizationId
description: Authorization Id which allows to capture the locked funds
in: path
required: true
schema:
type: string
- name: X-Correlation-Id
description: Correlation Id to keep track of workflow
in: header
schema:
type: string
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/capture'
responses:
'200':
$ref: '#/components/responses/capture-200'
'422':
$ref: '#/components/responses/capture-422'
components:
securitySchemes:
X-Api-Key:
type: apiKey
in: header
name: X-Api-Key
responses:
capture-200:
description: Capture of funds has succedeed
headers:
X-Correlation-Id:
$ref: '#/components/headers/X-Correlation-Id'
content:
application/json:
schema:
$ref: '#/components/schemas/capture-200'
capture-422:
description: Capture did not succeed
headers:
X-Correlation-Id:
$ref: '#/components/headers/X-Correlation-Id'
content:
application/json:
schema:
$ref: '#/components/schemas/capture-422'
schemas:
capture:
type: object
required:
- accountId
- amount
- currency
properties:
accountId:
type: string
amount:
type: number
example: 9.99
currency:
type: string
example: EUR
capture-200:
type: object
properties:
transactionId:
type: string
description: The transaction id which will allow to refund
capture-422:
type: object
properties:
success:
type: boolean
default: false
reason:
type: string
example: Conditions could not be met
headers:
X-Correlation-Id:
schema:
type: string
security:
- X-Api-Key: []
Testing against the API
Now that we defined diagrams and the contract is settled, we are ready to implement it.
Let's say you have a server up and running with all of the requirements in place.
Wouldn't it be better to have a Postman collection based on the OAS instead of manually creating it?
Converting OpenAPI to Postman Collection
We can use the package openapi-to-postmanv2 for generating the Postman collection.
npm install openapi-to-postmanv2-
openapi2postmanv2 -s open-api.yaml -o postman-collection.json -pit will generate the a Postman collection almost pre-filled.
Obvious you will need to fill the blanks such as the X-Api-Key and alike.
Conclusions
Building the Postman collection through the OpenAPI Specification will help find breaches in the development.
Just keep in mind that changes to the contract often happen during development or even when all the functionality is being tested.
Hope this workflow helps in any way possible to fasten and tighten the development.
What are your thoughts? Please share your experience and all constructive feedback is welcome!
Links
Specifications
Tools
Top comments (0)