Running Postman Collections in Azure Pipelines

Introduction

Running a set of Postman collections within a dedicated Azure Pipeline provides the ability to automate API testing of Development, Testing and even Staging environments with minimal manual intervention. Furthermore, through the use of regular runs, it encourages the adoption of the shift-left approach:

test early and often

Hence, teams can detect regressions and bugs early in the development lifecycle.

Create Postman Collection

The first step is to gather your Postman API requests under the same Collection. In this example, the collection is named Post-Office. Under the Tests tab, there is a simple assertion which verifies the expected status code returned after the API call.

Note: The endpoint *-foo.com is only an illustrative example and you should replace it with your own in your current context.

Create environments + variables

Ensure that you have created corresponding Postman Environments for the different contexts in which your code is being deployed to. Environments provide a way to organize variables which are only relevant for a specific context (API keys, external web-services, etc...).

Export the collection

You can export the relevant Postman Collection by hovering over it on the sidebar and clicking on the "..." (View more actions). Finally click on Export

A file with a default filename of *.postman_collection.json (in this case Post-Office.postman_collection.json) will be downloaded

Export Postman Environment Variables

Go to the Environments tab on the side bar, then click on the "..." and finally select export.

Create a new Pipeline

Once you have exported the collection and environment json files, ensure that you push them to the repository of your web app on Azure DevOps.

Now it is time for some plumbing! The task is to create a new pipeline that will execute your Postman collection scripts automatically.

Go to Pipelines and click on New. Select the classic editor.

Add Tasks

Task 1: Install newman package

For the first job, we will be using the newman package from npm to execute our postman collection.

Newman is a command-line Collection Runner for Postman. It enables you to run and test a Postman Collection directly from the command line.

-- https://learning.postman.com/docs/running-collections/using-newman-cli/command-line-integration-with-newman/

Task 2: Execute Postman Collection

To execute the postman collection with newman, we will need to provide the following command:

 newman run sample-collection.json -e env.postman_environment.json

The first argument of the newman run is the file path of your postman collection. The option -e take as input the file path of the Postman Environment file.

Making use of Pipeline variables

In the previous screenshot, we can see that we have hardcoded the environment filename for the option -e:

newman run sample-collection.json -e env.postman_environment.json

Instead of specifying directly the environment filename within the command, we can leverage the use of Pipeline variables.
We create a variable CUSTOM.ENVIRONMENT_FILE to store the filename.

We can then reference the variable inside the command as follows:

# Navigate to postman directory
DESTINATION='$(System.DefaultWorkingDirectory)/postman'
cd $DESTINATION

# Absolute path for environment file
ENVIRONMENT_FILE='$(System.DefaultWorkingDirectory)/postman/environments/$(CUSTOM.ENVIRONMENT_FILE)'

newman run collections/Post-Office.postman_collection.json -e $ENVIRONMENT_FILE

After the execution of the Postman Collection, the results of all tests and requests can be exported into a custom file.
This is made possible through the use of reporters. In our case, we will use a JUNIT reporter to export the test results to a custom file.

newman run collections/Post-Office.postman_collection.json  -e $ENVIRONMENT_FILE --reporters cli,junit --reporter-junit-export test-results/outputfile.xml

outputfile.xml represents the custom file where you want to collect the test results.

View more about custom reporters here: Newman Custom Reporters

Task 3: Publish Test Results

The final step is to publish the test results and view the results on the Tests tab of the pipeline.

Add the Publish Test Results task.

View the test results on the Tests tab

That's its folks! Thank you 😃

References

• https://learning.postman.com/docs/running-collections/using-newman-cli/command-line-integration-with-newman/
• https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/what-is-azure-pipelines?view=azure-devops