In the perfect world you'd have multiple development environments, Dev/QA/Stage/Prod, especially if you have multiple microservices being developed concurrently. But in reality you might only have one Dev environment. So when do you run all of your automation integration tests? You might run them nightly on Dev or only on Stage after a release point. There are pros and cons to all these methods. Missing API regressions could be a very big con.
Another option is to trigger your service-specific API tests in an integration pipeline right after the corresponding Azure Devops Dev pipeline for the actual service runs. This is pretty easy to do in Azure DevOps YML. In my test integration pipeline under "resources:" I added a "pipelines:" section.
The options "pipeline", "source", "project" and "trigger" are simple and explained in this Azure DevOps article. Basically your integration test pipeline is triggered right after the Dev pipeline. In this example the Dev pipeline is named "pin-app-payor-job-service". Below is the trigger result in ADO Pipelines
My team writes failed test runs to a Teams channel so everyone can see them. I have DEV another article on how to do that here.
Make sure your automation tests are scoped to the service under test. Running "Every test under the sun" is annoying and generates too much noise. We're currently using Specflow with XUnit so we tag our Scenarios according to the API that it tests.
Then when we run tests in our pipeline yml with the "dotnet test" command we pass in the specific tag as a --filter Category for XUnit.
Hope this article helps you run your automation at the right time!
Top comments (0)