Let's Talk About Power Apps Test Studio

The Test Studio for Power Apps has been around now for a couple of years, I was so excited when it launched, but I realised recently, I never use it.

So I wanted to look into exactly why, and do my usual what works and where it fails, but just in case you don't know about it let me give you a brief intro.

What is Test studio

Test studio is designed to be a automated testing tool. You can create or record steps of an interaction within a Power App. You can then test to see if each step returned the expected result. It uses Power FX so is easy to use and is contained within the App so can be deployed across environments.

There are 3 sections, a suite, a case, and a step. Suites contain multiple cases that run sequentially, so in theory a suite would test one piece of value e.g. create order, or update order, or delete order. Each case contains multiple steps, with each step an interaction with the app/piece code to run. You can also run code at the beginning of the case, end of the case, and end of the suite.

The Wins

Recorder
The fact that you can simply record your actions is really nice, it means setting up tests is really easy. In theory someone non technical could create the tests and there is no 'lost in translation' between requirements and result.

Power FX
Using the same language as the app makes updating the tests easy, with the opportunity to even copy and paste code from the app. Additionally being able to use connections allows the test case to integrate nicely with other systems.

It works
Im guessing this is Selenium based (the original Power Automate Desktop was so Microsoft have history with it), and it works really well. When I have tried to map this in my own instance of Selenium the spying/identifying of the web components is hit or miss, so its impressive this is 100%ish successful.

Azure Devops Integration
Test suites/cases can be integrated directly into pipeline deployments with Azure Devops. This is exactly what pro code solutions do when they are deployed, with test cases ran to ensure expected results. Embedding it into the process protects production apps better then any human process.

The Misses

Connections
This one really annoys me, if I want to save my test results to a Dataverse table I need to add that connection to my app. So every user has to approve the connection, and yes its now a premium app. Its mad that a test has direct impact on the production package.

Publishing

Every time I want to do anything with the test studio I have to publish it. Want to test in the test studio, publish it, want to copy the test url, publish it. Even if I haven't made any changes to the test or the app (e.g. run test in studio so published it, then I click test url and I have to publish again). It can be really slow to publish so doing a few update test cycles can take forever. How crazy is my publish history now? I'm at a loss why I can't test the current published version, especially if I have multi devs working on the app simultaneously.

Its Not Perfect Enough
I know that sounds a bit vague but I have had continued issues with the test case not running, it simply opens the app and then nothing happens. I raised a support ticket with Microsoft and after multiple meetings and network recordings they eventually came back with 'its one of 4 issues':

• The test case is too long (it was only 10 actions)
• The test suite was too long
• The App screen was too complex
• It had a custom connector

None of these should be an issue, and more annoyingly it work sometimes, which makes me believe they have no idea.

There are also a few 'known issues' that Microsoft has shown for a year + now and not fixed.

The UI

Although the UI is nice for simple tests, if you want to do anything complex or cloning its terrible. Every step has to be added and edited line by line. You maybe able to copy suites/tests, but you cant copy steps. What really is needed is a code edit review or at least a way to import a test case. You can export it as yaml, so why can't I edit that yaml file and then re-import?

Automation
I know it works with Azure Devops, but I cant imagine that being used by the majority of Power App developers. It does not work with Power Platform Pipelines or any other system. The fact that most testing is developers clicking on a url and then waiting for it to complete is not automation. They really needed to build in a headless browser like https://playwright.dev/ (headless is a browser that runs in the cloud, so it does not display the web page but still interacts like it does), that way developers could simply call the url from an api. With a simple api end point it would be very easy to integrate with Power Platform Pipelines and/or any automated deployment tool.

In Conclusion

This is definitely another, if only. With Microsoft focusing on ticking the "we have automated testing" tick box, rather then thinking what developers actually need. And what do I think developers want, well:

Remove the connections from the test studio, just use a prebuilt 'Test Result' Dataverse table to store the results. That way you can remove it from the App so the connection is not shown in the published app. If you want to do other connections like SharePoint or Outlook you can trigger them from the Dataverse table with either a Flow or LowCode Plugin.

Create a headless end point, that way I can extend my Power Platform Pipeline with a flow that calls the test, ideally it would wait and respond with a pass/fail. But if that takes to long we could simply use that Dataverse Table trigger to continue/end the pipeline run

Allow a yaml editor, we are already seeing yaml for the components, makes sense that we should be able to write out our tests in yaml too (keep the recorder and UI for devs who want that, but give us code as well).

And finally invest in stability, test studio needs to just work, it needs to be consistent, work on all apps, and all of the time. Just imagine if a critical deployment was stopped because it failed the test only to find out the test failed for no reason, or worst the update made it not compatible will testing going forward.