With the growing demand for online communication, there has been a noticeable increase in platforms to provide just that. With such service comes high expectations in quality and usability to become the best! We hear about those high expectations quite often, some people developing video calling applications that use CPaaS expect them to work and scale perfectly and don’t load test their own applications. While such an approach can cause a lot of issues in the future, it shows how important properly testing a communications platform, its video calling APIs and SDKs is. One of the best and easiest ways to notice potential issues is using a cloud testing platform to run frequent performance/load tests and gather metrics to analyze the performance, call quality and scalability.
Automated testing can reveal a variety of issues due to higher service resource demand and potential use of non-perfect network conditions. If testing a communications platform and its performance wasn’t done properly, it can result in bad user experience because of regressions, low performance, video being prioritized over audio and even unexpected behavior caused by browser differences. Tests integrated into CI/CD pipelines help our customers to always be calm and sure that their service works perfectly after every new deployment. As we see the demand for testing growing among the CPaaS and applications for video calling, we summed up some information on how Loadero helps our customers in the field to test real-time video services.
What’s important in testing a communications platform
Handling poor network connections
In real life users don’t always have a good and stable network connection when they are using your service. Some might have a high packet loss rate, others will have a slow connection or high jitter, all that can make user experience worse. And also the network conditions are not constant, they may change along the way when a user switches to another network, loses connection to his WiFi when exiting a building, has a worse connection due to entering a tunnel, etc. All this will happen every now and then. We know the importance of testing those cases and have a setting in Loadero that will allow you to check what happens when your service is used with an imperfect network connection. That’s why it is important to pick the desired network condition preset carefully when testing a communications platform – it might impact the results! You can also simulate change of the network conditions during the test run using our custom commands for that. Check out this blog post about configuring various network conditions for participants in your tests.
Quality of audio and video
Of course, regular checks of audio and video quality are necessary to keep peace of mind and know that customers get great service. To validate quality automated tests can be integrated into CI/CD pipeline and launched regularly. This can be done with small-scale tests that just measure some metrics that are the most important. But it also is a common practice to launch load tests on a regular basis to provide enough data points to make objective conclusions on system behavior. This can also be combined with competition analysis, to understand what holds your platform back from being the best in class.
To save time on analyzing such regular tests, it is a good idea to use assertions to set expectations and automatically check if they were met. We offer a wide range of WebRTC assertions to configure in your tests. You can automatically check framerate, bitrate and other metrics for incoming and outgoing data for each test participant. If you already have a WebRTC test created in Loadero, learn using asserts and explore how this can save your time and help always be aware of audio and video quality in calls on your platform.
Worldwide coverage
With many services providing video communication APIs competing, customer expectations grow constantly too. One thing that most users expect from CPaaS is low latency for all users. There are different parameters that can have an impact on latency, a very common one is location. If you can guarantee a less than 200ms latency to customers on the same continent, that’s great. But will it be the same if one user is located in Asia, another one in Europe and the third one in the US? This definitely has to be checked to confirm you are ready to serve users worldwide or to find issues to be fixed. Loadero provides access in 12 different locations across the world, allowing to create tests that will simulate real-world scenarios where users are not located in the same country or even continent. You can use the feature in your small-scale tests to compare user experience in various locations, and also you can use the same approach in your load tests to check if anything changes when the infrastructure experiences a high load. You can read more about checking performance for users in different locations in this blog post. Nowadays every service is expected to work worldwide and, of course, CPaaS are not an exception, so make sure you check the performance for users in different parts of the world.
Browser support
Not often, but in some cases, new browser versions introduce changes that can change the platform’s behavior unexpectedly. Not only that, but also bugs are fixed on each new release and new bugs are introduced as well. Having good browser coverage allows detecting differences in how your application behaves on different browser platforms. Loadero uses real browsers as test participants and we know the importance of cross-browser testing. That is why we allow configuring one of the five latest versions of Google Chrome or Mozilla Firefox for our test participants. Once a new version of the browser is released, our team works hard to get it added to our tool ASAP. A quick small-scale test that has 10 participants with 10 different browser versions can cost less than 1$ USD, but will help you make sure that your service handles the new and previous versions equally well.
Types of tests to run
Performance tests
Performance testing communications platform is where Loadero can help get insights on how well the service behaves from the user side on various configurations, including – different network conditions, locations, media and computing resources. Each of these parameters can impact the user experience in a major way, so none of those should be left untested.
WebRTC data analysis
When running tests in Loadero on applications that support WebRTC protocol, then WebRTC data will be gathered during the whole test duration and later be provided to you, these WebRTC metrics could represent how well the session went. This data is represented in graphs, data tables and also plain text data dump for each participant running the test. In the beginning without any prior knowledge WebRTC data could seem intimidating, for starters – there is a lot of it and at first sight does not give any direct information about the quality of the call. For easier use, we gather the data and provide it in aggregated statistics, these are some that might be most interesting to you: frames per second (FPS), audio/video bitrate, jitter and round trip time (RTT). In short, both – FPS and audio/video bitrate directly impact the signal quality of the call, for these metrics, generally higher is better. On the other hand jitter and round trip time might not impact the signal quality but rather overall user experience, because of stuttering, etc. For these metrics generally lower is better. So it’s important to understand how the application adapts, whether the bandwidth estimator is working correctly and quick enough. For more detailed information about all the WebRTC data fields, check out our wiki page about asserts here!
Machine statistics
In the real world each application visitor has a different computer configuration – some visitors might have an abundance of machine resources, some might be pushing the capability of their machine. When running performance tests, machine statistics are gathered by Loadero to show how much of the machine resources were used when running the test. Such data can give a good understanding of what would be the minimum requirements for stable use of the CPaaS platform, as well as quickly detect memory leaks and excessive CPU usage.
Network analysis
Digging deeper into investigating how the network behaved during the test might bring you some fresh look on how optimized is your application. That is why in the test run results we provide data about network performance. Check out the image above, you can find the same table in your test results as-well, in this table you are able to see incoming/outgoing network statistics – how many network packets were sent, got errors or were dropped and also how much network was sent in bytes. In short – when using a regular network setting there should not be any dropped packets, as well as no packet errors should appear, but obviously this can change depending on network conditions used.
There is no sure standard to determine what would be a low amount of traffic for a CPaaS, but from our experience – for start, it’s valuable to look for unexpected spikes in network packet count and bytes sent to avoid traffic congestion for the user and only then start trying to optimize the platform. The easiest way to look for such spikes is in the single participant result view in the Machine statistics tab, where you will be able to see graphs for network statistics – bytes, packets, errors, packets dropped.
Please, keep in mind that network metrics can highly change depending on what communication protocol is implemented and in some cases also the used media for the participant has a great impact in this field.
Load tests
Load testing a communications platform is very important. Imagine a scenario where a new startup launches their new live event broadcasting web application and promoters are starting to sign up to broadcast huge events, but as it turns out, a lot of these events are happening on the same weekend. How confident can the team developing this application be, that it can handle such load on the service? In this case, load testing could get extra confidence in the application and avoid potentially bad user experience for the viewers. Recently we noticed that many new live events services appear and some of them are running load tests in Loadero before an actual large event to make sure they are ready for the big day. We respect such a responsible approach and are happy to help our customers to check how their application handles hundreds and thousands of concurrent users. But if you are a provider of a communications platform service, the number of concurrent users to serve can be even greater than that. And while online event service providers usually know when the load on their service will increase, CPaaS should be always prepared for a spike in the number of users. So load testing a communications platform means checking if it is ready for success.
High application demand can cause slow service or in the worst case even downtime. For proper load tests that simulate real-life usage as closely as possible, we generate the load using the real browsers to simulate users. That is what real users do – they access a service using various browser versions which could implement WebRTC protocol differently. A load test in Loadero can have all the important parameters we mentioned earlier in it: various browser versions, different network conditions and worldwide coverage. And the best part is that if you already have a small-scale test configured for your needs, scaling the test up for large-scale load testing can be done in a matter of minutes.
Steps to start testing a communications platform
There are different ways our customers approach testing their products. Usually they prepare an application that uses the platform’s video API to be tested and create an array of tests for it. What we offer is starting with our trial and running some very basic tests free of charge. You will be able to start preparing for your larger tests and also see the full results reports to plan how you will analyze those in the future. Once that is done, you can plan test scenarios to create that will simulate real usage. This is the time to start thinking about the parameters to use in your tests and configuring them for different participants.
When you are working on your test scripts, make sure to plan how the test will work when you reuse the same script for large-scale load testing. Feel free to also contact our team if you have any questions about using Loadero for your existing or future tests, we are glad to help you use our tool effectively.
Step one – come up with a scenario
Obviously, depending on your platform specifics, the scenario can change heavily, but the main idea is clear – before any automation can be written, you need to know what should be tested. Start out by creating a scenario that includes just one or two participants. A good starting point is creating a one on one communication scenario, which will generally contain the most business-critical flow. In case group communication is available, that also is a great starting point.
Step two – write the script
This step for test automation is one of the most time-consuming and requires some knowledge about programming. Luckily we have also written a series of guides on writing Nightwatch.js test scripts, check out the first part here! At first, keep it simple and let it get advanced in time. If you can’t create a test script for your needs yourself, don’t worry. Engineers from our team can do this for you. To get the test creation cost estimates, contact us with your requirements description.
Step three – resolving issues
Let’s be honest, nobody is perfect and most likely some issues will pop up at first when developing the script. Once again, we have a separate blog post for this which covers 6 Ways of Selenium test scripts debugging, check it out – it might bring up some ideas! In short – if you have launched your script in Loadero and something is not right, here are some of the things you could do:
- Join the communication yourself
- Use session recording mode and then visually inspect the test execution
- Add screenshots to all important actions like – joining the communication room, logging in, etc;
- Carefully check Selenium log – there might be some error hiding.
Step four – adding post-run assertions
To improve the quality of testing, we suggest adding post-run assertions that will check the metrics we mentioned earlier. Obviously, you can add how many you want, but usually three to six asserts are enough at first, but when the application gets more mature, it’s not unusual to have around 20 asserts to cover most of the parameters. At start the expected values can be guessed – after some runs, you will be able to figure out the baseline. If there are known requirements for WebRTC statistics or machine resource usage statistics, then feel free to use those, so the test result immediately will be more precise. We provide almost unique 30 metric paths that you can assert against, check out our blog post to learn more about post-run assertions. Each metric path also can be used with math aggregators like – minimum, maximum, average.
Step five – scaling up
Have you done all the previous steps and everything seems to run well? If your answer is YES, then this might be a good time to start scaling up the tests. We in Loadero provide a way to scale up your existing tests in a very easy manner by using the group-participant configuration. For most scaling up is as simple as increasing the count of groups (edit the previously made one and change the count field), which in the end will increase how many participants will launch the script.
How many participants to use? That’s really up for discussion on your side, but we would suggest starting slow – don’t start with thousands of participants, most of the time it’s too much. If there is a target on how many simultaneous participants your platform would like to achieve, we suggest doing it in about 5 incremental steps, to really see how the platform is able to handle different kinds of load and how good the load is being balanced between all servers.
Properly testing a communications platform is a complex and sometimes tricky task, but if you follow the advice given in this blog post, that would be a great start. Don’t forget that you can run multiple small tests free of charge to explore Loadero’s result reports yourself and plan on your future tests.
Make sure to take a look at our wiki and visit our Blog for more articles like this. With this information you will be able to create many different insightful tests for your platform or application. Start creating your tests now by signing up for a free trial account and don’t hesitate to contact our helpful support team if you have questions.
Thanks for reading! I hope this was helpful for you!
Top comments (0)