Gone are the days of being gathered around the family TV in the living room to watch the big broadcast event of the week. Today, consumers are accessing their content through smartphones, tablets, and computers, at home and on the go, with or without a WiFi connection. Times they are a-changing, and because our goal is to keep our customers’ customers happy, it’s important to us that our testing can cope with all of these scenarios – and more.

As with any stable software product, unit tests are the solid foundation of our codebase. However,  we are always confronted with a reality that requires us to test entire systems together, which is out of the scope of unitary tests. Here is where the power of automated integrative, or end-to-end (E2E), testing comes in.

Goals of Automated E2E Testing

Unit tests function as a repository containing all software component requirements in isolation. They are an essential tool for ensuring that new features do not lead to regressions or conflict with existing requirements and, as we like to say, exist primarily for developers’ peace of mind. Because they are repeatedly executed during the development stage, they need to run really fast.

On the other hand, E2E testing has some unique characteristics: namely, it’s slow, resource-greedy, requires strong connectivity and is typically only performed on candidate releases. These constraints typically require custom-built machines – but this investment can have a high ROI for products that must work in a wide range of scenarios.

Here at Teltoo, we prioritize conducting E2E tests on these key elements of OTT video delivery:

• Build Verification Testing (Smoke Tests): This process tests the most significant, business-critical system requirements to ensure proper video playback, including ABR, captioning, multi-channel audio, streaming technologies, browsers, operating systems, video players, and devices. This sums up to a tremendous combination of possible tests. We run over 24 hours of video playback in as little as 6 minutes – so proper parallelization is a must for us.
• Stability Tests: These are similar to Smoke Tests but focus on ensuring performance over long periods of usage. This is key to OTT success, given that in a single use case, a single consumer can spend several hours using the system, exchanging and processing gigabytes of video. Crashes midway through playback are unacceptable.
• Stress Tests: This is a non-functional type of testing that ensures that new releases match the expected benchmarks. Again, in OTT video, it is key that the system can scale and serve a massive amount of audience during traffic spikes.

This is a heavy testing load for a small team, so we’ve prioritized building out an automated E2E testing system using Selenium, Webdriverio, and Appium – we’ve found the automation to be meaningful at the point when the product needs to work with a variety of platforms and/or browsers that make regular testing unreasonably challenging. Here’s how we built ours.

Introducing Selenium

As a tool that enables ‘browser automation’, Selenium allows us to control browser input remotely in order to test all the parts of our system (SDK, player plugins, etc.) by simulating the entire user journey. Here is an example of a standard Selenium testing system:

Selenium needs to communicate with the browsers (Chrome, Firefox, Safari, etc.) via a driver proxy. This setup can be installed automatically with pre-packaged tools like this. Selenium’s grid setup system allows the user to have multiple machines running as “worker nodes” – a useful feature which allows for multiple simultaneous test launches under a variety of platforms.

For more information on setting a Selenium grid environment, click here.

Setting Up a Multi-Remote Environment with Webdriverio

To set up a working environment, we use Webdriverio, which is a framework that allows us to define and execute tests locally while communicating with a remote Selenium server. Common frameworks and reporters such as mocha or jasmine can be used, and the syntax is very similar to when writing unit tests, but with one primary difference: we are controlling the browsers remotely.

Webdriverio handles this by giving us a “browser” object which is used to query page objects and launch remote scripts programmatically. One of the key parts of our testing development is to listen to player events, and while this is theoretically not possible because the player is in a different machine, we can use the Webdriverio command ‘executeAsync’  to wait for a browser event as follows:

   var results = browser.executeAsync(function(done) {
       jwplayer().on('pause', function() {
         done();
       });
   });

We use what Webdriverio calls ‘multiremote mode’ to launch simultaneous browsers in order to test webRTC browser communications and audio/video segment exchange. This framework, which is defined in the wdio.conf.js file by modifying the “capabilities” object, creates a different session for each browser demanded and then closes all sessions when the testing is done. You can also have a separate configurable file that contains an array with the browsers to be tested and gets loaded when the test runner launches. This would look something like this:

var browsers = [
 {
   browser: "firefox",
   id: "browser0"
 },
 {
   browser: "chrome",
   id: "browser1"
 }
];

const teltooConfig = require('./test/config/teltoo.config.js');

const browsers = teltooConfig.browsers;

/* add the defined browsers to the webdriverio configuration */
function addBrowsersToCapabilities() {
 for (var i=0; i<browsers.length; i++) {
   config.capabilities[browsers[i].id] = { capabilities: { browserName: browsers[i].browser }};
   if(browsers[i].browser === 'chrome') {
     config.capabilities[browsers[i].id].capabilities["goog:chromeOptions"] = browsers[i]["goog:chromeOptions"];
   } else if (browsers[i].browser === 'firefox') {
     config.capabilities[browsers[i].id].capabilities["moz:firefoxOptions"] = browsers[i]["moz:firefoxOptions"];
   }
 }
};

addBrowsersToCapabilities();

Android, iOS, and Beyond with Appium

Testing with browsers on a desktop is an obvious need. But we’re also seeing huge increases in video consumption across mobile and other platforms (Android TV, etc.), meaning that we need to test beyond the most basic scenarios as well. Although we’ve typically done this using Selenium, we are increasing our testing capabilities by using Appium. Alternative options include external service providers like Saucelabs or Browserstack.

Appium runs by launching a server in your desktop machine which communicates with the “devices under test” (mobile, tablets, etc.). It can actually open browsers and test web pages as well as opening and testing Android and iOS apps. This is done by sending instructions that are interpreted and executed in the device thanks to third-party tools like Android UIAutomator or iOS XCUITest Driver under the hood. Therefore, Webdriverio can connect to the Appium server and send commands that are then proxied to the devices.

Appium needs some environment variables such as Java or Android vars and the configuration can be complex. One useful tool is the Appium-Doctor, which shows the dependencies that need to be met in order to make everything work. It is also possible, and in fact quite practical for local development, to use device emulators. Launching an emulated Android device is possible using the Android command line tools, whereas iOS emulation requires Xcode and is done automatically when Appium starts a session.

In our particular case, the Appium integration has become invaluable because it frees up significant resources that would have otherwise gone towards device emulation. This option ensures that our updates are compatible across all devices and platforms.

Final Thoughts

Automating the testing of the entire product stack across different browsers and platforms is not easy, but it is certainly possible! Some challenges remain, including:

• Multi-remote testing with mobile devices, which requires launching multiple emulators at the same time and synchronizing their behavior.
• Testing across different video players of different vendors and versions: each of these have a different set of SDKs and APIs that change a lot over time – making continuous compatibility an art form.

Nevertheless, having a solid E2E integration framework will give you confidence when it comes to delivering new features with the highest standard.

Happy testing!