When tests don’t work….

One of those issues that I could very easily have missed. Over Christmas, in between working through CheckStyle hell I had a play around with adding Opera to the list of browsers available on my grid. I know its kind of pointless as being based on the same engine as Chrome, but it is one of my favourite browsers so I thought I’d give it a go.

I’ve made a few incremental changes to my grid since I last wrote about it. I have now integrated my hub and node launch commands into a single PowerShell script and use a nodeconfig file for setup. I will go through all of that next week. This week I am looking at why one of my tests wasn’t failing when it should have been.

The background

Over the last year I have found myself using Opera more and more in my general browsing at home. Given that, what better way to relax than add it to my list of supported test browsers…. Unfortunately it wasn’t long before I happened to be looking at the node screen whilst the tests were running. I was launching Chrome twice, but never Opera, despite my tests passing.

A look at the code I was using to launch the Webdriver:

Add a new OperaDriverManager class modelled on the others

package tech.alexontest.poftutor.infrastructure;

import org.openqa.selenium.opera.OperaDriver;
import org.openqa.selenium.opera.OperaDriverService;
import org.openqa.selenium.opera.OperaOptions;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;

import java.io.File;
import java.io.IOException;

public final class OperaDriverManager extends AbstractDriverManager implements WebDriverManager {

    private OperaDriverService operaDriverService;

    private final File operaDriverExe;

    private final boolean isLocal;

    OperaDriverManager(final boolean isLocal) {
        final String path = getClass().getClassLoader().getResource("operadriver.exe").getPath();
        operaDriverExe = new File(path);
        System.setProperty("webdriver.opera.driver", path);
        this.isLocal = isLocal;
    }

    @Override
    public void startService() {
        if (null == operaDriverService) {
            try {
                operaDriverService = new OperaDriverService.Builder()
                        .usingDriverExecutable(operaDriverExe)
                        .usingAnyFreePort()
                        .build();
                operaDriverService.start();
            } catch (final IOException e) {
                e.printStackTrace();
            }
            System.out.println("OperaDriverService Started");
        }
    }

    @Override
    public void stopService() {
        if (null != operaDriverService && operaDriverService.isRunning()) {
            operaDriverService.stop();
            System.out.println("OperaDriverService Stopped");
        }
    }

    @Override
    public String createDriver() {
        final OperaOptions options = new OperaOptions()
                .setBinary(new File("C:/Program Files/Opera/50.0.2762.58/opera.exe"));
        options.setCapability("browserName", "operablink");
        // add additional options here as required
        if (!isLocal) {
            setDriver(new RemoteWebDriver(getGridUrl(), options));
        } else {
            setDriver(new OperaDriver(options));
        }
        System.out.println("OperaDriver Started");
        return DesiredCapabilities.operaBlink().getBrowserName();
    }
}

Note the highlighted line 54 and the need to give the path to the binary. A default installation on both the local machine and the selenium grid node means that the path should be valid for both.

The problem is line 63 return DesiredCapabilities.operaBlink().getBrowserName();

I remember thinking that this contacted the driver to get the capabilities of the driver, although now I have no idea why I would have thought that. I guess it was probably 2 am again and I was tired. Anyway a quick bit of research tells me that this just returns the ‘browser name’ from a new capabilities object, and doesn’t even confirm that this is the type that I asked for.

I also need to add an extra option to my DriverType Enum and the corresponding switch section of the DriverManagerFactory class. Nothing difficult there so I will let you work them out for yourself or checkout the project on GitHub.

Looking at the Test class itself

    @CsvFileSource(resources = "operaTests.csv")
    void driverFactoryWorks(final DriverType driverType, final String browserName) {
        setDriverManager(DriverManagerFactory.getManager(driverType));
        getDriverManager().startService();
        assertThat(getDriverManager().createDriver())
                .as(String.format("Checking That browser is of type %s", browserName))
                .isEqualToIgnoringCase(browserName);
        final String homePageURL = "https://www.example.com/";
        final WebDriver driver = getDriver(homePageURL);
        assertThat(driver.getCurrentUrl())
                .as(String.format(
                        "Check that browser '%1$s' successfully loads the homepage '%2$s'", browserName, homePageURL
                ))
                .isEqualToIgnoringCase(homePageURL);
    }

As we can see from, I compare the value from the new Capabilities object against an expected value. Completely pointless and now, when running, I get debugging to remind me to use Options instead of capabilities as well. So lets see if we can do any better than that:

1. Well, I can return the ACTUAL Options object that was used in the Browser constructor. This will at least validate that my code has asked for the correct browser.
2. The URL check is fine for ensuring that I can navigate to the page, but…

How do I go about verifying the actual browser that has been launched? Well it seems that there isn’t really a good answer to that question. I can get the user agent string maybe and see if that can help at all. They are unreliable at best, but maybe I can find something to work. so modifying all the *driver Options classes like below.

    @Override
    public String createDriver() {
        final OperaOptions options = new OperaOptions()
                .setBinary(new File("C:/Program Files/Opera/50.0.2762.58/opera.exe"));
        options.setCapability("browserName", "operablink");
        // add additional options here as required
        if (!isLocal) {
            setDriver(new RemoteWebDriver(getGridUrl(), options));
        } else {
            setDriver(new OperaDriver(options));
        }
        System.out.println("OperaDriver Started");
        return options.getBrowserName();
    }

I can then verify what I have asked for.

(Edit: 27/1/18 The first time I looked at this post live I realised that my first attempt at this matching strings totally failed to recognise if Chrome failed to launch – Here is the improved and working version using regex)

    void driverFactoryWorks(final DriverType driverType, final String browserName) {
        setDriverManager(DriverManagerFactory.getManager(driverType));
        getDriverManager().startService();
        assertThat(getDriverManager().createDriver())
                .as(String.format("Checking That browser is of type %s", browserName))
                .isEqualToIgnoringCase(browserName);
        final String homePageURL = "https://www.example.com/";
        final WebDriver driver = getDriver(homePageURL);
        final String agentString = (String) ((JavascriptExecutor) driver).executeScript("return navigator.userAgent;");
        final Pattern regex = driverType.getRegex();
        assertThat(agentString)
                .as("AgentString does not match pattern for %s", driverType.name())
                .containsPattern(driverType.getRegex());

        assertThat(driver.getCurrentUrl())
                .as(String.format(
                        "Check that browser '%1$s' successfully loads the homepage '%2$s'", browserName, homePageURL
                ))
                .isEqualToIgnoringCase(homePageURL);
    }

line 9 returns the user agent Strings and we can verify them if we can identify anything unique about each browser.

Google Chrome:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36

Microsoft Edge:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299

Firefox

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0

Internet Explorer 11

Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; .NET4.0C; .NET4.0E; rv:11.0) like Gecko

Opera

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 OPR/50.0.2762.58

Now I just need to find some uniqueness and it can be stored as  a property of the DriverType

(Edit: 27/1/18 Corrected working version using regex)

package tech.alexontest.poftutor.infrastructure;

import java.util.regex.Pattern;

public enum DriverType {
    CHROME("^(?!.*?\\bOPR/\\b)^(?!.*?\\bEdge/\\b).*?\\bChrome/\\b.*?\\bSafari/\\b.*$"),
    CHROME_LOCAL("^(?!.*?\\bOPR/\\b)^(?!.*?\\bEdge/\\b).*?\\bChrome/\\b.*?\\bSafari/\\b.*$"),
    CHROME_LOCAL_HEADLESS("HeadlessChrome"),
    EDGE("Edge"),
    FIREFOX("Firefox"),
    IE("rv:11.0"),
    OPERA("OPR/"),
    OPERA_LOCAL("OPR/"),;

    private final Pattern regex;

    DriverType(final String regexString) {
        this.regex = Pattern.compile(regexString);
    }

    public Pattern getRegex() {
        return regex;
    }
}

and there you have it:

It is now failing a test to confirm that it doesn’t launch there. No problem with the local Opera either. Oh well That’s enough for this week.

 

Progress made this week:

• Added Opera support in my tests and for the grid.
• Opera just doesn’t work on the grid.
• Used the user Agent string to verify the correct browser is launched in the framework tests.
• (Added 27/1) Fixed the tests after release.

Lessons learnt:

• Always check carefully what your tests are actually testing.
• Keep your eyes open for problems.
• The User agent String is a crude way to get browser information, but it can be used effectively.
• Enums can be useful to act as a store of relevant data.
• (Added 27/1) A reviewer to submit PRs to might help spot oversights.
• (Added 27/1) @EnumSource for JUnit 5 test parameters is pretty handy.

See you next week!

Alexander

Edit: 27/1/18 Having messed up the first time I wanted to be sure that the regex would work properly. I made a little check project to verify failing as well as passing.

As I am not checking this in, but it gave me a chance to include a new JUnit 5parameter resolver (@EnumSource) I include the classes here.

package regextests;

import java.util.regex.Pattern;

public enum DriverType {
    CHROME("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36",
            "^(?!.*?\\bOPR/\\b)^(?!.*?\\bEdge/\\b).*?\\bChrome/\\b.*?\\bSafari/\\b.*$"),
    OPERA("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 OPR/50.0.2762.58",
            "OPR"),
    IE11("Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET4.0C; .NET4.0E; .NET CLR 3.5.30729; .NET CLR 3.0.30729; rv:11.0) like Gecko",
            "rv:11.0"),
    EDGE("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299",
            "Edge"),
    FIREFOX("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0",
            "Firefox"),
    SAFARI("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5",
            "^(?!.*?\\bChrome/\\b).*?\\bSafari/\\b.*$"),
    ;

    private final String agentString;
    private final Pattern regex;

    DriverType(final String agentString, final String regexString) {
        this.agentString = agentString;
        regex = Pattern.compile(regexString);
    }

    public String getAgentString() {
        return agentString;
    }

    public Pattern getRegex() {
        return regex;
    }
}

and the parameterised test using @EnumSource.

package tech.alexontest.regextests;

import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.EnumSource;
import regextests.DriverType;

import static org.assertj.core.api.Assertions.assertThat;

class AgentStringTests {

    @ParameterizedTest
    @EnumSource(DriverType.class)
    void regexWorks(final DriverType driverType) {
        //iterate over all the values in the Enum
        for (DriverType driverTypeToCheck : DriverType.values()) {
            if (driverType != driverTypeToCheck) {
                assertThat(driverTypeToCheck.getAgentString())
                        .as("AgentString %2$s matches pattern for %1$s.", driverType.name(), driverTypeToCheck.name())
                        .doesNotContainPattern(driverType.getRegex());
                System.out.println(String.format("AgentString %2$s does not match pattern for %1$s.", driverType.name(), driverTypeToCheck.name()));
            } else {
                assertThat(driverTypeToCheck.getAgentString())
                        .as("AgentString matches")
                        .containsPattern(driverType.getRegex());
                System.out.println(String.format("AgentString %2$s matches pattern for %1$s.", driverType.name(), driverTypeToCheck.name()));
            }
        }
    }
}

I do particularly like the @EnumSource for parameter resolution if I am trying to test the objects referenced by it. I can add new types to the enum and the test will automatically run the additions. The downside is that the enum must be the sole parameter so must contain all the data you require.

Of course the nice thing about a simple unit test like this, with no browsers to launch it runs in extra quick time.

Series Navigation

<< Driver Factory Part 3 – RemoteWebDrivers and my very own gridEnter the Matrix: Selenium grid made simple >>