Test Configuration with .NET Core and NUnit 3

Test Configuration with .NET Core and NUnit 3

This entry is part 7 of 9 in the series Launching WebDrivers in .NET Core the easy way

Launching WebDrivers in .NET Core the easy way

Getting configuration from the environment.

What is the big idea?

It is of course perfectly possible to code all the webdriver configuration into your tests as I have done so far, but what if I want someone else to be able to pull and run my code without having to go in and make code changes?

  • Maybe they have a grid set up on a different URL.
  • Perhaps they want to run the same tests against diffferent browsers on different days or on dedicated machines.
  • Possibly even they will be using it in a full CI pipeline and need to be able run the same tests against different base urls in dev / staging / production environments.

Before I go any further lets give some thought to what I want to achieve:

  1. There should be a default configuration that is used even if no configuation is specified.
  2. There should be the ability to use standard xxxx.runsettings files in Visual Studio. I have yet to work out how to use them in JetBrains Rider, but they can be set from the command line.
  3. It should be possible to override just the url of the Selenium Grid with a file in ‘My Documents’ or its equivalent.
  4. It should be possible to force a complete configuration with a file in ‘My Documents’ or its equivalent.

Three and four are to allow anyone to pull code from the repository and run it on their own machine, directing the RemoteWebDrivers to their own selenium grid, or forcing to a single local browser if preferred, e.g. for debugging / cross-browser toughening.

As I started writing this post I only intended this as a proof of concept on my WebDriverManagerTests project. As is often the case however, this has taken me somewhat longer to write and I have in fact pulled the important code already into a reusable project so I don’t have to recreate it for every future test project.

Setting configuration using .runsettings files

As explained in the Microsoft documentation, you can apply a runsettings file for test configuration in Visual Studio:

Specify a run settings file in the IDE
Select Test > Test Settings > Select Test Settings File and then select the .runsettings file. The file appears on the Test Settings menu, and you can select or deselect it. While selected, the run settings file applies whenever you select Analyze Code Coverage.

https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2017 – February 1 2019

This allows you to access the values in this file from within your code. So considering the WebDriverManagerTests I wrote about earlier. I want two different settings files.

  1. Run as an unit tests using my fake classes
  2. Run as system tests using a properly configured ‘real’ WebDriverFactory

This means that I need to be able to specify a full IWebDriverConfiguration as well as whether to use a fake.

So for running system tests something like this should be fine:

<?xml version='1.0' encoding='utf-8'?> <RunSettings>   <RunConfiguration>     <MaxCpuCount>1</MaxCpuCount>     <!-- TestSessionTimeout is only available with Visual Studio 2017 version 15.5 and higher -->     <!-- Using real WebDrivers is slooooow, 5 minutes timeout should be plenty-->     <TestSessionTimeout>300000</TestSessionTimeout>   </RunConfiguration>   <TestRunParameters>     <!-- Using real WebDrivers to run these as a System tests rather than unit tests. -->     <Parameter name="useRealWebDriver" value="true" />         <!-- Sample WebDriver Configuration settings -->     <Parameter name="browserType" value="Chrome" />     <Parameter name="isLocal" value="true" />     <Parameter name="platform" value="Windows" />     <Parameter name="headless" value="false" />     <Parameter name="windowSize" value="FHd" />   </TestRunParameters> </RunSettings>

And for unit tests:

<?xml version='1.0' encoding='utf-8'?> <RunSettings>   <RunConfiguration>     <MaxCpuCount>1</MaxCpuCount>     <!-- TestSessionTimeout is only available with Visual Studio 2017 version 15.5 and higher -->     <!-- Using the fakes are fast, 20 seconds should be plenty -->     <TestSessionTimeout>20000</TestSessionTimeout>   </RunConfiguration>   <TestRunParameters>   </TestRunParameters> </RunSettings>

Note: in this case I am not really setting anything as it should use defaults.

The TestSessionTimeout allows me to prevent the test run from hanging if anything goes wrong.

Accessing the configuration parameters in Nunit 3 tests

From the Nunit 3 documentation for TestContext:

TestParameters
Test parameters may be supplied to a run in various ways, depending on the runner used. For example, the console runner provides a command-line argument and v3.4 of the NUnit 3 VS Adapter will supports specifying them in a .runsettings file. The static TestParameters property returns an object representing those passed-in parameters.


The TestParameters object supports the following properties:
Count – The number of parameters.
Names – A collection of the names of the supplied parameters.
this[string name] – The value of a parameter. In Vb, use Item.


The TestParameters object supports the following methods:
Exists(string name) – Returns true if a parameter of that name exists.
Get(string name)– Returns the same value as the indexer.
Get<T>(string name, T defaultValue) – Returns the value of the parameter converted from a string to type T or the specified default if the parameter doesn’t exist. Throws an exception if conversion fails.


Note that all parameter values are strings. You may convert them to other Types using the generic Get method listed above or using your own code. An exception may be thrown if the supplied value cannot be converted correctly.

https://github.com/nunit/docs/wiki/TestContext

string

So starting with the simplest case, strings, I need to check if the setting exists, return it if so, or the provided default if not. The following static method should suffice.

using NUnit.Framework;         public static string GetStringSettingOrDefault(string settingName, string defaultValue)         {             return TestContext.Parameters.Exists(settingName) ?                 TestContext.Parameters.Get(settingName) :                 defaultValue;         }

bool

In this case I need to parse any string returned to a bool as safely as possible. I decided to lower case compare against the default value for this. In other words, if it exists but is not the default value set to !default.

using NUnit.Framework;     public static bool GetBoolSettingOrDefault(string settingName, bool defaultValue)     {         return !TestContext.Parameters.Exists(settingName)                || TestContext.Parameters.Get(settingName).ToLower().Equals(defaultValue.ToString().ToLower()) ?             defaultValue :             !defaultValue;     }

Enum

Saving the trickiest for last

  • As ever I need this method to be as safe as possible: Return the default value instead of throwing Exceptions on failure to parse.
  • It also needs to be generic; in other words it needs to be able to return a value for any Enum class as I have three Enums in my WebDriverConfiguration object.

Writing generics is somewhat outside my comfort zone, but its all good practice so here goes:

using System; using NUnit.Framework;         public static T GetEnumSettingOrDefault<T>(string settingName, T defaultValue)         {             T returnValue;             if (!TestContext.Parameters.Exists(settingName))             {                 returnValue = defaultValue;             }             else             {                 try                 {                     returnValue = (T)Enum.Parse(typeof(T), TestContext.Parameters.Get(settingName), true);                 }                 catch (Exception ex) when (ex is ArgumentException || ex is OverflowException)                 {                     returnValue = defaultValue;                 }             }             return returnValue;         }

My testing seems to show that it works, but I’m open for feedback. If you have any suggestions for improvement feel free to comment below or on Twitter.

(On re-reading this it sounds like I could have used the provided generic Get method and handled the exception. Oh well!)

Making it reusable

I can use those three methods to write a static TestSettings class for each project that I want to use my WebDriverFactory in, but that’s a pain to have to cut and paste each time; Lets share them.

My first thought was to add this to the NetCoreWebDriverFactory package. I realised however that this then adds an addiional dependency on Nunit. I want to keep the dependency list as short as possible and not imply that the WebDriverFactory is only useful for Nunit tests.

Plan B then, create a new package. Welcome to WebDriverFactoryNunitConfig. The above three methods can be packaged in a static Utils class and I’ll add a static WebDriverSettings class to get the configuration. Again your feedback on this decision is welcome.

using System; using AlexanderOnTest.NetCoreWebDriverFactory; using OpenQA.Selenium; using static AlexanderOnTest.WebDriverFactoryNunitConfig.TestSettings.Utils; namespace AlexanderOnTest.WebDriverFactoryNunitConfig.TestSettings {     public static class WebDriverSettings     {         public static Browser Browser { get; } = GetEnumSettingOrDefault("browserType", Browser.Firefox);         public static Uri GridUri { get; }               new Uri(GetStringSettingOrDefault("gridUri", "http://localhost:4444/wd/hub"));         public static bool IsLocal { get; }             = GetBoolSettingOrDefault("isLocal", true);         public static PlatformType PlatformType { get; } = GetEnumSettingOrDefault("platform", PlatformType.Windows);         public static WindowSize WindowSize { get; } = GetEnumSettingOrDefault("windowSize", WindowSize.Hd);         public static bool Headless { get; } = GetBoolSettingOrDefault("headless", false);     } }

Should work for grabbing from the active runsettings file or default. Of course I need a ‘useRealWebDriver’ TestSetting for this particular test class: that has to belong in my unit test project but of course I can use the methods in my WebDriverFactoryNunitConfig for setting it.

And finally… Local file overrides

Fortunately Newtonsoft.json.net comes ‘for free’ as a required library in Selenium WebDriver so I don’t need to think about what to use to deserialise local json files. Now I just need to find and read them.

As ever I need to be mindful of cross platform issues here:

Adding a new utility to my Utils class then using generics:

public static T GetFromFileSystemIfPresent<T>(             string filename,             System.Environment.SpecialFolder fileLocation = Environment.SpecialFolder.MyDocuments)         {             string fileLocationPath = Environment.GetFolderPath(fileLocation);             string configFilePath = Path.Combine(fileLocationPath, filename);             if (string.IsNullOrEmpty(configFilePath) || !File.Exists(configFilePath)) return default(T);             T localConfig;             using (StreamReader file = File.OpenText(configFilePath))             {                 JsonSerializer serializer = new JsonSerializer();                 localConfig = (T)serializer.Deserialize(file, typeof(T));             }             return localConfig;         }

For getting the GridUri from “Config_GridUri.json” then I can:

public static Uri GridUri { get; }             = Utils.GetFromFileSystemIfPresent<Uri>("Config_GridUri.json") ??               new Uri(GetStringSettingOrDefault("gridUri", "http://localhost:4444/wd/hub"));

And the equivalent parsing to a WebDriverConfiguration for the full Local Configuration.

NOTE: In both cases I am allowing Exceptions to be thrown here. It is to be expected that people might make mistakes on these configurations and I don’t want them to be quietly ignored

Putting it all together then

using System; using AlexanderOnTest.NetCoreWebDriverFactory; using OpenQA.Selenium; using static AlexanderOnTest.WebDriverFactoryNunitConfig.TestSettings.Utils; namespace AlexanderOnTest.WebDriverFactoryNunitConfig.TestSettings {     public static class WebDriverSettings     {         public static Browser Browser { get; } = GetEnumSettingOrDefault("browserType", Browser.Firefox);         /// <summary>         /// Uri of the grid. Configuration Priority:         /// 1. A value provided in "My Documents/Config_GridUri.json" (Windows) or "/Config_GridUri.json" (Mac / Linux)         /// 2. The value in an applied .runsettings file         /// 3. Default (Localhost) grid.         /// </summary>         public static Uri GridUri { get; }             = Utils.GetFromFileSystemIfPresent<Uri>("Config_GridUri.json") ??               new Uri(GetStringSettingOrDefault("gridUri", "http://localhost:4444/wd/hub"));         public static bool IsLocal { get; }             = GetBoolSettingOrDefault("isLocal", true);         public static PlatformType PlatformType { get; } = GetEnumSettingOrDefault("platform", PlatformType.Windows);         public static WindowSize WindowSize { get; } = GetEnumSettingOrDefault("windowSize", WindowSize.Hd);         public static bool Headless { get; } = GetBoolSettingOrDefault("headless", false);         /// <summary>         /// Return the Configuration to use - Priority:         /// 1. A value provided in "My Documents/Config_WebDriver.json" (Windows) or "/Config_WebDriver.json" (Mac / Linux)         /// 2. The value in an applied .runsettings file         /// 3. Default values.         /// </summary>         public static IWebDriverConfiguration WebDriverConfiguration { get; }             = Utils.GetFromFileSystemIfPresent<WebDriverConfiguration>("Config_WebDriver.json") ??               new WebDriverConfiguration             {                 Browser = Browser,                 IsLocal = IsLocal,                 WindowSize = WindowSize,                 GridUri = GridUri,                 PlatformType = PlatformType,                 Headless = Headless             };     } }

Package it up with the static methods in a static Utils class and “The job’s a good’un.” Configure WebDriverFactory instances for any Nunit 3 project. Check a .runsettings file for each environment into the repository and use the CI/CD pipeline to choose the correct one for each release.

In Summary

Progress made:

  • Developed the ability to generate WebDriverConfigurations from .runsettings files on Nunit 3 test projects
  • The required code released as its own Selenium.WebDriver.WebDriverFactoryNunitConfig package
  • Cross platform testing to verify it works. (By far the longest part of preparing this post!)

Lessons learnt:

  • Cross platform testing is the toughest bit of .NET Standard library creation
  • Testing complexity increases dramatically as the number of packages increases (unsurprisingly!)
  • The decision I made in the WebDriverManager implementation to store a delegate constructor rather than the requested WebDriverConfiguration made debugging my configs here much harder than it could have been.
  • TestContext.Parameters.Get(settingName) to return a setting from a .runsettings file
  • Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments) finds the Path to ‘My Documents’ on Windows or ‘Home’ on MacOS and Linux
  • Path.Combine(folderPath, filename) can be used to find the required file.

A reminder:

If you want to ask me a question, Twitter is undoubtedly the fastest place to get a response: My Username is @AlexanderOnTest so I am easy to find. My DMs are always open for questions, and I publicise my new blog posts there too.

Next time:

I will at last be exploring dependency injection using the native .NET Core functionality in the microsoft.extensions.dependencyinjection namespace.

[Edit 26/6/19 Moved into the new series: Launching WebDrivers in .NET Core the easy way]

Series Navigation<< Preparing to inject test configuration – Refactoring to use a configuration object.Applying a .runsettings file from the command line in .NET Core >>
Comments are closed.