A couple of months ago I wrote a post explaining that don’t like to disable tests for a disabled feature in certain environments. I much prefer to switch the expected results and confirm the feature is disabled than simply to not test it.

Sometimes however this really isn’t appropriate and you have no choice than to avoid running those tests. Today I’ll have a very quick look at ways to do this with NUnit 3.

The options.

1. Use different test categories
2. Use Assume.That()
3. Add Categories at runtime using TestCaseSource

1. Use different test categories

Nunit3 uses a CategoryAttribute to allow filtering of tests in the test runner. A Category is called a Trait in Visual Studio 2019 test explorer or a TestCategory using dotnet test. e.g.

using System.Collections;
using NUnit.Framework;

namespace AoT.TestFiltering
{
    public class FilteringTests
    {
        private const string Feature1 = "feature1";
        private const string Feature2 = "feature2";
        
        [SetUp]
        public void Setup()
        {
        }

        [Test]
        [Category(Feature1)]
        public void ThisTestShouldRunWithFeature1Category()
        {
            Assert.Pass();
        }
    }
}

It is easy to apply some filtering in Visual Studio or the command line to only include some categories, or to exclude categories.

Benefits

• Really simple to apply and use.
• It is possible to add multiple categories to a test.
• Categories can be applied at class or even base class level and will be inherited.

Drawbacks

The filtering logic is very simple but limited

You can INCLUDE all tests with given categories or EXCLUDE all tests with given categories. If you have multiple categories attached to a test you cannot use complex “include this category but not ones that are also that category” type logic.

The tester / developer or CI pipeline has to have knowledge of which filters to apply in each environment.

It is all to easy for a stage to be misconfigured by someone who is not sure what should be enabled: Then a whole set of tests will not be run where they should be. You won’t even know that the feature is broken as your tests aren’t shouting at you.

Trust me: It is very frustrating when you find a whole bunch of tests just quietly being ignored, only after someone finds the broken feature manually. So much for short feedback loops!

In Summary

Categories are useful, but not the whole solution.

2. Use Assume.That()

This approach uses Nunit’s ability to skip a test that doesn’t meet specific conditions. e.g.

        [Test]
        public void ThisTestShouldNotSkipIfSelected()
        {
            Assume.That(true, "The required feature is not enabled");
            Assert.Pass();
        }

        [Test]
        public void ThisTestShouldSkipIfSelected()
        {
            Assume.That(false, "The required feature is not enabled");
            Assert.Pass();
        }

Benefits

You can ensure that tests don’t run unless certain conditions are met

Perfect for setting the condition in your runsettings file to prevent accidentally writing test data to a live production database.

Drawbacks

Pass rates <100%

This may not be true for all Continuous Integration pipelines, but certainly Azure Devops reports the tests passed as a percentage of total tests. It doesn’t really matter, but ideally you want all relevant tests passing to report as a 100% pass rate.

In Summary

Assumptions are a quick and easy way to ensure that inappropriate tests are skipped. Using the .runsettings file to inform the decision should protect from accidental test running, even by unwitting devs / testers in the IDE.

3. Add Categories at runtime using TestCaseSource

The idea here is to decide at runtime which tests to put in to the category that you wish to run. e.g.

        [TestCaseSource(typeof(FilteringTests), nameof(AlwaysAddFeature1Category))]
        public void ThisTestShouldAlsoBecomeAFeature1Test()
        {
            Assert.Pass();
        }

        public static IEnumerable AlwaysAddFeature1Category()
        {
            if (true)
            {
                yield return new TestCaseData().SetCategory(Feature1);
            }
            else
            {
                yield return new TestCaseData();
            }
        }

        [TestCaseSource(typeof(FilteringTests), nameof(NeverAddFeature1Category))]
        public void ThisTestShouldNotBecomeAFeature1Test()
        {
            Assert.Fail();
        }

        public static IEnumerable NeverAddFeature1Category()
        {
            if (false)
            {
                yield return new TestCaseData().SetCategory(Feature1);
            }
            else
            {
                yield return new TestCaseData();
            }
        }

Benefits

You can ensure that tests don’t run in your pipeline unless certain conditions are met

Now you can assign the same category (e.g. – “RunOnCI”) to tests dynamically so that EVERY stage has the same test filtering. This massively reduces the risk of someone running the wrong filter on a particular environment.

Be careful though, the test is still a test, you might still want to add the Assume to protect against accidentally running the wrong category somewhere.

CI reports the passrate against the tests that you wanted to run

No more 85% pass rates on successful releases.

Drawbacks

They are more complex..

More code, more chance of making a mistake when writing the tests and having unexpected results.

and confusing

There is something rather magical when you have tests suddenly appearing in your filtered test list as the tests run, but to a developer new to this code it is likely to confuse them.

In Summary

This is really powerful, but certainly has potential to confuse. I am generally averse to ‘being clever’ and prefer to ‘Keep It Simple Stupid.’

Complete code

using System.Collections;
using NUnit.Framework;

namespace AoT.TestFiltering
{
    public class FilteringTests
    {
        private const string Feature1 = "feature1";
        private const string Feature2 = "feature2";

        [Test]
        public void ThisTestShouldRunWithNoCategoryFilters()
        {
            Assert.Pass();
        }

        [Test]
        [Category(Feature1)]
        public void ThisTestShouldRunWithFeature1Category()
        {
            Assert.Pass();
        }

        [Test]
        [Category(Feature2)]
        public void ThisTestShouldRunWithFeature2Category()
        {
            Assert.Pass();
        }

        [Test]
        public void ThisTestShouldNotSkipIfSelected()
        {
            Assume.That(true, "The required feature is not enabled");
            Assert.Pass();
        }

        [Test]
        public void ThisTestShouldSkipIfSelected()
        {
            Assume.That(false, "The required feature is not enabled");
            Assert.Pass();
        }

        [TestCaseSource(typeof(FilteringTests), nameof(AlwaysAddFeature1Category))]
        public void ThisTestShouldAlsoBecomeAFeature1Test()
        {
            Assert.Pass();
        }

        public static IEnumerable AlwaysAddFeature1Category()
        {
            if (true)
            {
                yield return new TestCaseData().SetCategory(Feature1);
            }
            else
            {
                yield return new TestCaseData();
            }
        }

        [TestCaseSource(typeof(FilteringTests), nameof(NeverAddFeature1Category))]
        public void ThisTestShouldNotBecomeAFeature1Test()
        {
            Assert.Fail();
        }

        public static IEnumerable NeverAddFeature1Category()
        {
            if (false)
            {
                yield return new TestCaseData().SetCategory(Feature1);
            }
            else
            {
                yield return new TestCaseData();
            }
        }
    }
}

Final thoughts : TL/DR

• I stick with my previous recommendation. Far better to test differently in different environments than to have to try to run different sets of tests.

If you MUST however run different combinations on different environments:

• Categories alone can be useful, but I find expecting that pipelines stages are correctly configured and stay correct is optimistic, especially as it can take a long time to check. They are fine for simple situations but quickly scale to be a problem.
• Assumptions provide much needed control to safegaurd against accidental running, but do make for messy reporting.
• Dynamically assigning a single Category to your tests based on a runsettings file is incredibly powerful and ensures that you are not relying on CI configuration to be correct in each environment, but it complex and could be confusing.
• I have just felt the need to implement this. I guess in a few months I will know whether it was a good idea.

Did I miss anything or make a mistake?

Do you have a better plan? Please do let me know, in the comments below or on Twitter if I have made any mistakes or missed anything helpful. The comments below are largely taken from my previous post, but are even more relevant today.

Writing code that works for this is a lot easier than working out what configurability is required and testing that it works as intended.

Progress made:

• Defined a means to ensure that my checked in runsettings files determine which tests are run without complex logic at each stage of the CI pipeline

Lessons learnt:

• Testing for different functionality on different servers is a challenging and complex issue
• Even with a plan, getting it right is difficult especially when trying to anticipate all the combinations that might arise
• Providing a means e.g. a test setting that can be configured in the release without a code check in can save a lot of trouble
• Dynamic filtering is a pattern, it provides benefits and problems: You should determine the cost benefit analysis for your own situation

A reminder:

If you want to ask me a question, Twitter is undoubtedly the fastest place to get a response: My Username is @AlexanderOnTest so I am easy to find. My DMs are always open for questions, and I publicise my new blog posts there too.