I had one of those days last week. Three hours spent trying to work out why my code wasn’t behaving: A test parameter was clearly in the applied RunSettings file, but when I tried to call the value from code it never changed from my default value. Why was it hiding from me?

The background

You know the deal, before logging off I had the work 90% finished. In the morning, I JUST needed to add some configuration to ensure that I’m only running the appropriate tests on each environment. Nothing I haven’t done many times before. Being core code however, these tests, including my new ones have around 20 different runsettings configurations available so those files are starting to get a little unwieldy.

So I tidy them up a little, group the settings in logical groups and try to run the tests. But one of them is steadfastly refusing to run. The Assume.That() based off one of my runsettings ALWAYS returns false and ignores my test.

Troubleshooting

Ok: So starting out with the basics:

After an hour or so, no luck. Still no change and no wiser as to what on earth is going on.

Debugging

Time to start investigating with debug code. I typically use something like this to get the string value of the runsettings using NUnit 3s TextContext.

        private string GetSettingOrDefault(string parameterName, string defaultValue) => 
            TestContext.Parameters.Exists(parameterName) ? 
                TestContext.Parameters.Get(parameterName) : 
                defaultValue;

Now ternary operators are great for readability, but in C# they are a pain for debugging so lets split it and add some breakpoints:

        private string GetSettingOrDefault2(string parameterName, string defaultValue)
        {
            var parameterExists = TestContext.Parameters.Exists(parameterName);
            if (parameterExists)
            {
                return TestContext.Parameters.Get(parameterName);
            }
            else
            {
                return defaultValue;
            }
        }

and now we can inspect what’s going on with my TestContext:

OK. I am properly confused now.

Time to check out the full list of parameters:

Can you spot the issue?

When I found this issue there were 17 entries in the runsettings parameter list. Its a little easier to see here with only 7.

Hopefully you spotted the duplicated parameter: ShouldPray

• Parameters before the duplicate will load as expected
• The duplicated parameter will load the first value
• ANY parameters after the duplicate will not be loaded
• No warning will be given

Now I get that duplicates should not happen:

• I would have absolutely no problem with the test runner throwing an exception when finding a duplicated parameter. (This is how JetBrains Rider handles this situation.)
• I would even be OK with the duplicated parameter taking either the first or final values and all the rest working fine.
• But really: Loading everything up to the duplicated value, but nothing afterwards!
• Just for confirmation, the same happens when called from the command line using dotnet test.

Final thoughts : TL/DR

If you are having parameter problems, check the file for duplicates.

Don’t expect a warning and any later parameters will not be loaded.

If there is a duplicated parameter in your .runsettings file:

✔ Parameters before the duplicate will load as expected.

– The duplicated parameter will load the first value

❌ ANY parameters after the duplicate will not be loaded

❌ No warning will be given.

Progress made:

• 3 hours lost looking for a problem in my code, when in fact it was in my configuration.
• At least I found it myself: With it controlling which tests are run, it could easily be completely missed in the CI pipeline.

Lessons learnt:

• Config is just as capable of breaking your tests as it is your application. Don’t assume that it is correct.
• Duplicated parameters in a .runsettings will make it silently fail to pick some up.

A reminder:

If you want to ask me a question, Twitter is undoubtedly the fastest place to get a response: My Username is @AlexanderOnTest so I am easy to find. My DMs are always open for questions, and I publicise my new blog posts there too.

Demo code

using FluentAssertions;
using NUnit.Framework;

namespace RunSettings
{
    public class Tests
    {
        [Test]
        public void FirstTestParameterIsReturned()
        {
            string parameterName = "premier";
            TestContext.Parameters.Exists(parameterName).Should()
                .BeTrue("it has been set");
            TestContext.Parameters.Get(parameterName).Should()
                .BeEquivalentTo("true");
        }

        [Test]
        public void TestParameter_shouldRock_IsReturned()
        {
            string parameterName = ("shouldRock");
            TestContext.Parameters.Exists(parameterName).Should()
                .BeTrue("it has been set");
            TestContext.Parameters.Get(parameterName).Should()
                .BeEquivalentTo("true");
        }
        
        
        [Test]
        public void TestParameter_shouldRoll_IsReturned()
        {
            string parameterName = ("shouldRoll");
            TestContext.Parameters.Exists(parameterName).Should()
                .BeTrue("it has been set");
            TestContext.Parameters.Get(parameterName).Should()
                .BeEquivalentTo("true");
        }

        
        [Test]
        public void TestParameter_shouldCountFish_IsReturned()
        {
            string parameterName = ("shouldCountFish");
            TestContext.Parameters.Exists(parameterName).Should()
                .BeTrue("it has been set");
            TestContext.Parameters.Get(parameterName).Should()
                .BeEquivalentTo("true");
        }

        
        [Test]
        public void TestParameter_shouldPray_IsReturned()
        {
            string parameterName = ("shouldPray");
            TestContext.Parameters.Exists(parameterName).Should()
                .BeTrue("it has been set");
            TestContext.Parameters.Get(parameterName).Should()
                .BeEquivalentTo("true");
        }


        [Test] public void UltimateTestParameterIsReturned()
        {
            string parameterName = ("ultimate");
            GetSettingOrDefault(parameterName, "false");
            bool check = !TestContext.Parameters.Exists(parameterName) || 
                         TestContext.Parameters.Get(parameterName).ToLowerInvariant().Equals("false");

            Assume.That(TestContext.Parameters.Exists(parameterName));
            TestContext.Parameters.Exists(parameterName).Should()
                .BeTrue("it has been set");
            TestContext.Parameters.Get(parameterName).Should()
                .BeEquivalentTo("true");
        }

        private string GetSettingOrDefault(string parameterName, string defaultValue)
        {
            var parameterExists = TestContext.Parameters.Exists(parameterName);
            if (parameterExists)
            {
                return TestContext.Parameters.Get(parameterName);
            }
            else
            {
                return defaultValue;
            }
        }
    }
}

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <TestRunParameters>
        <Parameter name ="premier" value="true" />
        <Parameter name ="shouldPray" value="false" />
        <Parameter name ="shouldRock" value="true" />
        <Parameter name ="shouldRoll" value="true" />
        <Parameter name ="shouldCountFish" value="true" />
        <Parameter name ="shouldPray" value="true" />
        <Parameter name ="ultimate" value="true" />
    </TestRunParameters>
</RunSettings>

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <TestRunParameters>
        <Parameter name ="premier" value="true" />
        <Parameter name ="shouldRock" value="true" />
        <Parameter name ="shouldRoll" value="true" />
        <Parameter name ="shouldCountFish" value="true" />
        <Parameter name ="shouldPray" value="true" />
        <Parameter name ="ultimate" value="true" />
    </TestRunParameters>
</RunSettings>