Investigating JUnit 5 asserts

Investigating JUnit 5 asserts

Pesky asserts!

Testing tests.

Of course if your tests are going to be informative, it is not enough that they pass when everything is right and fail when everything is wrong. It is vital that when a test does fail, it gives an informative error message to help identify the problem. Ideally, your well designed test will fail at an assert that gives a clear description of the problem. That requires some effort to find out what your test will report if there is a problem. You need to test your tests.

In my case, the failure was unexpected, but exactly the kind of thing that might happen when automatically testing websites. My first attempt at this setting up this site was pretty, but really did not reflect what I am trying to do, so I used a new theme. One of the changes in the new theme was new CSS that capitalised the title. (From ‘Alexander on Testing’ to ‘ALEXANDER ON TESTING’) This failed my previously working test!

So time to write some tests to find out how usable JUnit 5 Assertions are.

To test this I need to choose some test cases.

1. String expected = “Hello” against itself
2. String expected = “Hello” against String sameAsExpected = “Hello”
3. String expected = “Hello” against String actual = “HELLO”
4. String expected = “Hello” against String different = “Goodbye”

  • I expect 1 and 2 should always pass.
  • 3 should fail until we use ‘equalsIgnoreCase’
  • 4 should always fail

To avoid too much code, and allow me to see the results of ALL failing tests I am going to use

assertAll(String heading, Executable... executables)

Asserts that all supplied executables do not throw exceptions.
This does not fail the test at the first failed assertion, but completes the test and reports all failed assertions. This approach is also known as soft assertions.
To complicate things, I need to be using JUnit 5 tests for the AssertAll to work. That means reinstating the junit-platform-gradle-plugin from last week.
Again discovering the shortcomings of current JUnit 5 support I also discovered that (from Intellij IDEA at least) you cannot run only selected tests from the IDE. It ALWAYS runs all tests in the project. This is a pain as I have no need to waste time firing up a WebDriver for these tests. I’ll annotate last week’s test with @Ignore for now so I am only running the tests I write this week. I’ll give the full code at the end, but for now lets go through code and results one at a time.

1 using assertEquals

assertEquals(Object expected, Object actual, String message)

Asserts that expected and actual are equal.

Code:

@Test
void assertEqualsStringComparisonTest() {
    assertAll("JUnit assertEquals String comparisons",
            () -> assertEquals(expected,expected, "Same objects should never fail"),
            () -> assertEquals(expected, sameAsExpected, "Different Strings with same content also pass."),
            () -> assertEquals(expected, actual, "Strings differing in case should fail."),
            () -> assertEquals(expected, different, "Strings differing in content should always fail.")
    );
}

and the results as reported:

JUnit Jupiter:AssertTest:assertEqualsStringComparisonTest()
MethodSource [className = ‘pftester.AssertTest’, methodName = ‘assertEqualsStringComparisonTest’, methodParameterTypes = ”]
=> org.opentest4j.MultipleFailuresError: JUnit assertEquals String comparisons (2 failures)
Strings differing in case should fail. ==> expected: <Hello> but was: <HELLO>
Strings differing in content should always fail. ==> expected: <Hello> but was: <Goodbye>

Mmm. OK. That is pretty intelligible, but different cases fails the assertion. What if I want it to ignore case?

 

2 using assertLinesMatch

Looking at the Assertions class there is one that is written specifically for working with (Lists of) Strings.

assertLinesMatch(List<String> expectedLines, List<String> actualLines)

Asserts that expected list of Strings matches actual list.

Code:

@Test
void assertLinesMatchStringComparisonTest() {
    final List<String> expectedList = Arrays.asList(expected, expected, expected, expected);
    final List<String> actualList = Arrays.asList(expected, sameAsExpected, actual, different);

    assertLinesMatch(expectedList, actualList);
}

Nice succinct code, especially if you can easily grab a list of Strings in your test. The results as reported:

JUnit Jupiter:AssertTest:assertLinesMatchStringComparisonTest()
MethodSource [className = ‘pftester.AssertTest’, methodName = ‘assertLinesMatchStringComparisonTest’, methodParameterTypes = ”]
=> org.opentest4j.AssertionFailedError: expected line #3:`Hello` doesn’t match ==> expected: <Hello
Hello
Hello
Hello> but was: <Hello
Hello
HELLO
Goodbye>

Yuck! That’s horrid. Good luck making sense out of that with long lists. I’m not sure if this is failing at first failure, or accepting HELLO and only failing on Goodbye. That’s so ugly I’m not even going to investigate.

 

3 using assertTrue and String.equalsIgnoreCase(String)

assertTrue(boolean condition, String message)

Asserts that the supplied condition is true.

Code:

@Test
void assertTrueIgnoreCaseStringComparisonTest() {
    assertAll("JUnit assertTrue (equalsIgnoreCase(..)) String comparisons",
            () -> assertTrue(expected.equalsIgnoreCase(expected), "Same objects should never fail"),
            () -> assertTrue(expected.equalsIgnoreCase(sameAsExpected), "Different Strings with same content should pass."),
            () -> assertTrue(expected.equalsIgnoreCase(actual), "Strings differing in case should not fail now."),
            () -> assertTrue(expected.equalsIgnoreCase(different), "Strings differing should always fail.")
    );
}

and the results as reported:

 JUnit Jupiter:AssertTest:assertTrueIgnoreCaseStringComparisonTest()
MethodSource [className = ‘pftester.AssertTest’, methodName = ‘assertTrueIgnoreCaseStringComparisonTest’, methodParameterTypes = ”]
=> org.opentest4j.MultipleFailuresError: JUnit assertTrue (equalsIgnoreCase(..)) String comparisons (1 failure)
Strings differing should always fail.

OK Now we are getting somewhere – but the fail message isn’t clear and it doesn’t report expected and actual results. Let’s see how we can improve it.

 

4 using assertTrue, String.equalsIgnoreCase(String) and a messageSupplier

assertTrue(boolean condition, Supplier<String> messageSupplier)

Asserts that the supplied condition is true.

This allows me to use a lambda to provide a better message of the form:

assertTrue(expectedString.equalsIgnoreCase(actualString),
        () ->  String.format("%1$s -> The expected is: '%2$s' while the actual is: '%3$s'", description, expectedString, actualString));

Nested lambdas however are hard to read and no need to repeat myself. Time for a helper method.

Code:

@Test
void assertTrueIgnoreCaseWithExplanationStringComparisonTest() {
    assertAll("JUnit assertTrue (equalsIgnoreCase(..)) String comparisons with explanation",
            () -> assertTrueIgnoreCaseWithExplanation("Same objects",expected, expected),
            () -> assertTrueIgnoreCaseWithExplanation("Same contents", expected, sameAsExpected),
            () -> assertTrueIgnoreCaseWithExplanation("Different Case", expected, actual),
            () -> assertTrueIgnoreCaseWithExplanation("Different Words", expected, different)
    );
}


private void assertTrueIgnoreCaseWithExplanation(final String description, final String expectedString, final String actualString){
    assertTrue(expectedString.equalsIgnoreCase(actualString),
            () ->  String.format("%1$s -> The expected is: '%2$s' while the actual is: '%3$s'", description, expectedString, actualString));

}

and the results as reported:

JUnit Jupiter:AssertTest:assertTrueIgnoreCaseWithExplanationStringComparisonTest()
MethodSource [className = ‘pftester.AssertTest’, methodName = ‘assertTrueIgnoreCaseWithExplanationStringComparisonTest’, methodParameterTypes = ”]
=> org.opentest4j.MultipleFailuresError: JUnit assertTrue (equalsIgnoreCase(..)) String comparisons with explanation (1 failure)
Different Words -> The expected is: ‘Hello’ while the actual is: ‘Goodbye’

That’s better. It is just about usable, and of course we could always create our own Asserts.java class and save that method as a static method for reuse.

 

So is that what I would do?

Well no actually. I would use Joel Costigliola’s assertj library. That means adding a dependency to our build.gradle. Here we only actually need it in the testCompile group, but as I plan to use it widely in my main code I’m going to drop it straight into the compile group.

// https://mvnrepository.com/artifact/org.assertj/assertj-core
compile group: ‘org.assertj’, name: ‘assertj-core’, version: ‘3.8.0’

Code:

@Test
void assertjTest() {
    assertAll("String comparison using Joel Costigliola's assertj library.",
            () -> assertThat(expected).isEqualToIgnoringCase(expected),
            () -> assertThat(sameAsExpected).isEqualToIgnoringCase(expected),
            () -> assertThat(actual).isEqualToIgnoringCase(expected),
            () -> assertThat(different).isEqualToIgnoringCase(expected)
            );
}

This is concise and readable, so what does it report?

JUnit Jupiter:AssertTest:assertjTest()
MethodSource [className = ‘pftester.AssertTest’, methodName = ‘assertjTest’, methodParameterTypes = ”]
=> org.opentest4j.MultipleFailuresError: String comparison using Joel Costigliola’s assertj library. (1 failure)
Expecting:
<“Goodbye”>
to be equal to:
<“Hello”>
ignoring case considerations

Perfect. Job done. Would be even clearer without the wrapper from the assertAll call.

All that’s left to do now is to swap in my shiny new assertion into my original test.

assertThat(driver.findElement(By.cssSelector(".site-title")).getText())
        .isEqualToIgnoringCase("Alexander on Testing");

and undo most of my changes this week.

Next week I will be looking at how to use Abstract classes to build tests with a new browser for each test, and if I can make one with one browser reused for a whole Test class.

Progress made this week:

  • Investigated JUnit 5 assertions and how they can be made to report failures well.
  • Adopted assertj instead for its ease of use and clarity.
  • Our first test now passes, and should give a meaningful message if my site changes / breaks.

Lessons learnt:

  • Intellij IDEA cannot run individual tests or classes when using the unit-platform-gradle-plugin (nor I suspect can anything else!
  • JUnit 5 Asssertions contain some cool ideas, but I’m not sure how much I will use them.
  • JUnit 5 assertAll(..) is the one exception I could see me using.

Full classes used this week:

build.gradle

group 'com.alexanderontesting'
version '0.1'

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'org.junit.platform:junit-platform-gradle-plugin:1.0.0-RC2'
    }
}

apply plugin: 'org.junit.platform.gradle.plugin'

junitPlatform {
    platformVersion '1.0.0-RC2'
    reportsDir file('build/test-results/junit-platform')
    //enableStandardTestTask true
}

apply plugin: 'java'

sourceCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '3.5.1'
    compile group: 'org.assertj', name: 'assertj-core', version: '3.8.0'

    testCompile group: 'junit', name: 'junit', version: '4.12'
    testCompile group: 'org.junit.jupiter', name: 'junit-jupiter-api', version: '5.0.0-RC2'
    testCompile group: 'org.junit.platform', name: 'junit-platform-launcher', version: '1.0.0-RC2'
    testRuntime group: 'org.junit.jupiter', name: 'junit-jupiter-engine', version: '5.0.0-RC2'
    testRuntime group: 'org.junit.vintage', name: 'junit-vintage-engine', version: '4.12.0-RC2'
}

AssertTest.java

package pftester;

import org.junit.jupiter.api.Test;

import java.util.Arrays;
import java.util.List;

import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.*;

class AssertTest {
    private final String expected = "Hello";
    private final String sameAsExpected = "Hello";
    private final String actual = "HELLO";
    private final String different = "Goodbye";

    @Test
    void assertEqualsStringComparisonTest() {
        assertAll("JUnit assertEquals String comparisons",
                () -> assertEquals(expected,expected, "Same objects should never fail"),
                () -> assertEquals(expected, sameAsExpected, "Different Strings with same content also pass."),
                () -> assertEquals(expected, actual, "Strings differing in case should fail."),
                () -> assertEquals(expected, different, "Strings differing in content should always fail.")
        );
    }

    @Test
    void assertLinesMatchStringComparisonTest() {
        final List<String> expectedList = Arrays.asList(expected, expected, expected, expected);
        final List<String> actualList = Arrays.asList(expected, sameAsExpected, actual, different);

        assertLinesMatch(expectedList, actualList);
    }

    @Test
    void assertTrueIgnoreCaseStringComparisonTest() {
        assertAll("JUnit assertTrue (equalsIgnoreCase(..)) String comparisons",
                () -> assertTrue(expected.equalsIgnoreCase(expected), "Same objects should never fail"),
                () -> assertTrue(expected.equalsIgnoreCase(sameAsExpected), "Different Strings with same content should pass."),
                () -> assertTrue(expected.equalsIgnoreCase(actual), "Strings differing in case should not fail now."),
                () -> assertTrue(expected.equalsIgnoreCase(different), "Strings differing should always fail.")
        );
    }

    @Test
    void assertTrueIgnoreCaseWithExplanationStringComparisonTest() {
        assertAll("JUnit assertTrue (equalsIgnoreCase(..)) String comparisons with explanation",
                () -> assertTrueIgnoreCaseWithExplanation("Same objects",expected, expected),
                () -> assertTrueIgnoreCaseWithExplanation("Same contents", expected, sameAsExpected),
                () -> assertTrueIgnoreCaseWithExplanation("Different Case", expected, actual),
                () -> assertTrueIgnoreCaseWithExplanation("Different Words", expected, different)
        );
    }

    private void assertTrueIgnoreCaseWithExplanation(final String description, final String expectedString, final String actualString){
        assertTrue(expectedString.equalsIgnoreCase(actualString),
                () ->  String.format("%1$s -> The expected is: '%2$s' while the actual is: '%3$s'", description, expectedString, actualString));

    }

    @Test
    void assertjTest() {
        assertAll("String comparison using Joel Costigliola's assertj library.",
                () -> assertThat(expected).isEqualToIgnoringCase(expected),
                () -> assertThat(sameAsExpected).isEqualToIgnoringCase(expected),
                () -> assertThat(actual).isEqualToIgnoringCase(expected),
                () -> assertThat(different).isEqualToIgnoringCase(expected)
                );
    }
}

That’s all for now. See you next week!
Alexander

Comments are closed.