It’s not much help failing unless they tell you what went wrong.

A while back I wrote an article about the challenges of authentication for Automated API tests. At work we have also recently started rolling out the first API tests on a new framework with all new Authorisation code. The rollout went fine until literally the last production environment and then bam: The tests failed throwing a random de-serialisation exception.

Worse still, the exception was thrown in a library without published symbols so I couldn’t even quickly debug to identify the issue. To cut a long story short, it was a painful experience working out where to add some debugging code to ensure that if it happens again, we can see in the test report what happened.

The moral of the story

I’m sure you are used to the concept of checking that your tests fail when the data they return is wrong.

But do you ever think to deliberately make the setup fail and check what happens?

Expect the unexpected:

Of course this left me with a number of options:

1. Throw an Exception: Given that the error was occurring inside a library package, this seemed to be the most appropriate approach, ensuring to include the failed api response in the exception message.
2. Log to Console: Every CI pipeline that I have used has an easily accessible console log of the release or build concerned. Unfortunately trawling through these logs to find the failure is a pain. I really don’t like this.
3. (NUnit) TestContext.Writeline(): This is my preferred logging approach. This log is linked and visible in Visual Studio and added as an attachment in Azure DevOps. I like to be able to identify straight from the test failure log exactly what went wrong.

Final thoughts:

Dear Tests,

It’s not much help failing unless you tell me what went wrong.

Thank you

Me, last week.

I consider it a failure if I have to test from the IDE to identify the problem.

Progress made:

• Debugged and fixed a number of issues in my blog caused by updates whilst I’ve been busy.
• Written my first blog post since moving in to our new home. Its hard to get back into writing!

Lessons learnt:

• When building out a new framework, you are going to find unexpected modes of failure, that’s fine. Just be sure to go and add the code to tell you what happened.
• Despite everything I’ve written here: Adding more tests that run and pass or fail as required is always great start; Don’t let the quest for perfection stop you getting them out there.

A reminder:

If you want to ask me a question, Twitter is undoubtedly the fastest place to get a response: My Username is @AlexanderOnTest so I am easy to find. My DMs are always open for questions, and I publicise my new blog posts there too.

Series Navigation

<< It’s all about the “Auth”