Since my last post I have managed to take my prototype Webdriver factory for .NET Core projects and package it up on nuget.org. This was not as simple a process as I expected, and has required a lot of additional thought and testing. I learnt a number of things along the way that I would want to be aware of if starting again: I try to fill my blog with posts about things I wish I had known, so this is an attempt to list and explain the most important things I wish I had known before starting.

Apologies for the long gap since my last post, it has been an insanely busy period for us at Altitude Angel leading up to the highly successful Operation Zenith demonstration at Manchester Airport last month. Such success is built on months of preparation, planning, testing and hard work by the whole team so I am immensely proud of what we achieved there.

I am not going to go through the process of creating nuget packages here, there are plenty of good guides out there already. I suggest this blog post by Vibhav Churi and this Microsoft documentation as good places to start.

Lesson 1: Reflection does not behave the same way from a packaged library as an included project.

This was a major surprise to me (although it probably shouldn’t have been) and actually took away a big part of the intention of my package. If you recall I was using

private static string DriverPath => Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);

from inside my static factory to find the driver path for the WebDriver constructor.

Having got everything working well and tested it on both Windows and Linux I felt safe to package it up and publish. So I created my pre release package, uploaded it to https://www.nuget.org/ and set about testing by pulling from there instead of within the project. To my shock this was failing to find the drivers.

Solution:

After much investigation including all the different forms of reflection I concluded that this did not have a valid workaround for my static factory: The best I could do was to create an instance factory and require the calling project to pass in the driver path.

I was already intending to provide an interface and instance factory, but requiring the user to know the reflection call to make and pass in is disappointing, the best I could do was add this to the documentation.

In all honesty if I had known this in advance I would not have even bothered to try.

 

Lesson 2: Once uploaded on nuget.org, even a pre-release library may be downloaded and used by others.

Having had unplaced confidence in my testing I was somewhat shocked when I realised that a library that I knew was not fit for purpose (i.e. did not work at all) with my name on was out there.

When I realised that downloads by others were happening, this put me under a lot of pressure to fix the problem quickly: Never a nice place to be.

Solution:

It is perfectly possible to set up your own nuget repository locally, and you should totally do this.

Be warned though, setting up multiple environments on different OSes to connect and use this repository is a painful necessity.

 

Lesson 3: Testing on multiple OSes requires hardware and considerable software configuration.

As previously discussed, it is possible to run Selenium tests against an X server running on the Windows Subsystem for Linux (WSL) but I would not recommend it.

Also having already killed one trusty old Mac Mini in the summer, I also started out with no way to test on MacOS at all. Yes I could just have left it out there and waited for someone to bother to report back, but as a tester at heart that was never going to be good enough for me.

I had previously set up a working Selenium grid with Windows, Linux and MacOS nodes for my Java project. Getting the project building and running tests from the command line was not so hard. It seems that trying to run a graphical IDE however is somewhat more of a challenge.

Visual Studio 2017 is only available for Windows leaving me with a choice of Visual Studio Code or JetBrains Rider for working on MacOS and Linux. It seems that both however are quite resource hungry and running them to debug on a machine with 4GB RAM and spinning disks is painfully slow. (I guess I am too spoilt with modern many thread CPUs and SSDs!)

Solution:

After exploring I settled with using JetBrains Rider for development work on MacOS and Linux.

I should probably should investigate remote debugging from Windows if I aim to do much more testing around this.

 

Lesson 4: Setting up free cloud Continuous Integration / Continuous Delivery has its limitations

Having written a suite of tests for 3 supported OSes it would be great if everything would just build and test for me when I change things. Unfortunately I don’t have all of that working just yet. Maybe one day I’ll go the whole hog and set up a complete multi OS build and deploy solution at home, but life’s too short so in the meantime what can I do?

Solution:

I have set up a Travis-CI pipeline that will attempt to build all the products and run a handful of Linux tests with every pull request.

For now at least, full cross platform testing will have to remain a manual process before each release and the final stage of Delivery (i.e. uploading a package to nuget.org) remains a manual process. I don’t anticipate making changes too often so the 30 minutes or so to run the tests and deploy is not so big a deal. I would have to make a lot of releases before the time taken to set up fully automated pipeline would be paid back in savings.

 

Progress made:

• Created and released my WebDriverFactory on nuget.org.
• Refactored and updated to version 2.0.0 currently as a release candidate. (Please do have a try with it: Instructions in my next post!)
• Created a full dev/test environment. Windows, Linux and Mac development environments and Selenium grid with all supported nodes.

 

Lessons learnt:

• Cross platform .NET Core development requires a lot of work to properly test across even one instance of each platform.
• Always test packages intalled from a local nuget repository before publishing to nuget.org.
• It is not feasible to set up a complete System test run on a free cloud CI platform. I do however have a few simple smoke tests running that should pick up the most obvious problems.

 

A reminder:

 

If you want to ask me a question, Twitter (@AlexanderOnTest) is undoubtedly the fastest place to get a response. My DMs are always open for questions, and I publicise my new blog posts there too.

 

The code for this series of posts is available on GitHub. At the time of writing, the master branch is at version 2.0.0 which has just been released. Further detail in my next post.