Don’t be that developer who is woken up in the middle of the night because of some problem with the web application. After all, you need your beauty sleep – some of us more than others. The best way to avoid problems with your application is to test thoroughly. Of course, there is a cost to testing, and it is easy to jump in too deeply such that every piece of code is outrageously well tested. Finding the right balance of just what to test and which ASP.NET Core testing tools to use is something that comes with experience.
Testing simple code that has a low impact too thoroughly is not as bad as failing to test complex code with a high impact, but it will still have a negative impact on the success of your project. Complicating the problem is that we have a number of ways to test code. You may be familiar with the idea of the testing triangle or, for those who think a bit higher dimensionally, the testing pyramid.
The purpose of the triangle is to demonstrate the number of tests you should have of each type. The bulk of your tests should be unit tests: these are tests which test a single public method on a class. As you move up the testing triangle the number of tests at that level decrease; conversely the scope of the tests increase. The cost of an individual test increases towards the top of the triangle.
In this article, we’ll look at testing tools which can help us out on each level of the pyramid. Some of these tools will be specific to .NET, but others will be portable to almost any development platform, and we’ll call out which is which as we progress. You will notice that as we increase the level of abstraction by moving up the triangle, the testing technologies become broader and applicable to more web development technologies.
The testing triangle is a good guide for your ASP.NET core testing strategies.
Unit tests are the smallest sort of tests that you’ll write. Ideally, they exercise a single method and should be trivial to write. In fact, many people suggest that if you find unit tests difficult to write then it is an indication that the code being tested is doing too much and should be split up. I use unit tests as an indicator that methods should be split up into multiple methods or even split into separate classes. These are the sorts of tests you should create during test-driven development.
Unit testing tools are not new in .NET or in any modern language. The migration to .NET Core has brought with it most of the first-class unit testing tools. Unit testing tools are divided into a number of categories: test frameworks, test runners and assertion libraries. The frameworks are a set of attributes that allow you to decorate your code such that they can be found and run.
Typically the testing frameworks also include a runner which will run the tests without starting up your entire application. Finally, there are assertion libraries that are used to check the conditions inside a test – most people stick with the assertions provided by their testing frameworks. All of these tools can run against any .NET project, not just ASP.NET.
My personal favourite testing framework is xUnit, which is now part of the open source .NET Foundation. This ensures that it will have a long life and is well recognized in the community. xUnit is a fairly opinionated framework that spends a lot of effort on being fast. Its terminology is slightly different from what you might have seen in the past, substituting terms like Fact and Theory in place of Test and ParameterizedTest. During the incubation of .NET Core, xUnit was the only unit testing framework that kept up with the constant betas.
Recently Microsoft open sourced their MSTest framework as v2. MSTest has been around for years and has been sorely in need of updating for much of that time. However, the new version is quite good and remarkably fast. Even two years ago I would not have even considered MSTest over xUnit, but it is now quite competitive.
Erik has an excellent post on this very blog doing a deeper comparison of several .NET unit testing frameworks.
Tests Runners execute the tests in your suite and present the results in an understandable format. If you’re using Visual Studio then there is no need to look any further than the built-in unit test runner. It works very well for ASP.NET core testing.
This part of the tooling has received a lot of love from the team over the last few releases. The speed of running tests and of test discovery is remarkably better. In addition to the standard runner, there are now live unit tests. This tooling runs your tests continuously as you write your code to tighten up the feedback loop between writing code and getting feedback.
Command-line tooling for running tests is also excellent on .NET Core. Tests can be run as easily as running dotnet test. The official documentation has a thorough entry on running command-line tests, and this approach is suitable for running on a build server. xUnit will actually detect that is running in a continuous integration (CI) environment and alter its output format to one that the CI server can parse.
Almost everybody makes use of the built-in assertion libraries that come with the testing frameworks. However if, like me, you’re particular about the way your assertions are constructed there are some really nice alternatives. Shouldly and Fluent Assertions provide a more BDD style of assertion. What does that mean? Instead of writing
Assert.That(location.X, Is.EqualTo(1337));
we can write the much more legible
location.X.ShouldBe(1337);
The error messages produced by these libraries are also much easier to read, saving you time tracking down testing failures.
Because these tools run on the .NET framework, they can be used to test F# code as well, just as F# code can be used to test C# code. There are some very nice testing tools in the F# space which, if you’re feeling adventurous, are worth looking at in more detail. F# for Fun and Profit have a wonderful series of entries on F# and unit testing.
Integration tests are larger than unit tests and typically cross over the boundaries between modules. What is a module you might ask: that’s a great question.
A module could be as small as a single class, so an integration test could be as small as just exercising the interactions between two classes. Conversely, a module may cross process boundaries, and be an integration between a piece of code and a database server.
While the catchword for unit tests is speed (so you can run them rapidly on your development box without interrupting your flow), the catchword for integration tests is parallelization. Each test is inherently going to take a long time, so to compensate we find something for all those dozens of cores you can get on your machine these days to do.
More often than not, the integration tests on a web project will involve submitting requests to the web server and seeing what comes back. Previously integration tests of this sort have been quite tricky to write. For ASP.NET Core testing, the situation has been improved with the introduction of the TestServer. This server allows you to submit requests to an in-memory HTTP server. This is much faster and provides a more realistic representation of what a fully-fledged server would return. You can read a much more in-depth article on integration testing at ASP.NET Core in the official docs.
One handy tool for integration tests which examine the HTML returned from an endpoint is AngleSharp. AngleSharp provides an API for parsing and exploring the DOM which makes checking attributes of the returned HTML a snap.
In the past, I’ve written some pretty impressive pieces of code that stand up and tear down databases against which integration tests can be run. Although it was a pretty snazzy piece of code at the time, the speed at which the tests could run was quite limited. Fortunately, we’ve moved a little bit out of the dark ages with Entity Framework Core. Just like the TestServer, EF Core provides an in-memory implementation of a database. This database can be used to test LINQ based queries with great rapidity. The one shortcoming of this approach is that queries written in SQL for performance or clarity reasons cannot be tested with the in-memory implementation. In those cases, I recommend reading Dave Paquette’s article on integration testing with EF Core and full SQL Server.
Acceptance tests cross-module boundaries like integration tests, but they are written from a user’s point of view. These tests are usually written in a way that describes the behaviour of the system rather than the function. By this, I mean that your tests simulate a user’s experience with the application.
In .NET are two major flavours of tools that facilitate these tests: SpecFlow and NSpec. The primary difference between these tools is their approach to describing the tests. SpecFlow uses the Gherkin language from the Cucumber tool language to describe tests. NSpec uses its own dialect to describe similar tests. Either of these tools will do a good job building tests, but I find SpecFlow to be a more usable tool; your mileage may vary.
The highest level of automated tests are UI tests. These tests actually drive the web browser in an automated fashion performing mouse clicks, typing text, and clicking links. In the .NET space, I actually like Canopy more than any other. In my experience, UI tests tend to be incredibly fragile and inconsistent. I’d like to believe that it is just my own incompetence that leads them to be such, but in discussions with other people I’ve yet to uncover any counterexamples. Much of the issue is that web browsers are simply not designed for automation and are missing hooks to program against. Typically testing ASP.NET web apps becomes an exercise in adding sleeps in the hopes that the UI has updated itself. Recently I’ve been hearing good things about a new tool called Cypress, but I’ve yet to use it. If you have I’d love to hear about it in the comments.
UI test tools are portable to any web application and not limited to ASP.NET.
Manual testing is often seen as terrible drudgery. A lot of companies try to avoid manual testing in favor of cheaper automated tests. Automated tests cannot tell you what the user experience of your website is like. It is important to get a real variety of typical users to look at your site. The problems that manual testers are able to find are different from the ones you will find using unit, integration, or any other test.
ASP.NET Core is really, really fast. The benchmarks for it place it in the top two or three web frameworks. This is a very impressive accomplishment considering that the team started their performance optimizations near the bottom of the list. However, as performant as the framework is, its performance can easily be ruined by running poor code on top of it. There are two classes of tests which can be run in this space: low-level and load tests.
Low-level performance tests answer questions like “what is the quickest way to deserialize this data stream?”, or “will heavy load in this section of the code cause too many memory allocations?”. These questions are rarely asked and for most applications that simply retrieve data from the database and display it, they don’t matter a great deal. But as soon as your code starts doing anything computationally expensive or memory intensive, they become important. The most popular tool in this space, and the one which you’ll frequently see used by the .NET Framework team itself is BenchmarkDotNet.
This ASP.NET Core testing tool provides attributes and a test runner, not unlike xUnit, but focused on testing. The output looks something like
As you can see in this example, the code is tested on a number of platforms with varying sample sizes. These tests can be used to find issues and then kept to ensure that there aren’t any regressions in performance as your application matures.
The best thing that could happen to your web application is that it becomes very popular and people flock to it; it can also be the worst thing. Knowing how your application performs under load will help keep your nights and weekends uninterrupted. The best tool in this space and one that I’ve seen recommended even by those who won’t otherwise touch the Microsoft stack is Visual Studio Team Services’ load testing tool. Using the power of the millions of machines in Azure VSTS is able to spin up very high-scale load tests that can stress your website far beyond what any desktop tool could. It is priced very reasonably and I highly recommend it.
VSTS load testing uses Azure for testing, consequently it scales to the moon
Both load testing and low-level tests are great tools for gauging the performance of your application before launching it. However, nobody can predict what actual user load on your website might look like. It is really useful to have a tool like Retrace which is able to drill into the performance of your live site highlighting slow queries, code bottlenecks and hidden errors.
During development Prefix, the sister tool to Retrace is invaluable at showing a high level overview of application performance. Load testing will only tell you that your site is slow while Prefix will give you the why. The price for Prefix (free!) is certainly attractive too.
Accessibility testing unlocks your application to more people. Your users may be blind, have limited mobility, or be color blind, and your site should be usable by all of them. There are as many tools in this space as there are different impairments. This article is too short to do a full analysis, but others have done so.
Decision fatigue is a real problem in the modern world and the variety of testing tools available to you will not make the problem any better. I hope that this article will at least point you in a direction that resembles success. If you’re testing ASP.NET Core at all, then that is a huge win, since any tests are better than none. Go forth and testify!
Stackify's APM tools are used by thousands of .NET, Java, PHP, Node.js, Python, & Ruby developers all over the world.
Explore Retrace's product features to learn more.
If you would like to be a guest contributor to the Stackify blog please reach out to stackify@stackify.com
