I am proud of UnitTest11. I believe I was one of the first to get out an indication of what a Modern C++ testing and mocking library could look like, and I also believe what I have came up with is a decent attempt, though I haven’t really marketed or pushed it. So why haven’t I committed any more updates in the last few months, despite there being useful features in the issues backlog I’ve wanted to add? Well, because in all honesty I haven’t been using UnitTest11. I have been programming in C++, and I have been carrying out TDD in C++, but I have not been using UnitTest11. I have been using bandit.
Bandit is a great little testing framework that uses C++11 features, and the Snowhouse assertion framework, to allow for very human readable tests. It is easy to use, and has comparative features to UnitTest11. It is has a few things UT11 lacks, such as multiple reporting formats, and is missing a few things UT11 possesses, such as a packaged mocking framework, but it is a joy to use. Why? Because it has less boilerplate than UnitTest11.
Setting up a TestFixture in my library requires a certain amount of boiler plate code. You need to compile and link the library, write a class that inherits from the TestFixture base class and implements the abstract Run function, and then you need to declare this TestFixture via a macro. Bandit is header-only, so requires no compilation/linking, and the ‘test fixture declaration’ and running logic is done in a single macro and with a series of functions in the bandit namespace. That boilerplate in UnitTest11 is an aspect of the library that annoys me as a developer. So why this boilerplate? Well, the reason comes down to two design decisions that Bandit and UnitTest11 differ on.
The first is the decision that the running of a test fixture constantly recalls the Given-When-Then-Finally functions for every Then test. This forces every test run to establish a clean slate, and encourages each Given-When-Then-Finally step to be as self-contained as possible. Bandit does not do this, and instead it runs each describe/it as they are encountered. after_each and before_each must be used to setup/teardown test fixtures, meaning it must be enforced by the developer instead of being a core part of the testing approach.
The second is the decision for Finally to be declared at the end of a Given-When-Then block. This means the system could not possible run those blocks until it reaches the end of the Run function. The after_each in bandit must be declared before the relevant describe/it functions, allowing for it to be ran as they are encountered.
These design decisions mean the lambda-heavy single-macro approach took by bandit could not be done for UnitTest11 unless these design decisions were completely reversed. A smarter man than me may come up with a solution. If they share it with me or fork the library I would gladly merge the feature in. I have wrestled with this for some time and not found a solution.
However, there is one feature of UnitTest11 I am rather proud of. It’s integrated mocking. And in my latest little side-project, I have rewritten it from the ground up for use with bandit (naturally, this rewrite is tested using the bandit library). When it has ‘cooked’ for a bit, I will most likely completely break this functionality out and make a specific git repository to share it. Until then, I encourage everybody reading this to check out the bandit library, to play around with it, and to recommend any good C++ mocking libraries to me.
Until now, UnitTest11 could only be built on GCC. I recently fixed an issue with the less-clever type deduction on Clang to allow Clang building. But the Visual Studio 11 C++ compiler hasn’t been able to handle the library for a very simple reason: Variadic templates.
I love ‘em, they’re a very useful feature of C++11. UnitTest11 uses them in non-trivial ways that would be difficult to write a fake wrapper around. Microsoft tried to get them into VS2012 and failed, finding them too difficult. They proceeded to release the November CTP for 2012 which did have Variadic Templates. But also had a lot of bugs related to Variadic Templates that made them unusable.
And now the Visual Stuio 2013 Beta is out, and the Visual Studio 12 C++ compiler has fixed those bugs and made Variadic Templates usable.
That’s not to say it compiled out of the box with VS12. I wish. Instead, there were two obstacles to compilation. One of these was trivial, the other took a full day of Googling and experimentation before I fixed them.
Obstacle 1: Constexpr. I also love constexpr, it simplified the template metamagic of yesteryear considerably (of course UT11 still features a fair share of Template Metamagic). And unfortunately even Visual Studio 2013 doesn’t support constexpr. So the solution was to remove the usages of constexpr, and to go back to the enum values. The usages of constexpr in UT11 were trivial to revert so this proved no issue.
Obstacle 2: Macros. I dislike using macros for the obvious reasons, but there is a limit to how much I could accomplish without them. If only so failed assertions knew the line they failed on, macros had to go in. Visual Studio does not have the same extent of macro support that GCC has by a long shot. I had to introduce a couple of extra layers to some of the macros to get it to correctly resolve several macros to the desired forms.
For validating the macros functioned as expected the ability of Visual Studio to output the C++ code after preprocessing was applied did prove very useful for this, so kudos to Microsoft for that compiler flag.
However, this was not the main problem encountered. The main macro problem encountered was the landmine of variadic macros. They’re in the standard, and have been for awhile. And Visual Studio has had a bug related to them for about as long. The compiler will, if it gets passed down to another macro, treat __VA_ARGS__ as a single parameter and not correctly evaluate it to however many parameters it contained. Since a lot of the macros I use in UT11 rely on variadic macros heavily, and this bug broke every single such macro, this bug is a bad thing.
Fortunately, I found a workaround. It is, to date, the only piece of code in UT11 that changes for different compilers. I’m unhappy about that, but it’s the best solution I found.
You see, whilst the bug causes __VA_ARGS__ to get parsed as a single parameter in the macro substitution, if the macro isn’t recognised on that pass as a macro by Visual Studio, it appears to perform the substitution and then recognise the macro on the second pass.
This is good, as it results in the expected processed code. Indeed, the fix should also compile without error in other standards compliant compilers. However, it’s not a pretty fix and so if it somehow gets fixed before the official VS2013 release I’d remove the workaround with a smile on my face.
With this workaround in place, compilation in Visual studio is possible. And I’d love to say the tests for UT11 all pass. Well, I can’t. The beta of 2013 has an annoying bug with objects and initialiser lists, where it appears to call the destructor before the move, resulting in a dead object being added to the container. A frustrating bug, to say the least, since it breaks the recently added multiple-category support.
Since having a fixture be recognised as under multiple categories relied on passing an initialiser list of the Category type to the DeclareFixture function, this is understandably a bad thing. But the code compiles and the failing tests demonstrate the broken functionality and prove their worth.
This bug is fixed in visual studio bug tracker, suggesting a future update of the Beta or the released version will function as expected, and despite these teething problems I find myself surprisingly impressed with Visual Studio 2013. But I am still disappointed it’s not caught up with GCC and Clang yet in terms of C++11 compiler support.
It’s often said that a downside of Test Driven Development is it often forces you to expose information related to a class that would of been private if written without Test Driven Development. I wish to provide some food for thought and to tell you this is not true. In fact, if you feel you are being forced to compromise design for testing, it is not the fault of the testing.
Yes, I am telling you that if you are doing this, you are doing it wrong. Having to expose internals for testing is a clear code smell, and a potent one at that. The solution is, instead, to stop and think. Why is the structure of the code forcing you to make this decision? Why are the responsibilities of this class so encompassing that I cannot tell what it is doing as an outside observer? What have I done wrong?
Exposing internals for testing is a code smell, and in a lot of those situations dependency injection is the detergent. If a classes responsibilities are too big (which is fundamentally what this code smell suggests), then breaking those responsibilities down is the most responsible thing the programmer can do.
Quote of the Day
“I never make stupid mistakes. Only very, very clever ones. ”
Merry Christmas and Happy New Year to all! Hope you’ve had a blast over the holidays. I know I have. I went to my first Company Christmas Shindig. It was great, I got to wear a dinner suit and drink from an open bar. What more could I want?
Now, on to the coding! Lately I’ve been using a lot of C#. It’s what we predominately use at work, which explains that one. Naturally, we try to deliver the best quality product possible, and one of the techniques we employ to do so is test driven development. To accomplish this we largely use NUnit as our testing framework and Moq for mocking out implementations of interfaces. And on average this works very well.
Now in my spare time I like to occasionally code. I am not willing to give up a hobby just because it’s also a career. Now, C# is great and all but I like to spread my wings, stretch my legs and explore other languages. And most modern languages have pretty good support for TDD. C++ isn’t a modern language. In programming terms it’s steampunk, Victorian fashions realised with modern designs. It’s unit testing and mocking libraries are not quite so easy to use in a TDD context.
I tried with quite a few, and had some reasonable success with UnitTest++ combined with Google Mock. But they never quite matched my needs or expectations, unfortunately. And as the old saying goes, if your wheel is difficult to attach to your car, build a new wheel. Then burn the car down with it. It’s an interesting saying. And it led to the creation of my latest little project, UnitTest11.
UnitTest11 is a testing and mocking framework written in C++11 to take advantage of the latest features provided by the language. It aims to involve less Macros than previous testing frameworks, and allow for code to be written in a more modern style. It’s based inherently around the concepts of “Given, When and Then” (Arrange, Act and Assert), and allows for tests to be expressed in this BDD-like fashion without always requiring you to be encumbered by the strictness of Behaviour-Driven Development.
The actual assertions are written to mimic the “natural language” constraint model of NUnit, namely they adopt an “AssertThat(Value, Predicate)” format, to allow for greater readability and flexibility.
The mocking framework is designed to allow for verifications to be made after the act, as assertions. This is something many other C++ mocking libraries, such as Google Mock, do not (to the best of my knowledge) provide good or even any support for. And since both are built into one library, they integrate with less discrepancies compared to when the Testing and Mocking libraries are unrelated.
At present if you wish to look at the library, feel free to do so (For the curious, the code is under the MIT licence). I’ve not completely finished development of it, the tests driving the library are still a mixture of the new UnitTest11 style and older UnitTest++ tests used to bootstrap the development. Also not every feature is fully implemented. For example, whilst the actual Mocking classes are present support for quickly and easily creating Mock Classes is still not fully implemented. I’m looking into the best way to accomplish this, but fear the Google Mock way of having a series of Macros for varying numbers of parameters may be the only viable option. That makes me a little sad.
Still, it works and I would hope it effectively proves the concept. And if Unit Testing and TDD in C++ interest you, feel free to check it out and let me know what you think. I’ll most likely post again about it, with actual code examples, sometime in the near future.
Quote of the Day
“Cease dependence on mass inspection to achieve quality. Improve the process and build quality into the product in the first place.”
William Edwards Deming
Well, that’s it. My fears confirmed, time management has killed me. There is no way I’m getting this done in time. I spent too long on the background under-the-hood code, which maybe related to how that’s the part of coding I find the most fun. Which could also explain why I’m on the Platform Team at work. We make the API and let the Application Team worry about the best shade of Cornflower Blue to show the customers. And that’s what I ended with, the start of what could be a solid API but something only tangentially related to a working game.
On the brightside my code is very modular, and I am definitely going to create something properly using a more refined variation on this design at a later date (and the lack of time limit will let me use proper TDD and the like).
If I take part in the next LD, I’ve learnt a lot here and had fun, even if I have nothing to demo by the end of it but a sprite moving around the screen. I learnt from my mistakes and will hopefully have more to show in the future.
Major respect for the other LDers who actually get something done without getting lost in background implementation details, as I did.
If I could do it again, I’d pick a pre-existing “engine” (hate that term) instead of writing my own. I just can’t write fast and hackish code without feeling unclean, but for rolling things yourself I’m not experienced enough to do anything but in the timeframe. I fought against that, and it took me too long.
Also I’d impose a much stricter “sprint” system, planning out from the start goals on an hour-by-hour basis (using hourly iterations) with a set of things that needed doing I could pull from. I tried this by the end, where I started to pick up the pace, but it was too little too late.
Code by the end is on github for the curious.https://github.com/MorleyDev/ld24Evolution
tl;dr – I failed hard, but had a blast doing so.