Saturday, December 13, 2014

Introducing the Trompeloeil C++ Mocking framework

Trompeloeil is a new mocking framework for C++, aimed at ease of use without sacrificing expressive power.

In arts, trompeloeil is intended to mock your mind, making you believe you see something that isn't what it appears to be. In unit tests, we use use mocks to fool the unit under test, so that we can break dependencies and test small pieces in isolation.

Trompeloeil requires a reasonably C++14 compliant compiler (gcc-4.9 and clang-3.5 are known to work.) It is a single header file only, so there is no need to worry about compiler-flag compatibility issues between the test code and the framework. Trompeloeil is released under the BOOST SOFTWARE LICENSE 1.0.

Simple Example

Let's dive in to mocking with Trompeloeil by writing tests for the order class of a warehouse, following the example in Martin Fowler's post Mocks Aren't Stubs.

The order class looks like below:
1
2
3
4
5
6
7
class Order
{
public:
  Order(const std::string& name, size_t amount);
  void fill(Warehouse& w);
  bool is_filled() const;
};
Warehouse in an interface looking like this:
1
2
3
4
5
6
class Warehouse
{
public:
  virtual bool hasInventory(const std::string& name, size_t amount) const = 0;
  virtual void remove(const std::string& name, size_t amount) = 0;
};
From the interface, a mock class can be created. The mechanism is nearly identical to that of GoogleMock.
1
2
3
4
5
6
class WarehouseMock : public Warehouse
{
public:
  MAKE_CONST_MOCK2(hasInventory, bool(const std::string&, size_t));
  MAKE_MOCK2(remove, void(const std::string&, size_t));
};
MAKE_MOCKn creates a mock implementation of a non-const member function with n parameters, and MAKE_CONST_MOCKn creates a mock implementation of a const member function. The first parameter of each macro is the name of the member function, and the second parameter is the function signature. Trompeloeil supports up to 15 parameters in a function signature.

From this we can write a first simple test for the Order class.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
TEST(filling_removes_inventory_if_in_stock)
{
  Order order("Talisker", 50);

  WarehouseMock warehouse;
  {
    REQUIRE_CALL(warehouse, hasInventory("Talisker", 50))
      .RETURN(true);

    REQUIRE_CALL(warehouse, remove("Talisker", 50));

    order.fill(warehouse);
  }

  ASSERT_TRUE(order.is_filled());
}
REQUIRE_CALL on lines 7 and 10 are what they sound like. For each, a matching call is required exactly once. The requirement must be met by the end of the surrounding scope, or a violation is reported.

If several REQUIRE_CALL could match the same call, they are tried in reversed order, until a match is found. This allows you to declare an open default early in your test, and add restricted specializations later when needed.

A weakness in the above, however, is that no sequencing is implied. The two REQUIRE_CALLs are considered logically parallel, i.e. the order of the calls is of no significance to the outcome of the  test. To ensure that the test passes only if hasInventory is called before remove, a sequence object is introduced:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
TEST(filling_removes_inventory_if_in_stock)
{
  Order order("Talisker", 50);

  WarehouseMock warehouse;
  {
    trompeloeil::sequence seq;
    REQUIRE_CALL(warehouse, hasInventory("Talisker", 50))
      .IN_SEQUENCE(seq)
      .RETURN(true);

    REQUIRE_CALL(warehouse, remove("Talisker", 50))
      .IN_SEQUENCE(seq);

    order.fill(warehouse);
  }

  ASSERT_TRUE(order.is_filled());
}
Now the test will fail if remove is called before hasInventory.

Let's add another test for trying to order more than is available in inventory:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
TEST(filling_does_not_remove_if_not_in_stock)
{
  Order order("Talisker", 51);

  WarehouseMock warehouse;
  {
    REQUIRE_CALL(warehouse, hasInventory("Talisker", 51))
      .RETURN(false);

    order.fill(warehouse);
  }

  ASSERT_FALSE(order.is_filled());
}


Expectations and Fixtures

This is straight forward, but there is a problem with repetition between the two tests. Let's refactor the code by breaking out the common code.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
using trompeloeil::_;

const auto talisker = "Talisker";

struct TaliskerStore
{
  size_t                stock;
  trompeloeil::sequence seq;
  WarehouseMock         mock;
  std::unique_ptr<trompeloeil::expectation> inventory
    = NAMED_REQUIRE_CALL(mock, hasInventory(talisker, _))
      .IN_SEQUENCE(seq)
      .RETURN(_2 <= stock);
};

TEST(filling_removes_inventory_if_in_stock)
{
  Order order(talisker, 50);
  {
    TaliskerStore warehouse{50};

    REQUIRE_CALL(warehouse.mock, remove(talisker, 50))
      .IN_SEQUENCE(warehouse.seq);

    order.fill(warehouse.mock);
  }
  ASSERT_TRUE(order.is_filled());
}

TEST(filling_does_not_remove_if_not_in_stock)
{
  Order order(talisker, 51);
  {
    TaliskerStore warehouse{50};

    order.fill(warehouse.mock);
  }
  ASSERT_FALSE(order.is_filled());
}
trompeloeil::_ on line 1 is a wildcard used in REQUIRE_CALL when any value is acceptable.

Since REQUIRE_CALL must  be met by the end of the surrounding scope, it works poorly in fixtures. NAMED_REQUIRE_CALL creates an expectation object to be held by a std::unique_ptr<trompeloeil::expectation>. This is seen on lines 10-13. The call must be met by the time the object is destroyed.

On line 13, the _2 refers to the second parameter of the function call (the one matched with the wildcard,) so the return value will be true if the amount asked for is available in stock, and false otherwise.

Mocking with side effects

An alternative rewrite is to move more of the test logic into the fixture, making it behave like a reasonable test store, implemented in terms of the mock.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
using trompeloeil::_;

const auto talisker = "Talisker";

struct TaliskerStore
{
  using named_expectation = std::unique_ptr<trompeloeil::expectation>;

  size_t                stock;
  trompeloeil::sequence seq;
  WarehouseMock         mock;
  named_expectation talisker_inventory
    = NAMED_REQUIRE_CALL(mock, hasInventory(talisker, _))
      .IN_SEQUENCE(seq)
      .TIMES(AT_LEAST(1))
      .LR_RETURN(_2 <= stock);
  named_expectation remove
    = NAMED_ALLOW_CALL(mock, remove(talisker, _))
      .IN_SEQUENCE(seq)
      .LR_WITH(_2 <= stock)
      .LR_SIDE_EFFECT(stock -= _2);
};

TEST(filling_removes_inventory_if_in_stock)
{
  Order order(talisker, 50);
  {
    TaliskerStore warehouse{50};

    order.fill(warehouse.mock);

    ASSERT_TRUE(warehouse.stock == 0U);
  }
  ASSERT_TRUE(order.is_filled());
}

TEST(filling_does_not_remove_if_not_in_stock)
{
  Order order(talisker, 51);
  {
    TaliskerStore warehouse{50};

    order.fill(warehouse.mock);

    ASSERT_TRUE(warehouse.stock == 50U);
  }
  ASSERT_FALSE(order.is_filled());
}
.TIMES(AT_LEAST(1)) on line 15 alters the default of requiring exactly 1 matching call. You can use .TIMES(2) to require exactly 2 matching calls, .TIMES(2,5) to require 2 through 5 matching calls. Naturally, there is also an AT_MOST(n) that can be used.

The .LR_RETURN(_2 <= stock) on line 16 looks a bit odd. .WITH(), .SIDE_EFFECT(), .RETURN() and .THROW() all refers to copies of local names, whereas the LR_-prefixed versions accesses them by reference. So, a .RETURN(_2 <= stock) would always compare _2 with the value stock had when the expectation object was created, whereas .LR_RETURN(_2 <= stock) compares _2 with the value the member variable stock has at the time of the matching call.

NAMED_ALLOW_CALL() on line 18 is a shorthand for NAMED_REQUIRE_CALL(...).TIMES(0,infinity), i.e. an absence of calls is OK and infinitely many calls are equally OK. There is also a FORBID_CALL() with obvious meaning.

.LR_WITH(_2 <= stock) on line 20, makes the call match only if _2 <= the value of the member variable stock at the time of the call. .WITH() and .LR_WITH() tests are always tried first when a matching signature is found. You can add as many .WITH() and .LR_WITH() as you like for an expectation. They are tried in the order they are added.

 .LR_SIDE_EFFECT(stock -= _2) subtracts _2 from the member variable stock. You can add as many .SIDE_EFFECT() and .LR_SIDE_EFFECT() as you like to an expectation. They are executed in the order they are added, provided that the signature and the .WITH() and .LR_WITH() conditions matched.

Sequencing may seem a bit odd when the number of calls required to match is not an exact number. In Trompeloeil a sequence restriction is satisfied when the minimum number of calls is reached. In the example above, this means that it is an error if remove is called before hasInventory, but it is OK to never call remove, since the minimum number of calls is 0, which is trivially satisfied. It is likewise OK to call hasInventory many many times before remove, and then call hasInventory again followed by further calls to remove.

Wrap up

Please give Trompeloeil a try and give feedback (or better yet, join the job and submit improvements.) My experience is that its use of direct expressions/statements in the expectations together with the strict control of deadline for a match through the normal C++ lifetime rules makes it very easy to work with.

Thursday, August 28, 2014

Asserting compilation errors in C++

Sometimes when crafting an interface, we want to ensure that some illegal constructs leads to compilation errors. After all, a good interface is easy to use correctly, and difficult to get wrong, and what can be more difficult to get wrong than something that doesn't compile?

We also know that untested often is buggy, or at least we cannot be sure that it is correct, and tests tests that aren't automated tend to be forgotten. This, of course, means that even if you've carefully crafted an interface where some construction is illegal, and made some manual tests for it, some bug fix later might ruin it and no test will catch that the illegal construct now compiles.

However, namespaces and using directives opens an opportunity. Below is the beginnings of a trap that catches illegal calls to a function named f.
int f(std::string); // function to trap abuse of

struct illegal;
namespace bait {
  illegal f(...);
}

using namespace bait;

f(3);   // calls bait::f
f("");  // calls ::f
With the aid of the C++11 decltype specifier, we can catch the resulting type of an expression at compile time, in this case getting the return type of a function call, without actually making the call.
decltype(f(3))  obj1; // illegal
decltype(f("")) obj2; // int
Note that bait::f is never implemented. All that is needed is the signature so that the compiler can find match the arguments and get the return type.

With these two, the trap can be triggered at compile time using C++11 static_assert and std::is_same<T,U>
#include <type_traits>
static_assert(std::is_same<decltype(f(3)),illegal>::value,

              "compiles when it shouldn't");
The above compiles, since f(3) matches bait::f, and no code is generated. Changing to a match of ::f, however
static_assert(std::is_same<decltype(f("")),illegal>::value,
              "compiles when it shouldn't");
gives a compilation error. On clang++ 3.4.2 the message is:
fc.cpp:18:1: error: static_assert failed "compiles when it shouldn't"
static_assert(std::is_same<decltype(f("")), illegal>::value,
^             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
and g++ 4.8.2 gives the message:
fc.cpp:18:1: error: static assertion failed: compiles when it shouldn't
 static_assert(std::is_same<decltype(f("")), illegal>::value,
 ^
The results seem reversed. A call that would cause compilation error compiles, and a call that would compile gives a compilation error.

Now what if f is in a namespace? Things becomes marginally more cluttered, but the technique remains the same. Putting the bait in an inline namespace solves the problem.
namespace ns {
  int f(std::string);
}


struct illegal;

namespace ns {

  inline namespace bait {
    illegal f(...);
  }
}
An inline namespace is a namespace, but a everything declared in it is visible in its surrounding namespace, similar to the using namespace directive used earlier.
ns::f(3);   // calls ns::bait::f
ns::f("");  // calls ns::f
The test thus becomes:
static_assert(std::is_same<decltype(ns::f(3)),illegal>::value,
              "compiles when it shouldn't");
A macro can help with code readability:
#define ASSERT_COMPILATION_ERROR(...) \
static_assert(std::is_same<decltype(__VA_ARGS__), \
                           illegal::type>::value, \
              #__VA_ARGS__ " compiles when it shouldn't")
Writing the test code as:
ASSERT_COMPILATION_ERROR(ns::f(""));
gives the (g++ 4.8.2) compilation error:
fc.cpp:20:3: error: static assertion failed: ns::f("")compiles when it shouldn't
   static_assert(std::is_same<decltype(__VA_ARGS__), illegal>::value, \
   ^
fc.cpp:23:1: note: in expansion of macro ‘ASSERT_COMPILATION_ERROR’
 ASSERT_COMPILATION_ERROR(ns::f(""));
 ^
I think this is pretty neat. It is now simple to test that illegal calls actually don't compile. You can add these tests to the unit test program that asserts the intended functionality.