Tuesday, December 19, 2023

About time - how to unit test code that depends on time

Suppose that the logic of your program depends on time. That is, you need to keep track of when something in the past happened, and what time it is now, and the logic of what to do depends on how much time passed between that previous event and now.

There are many programs with this kind of behaviour. My experience is primarily from networking, where we need to figure out if a response is timely or late. Such systems often uses timers, i.e. on some action, request a notification at a specific point in time in the future.

How do you design such a system so that it is testable?

The naïve approach, to just call std::chrono::<some_clock>::now(), whenever you need a time stamp, makes unit-tests more or less impossible, so avoid that.

Approach 1, the alias clock

Instead of directly referring to  std::chrono::<some_clock>::now() in your code, you refer to app_clock::now(), and in system builds app_clock is defined to be std::chrono::<some_clock>, but in unit-tests, they're some test clock.

This is an improvement. In the tests, you now have a means to control what time it is and how time advances. However, a major drawback is that you need to have different builds of your unit under test depending on situation.

Approach 2, template specialization access

This is a neat trick, using (abusing?) how the template machinery works in C++.

Create an encapsulation like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
template <typename ...>
constexpr auto clock_impl = std::chrono::some_clock{};

template <typename ... Ts>
struct app_clock
{
    static
    std::chrono::some_clock::time_point now()
    {
        return clock_impl<Ts...>.now();
    }
};

Now whenever you call app_clock::now(), it will call clock_impl<>.now() which is std::chrono::<some_clock>::now().

For our tests, we can define a test_clock.

1
2
3
4
5
6
7
8
struct test_clock
{
    using time_point = std::chrono::some_clock::time_point;
    static time_point now() { return {};}
};

template <>
constexpr auto clock_impl<> = test_clock{};

Now we have a specialization for clock_impl that is our test_clock. It is imperative that the signatures of test_clock::now() and the default std::chrono::some_clock::now() are identical.

See an example at https://godbolt.org/z/GbWYaGc7q

This overcomes the need for having separate compilations for tests and production.

Approach 3, the clock factory

 In this case, whenever the program needs to know what time it is, it calls clock_factory::get_clock(), which in production code returns some encapsulation of std::chrono::<some_clock>.

This is better. Now the only code that differs between a unit test build and a production build is what clock_factory::get_clock() returns.

Unfortunately these factories tends to be singletons, with all the problems that they bring.

If you want to model your test clock as a mock, you have the additional problem of how to ensure that the test code and the unit under test sees the same clock. It's also really difficult to correctly provide all the right expectations for the mock without over constraining the tests (exactly how many times should the time be asked for, and what time should be reported on each call?)

Approach 4, clocks from above

Instead of having code ask for clocks from factories, you can model your program so that every class that needs to know the time has a constructor that accepts a clock and stores it as a member variable.

This gets rid of the singleton (yay!!!), but it adds a lot of extra storage and all the other problems remain.

Approach 5, pass time stamps

Here the problem is turned on its head. What if the code doesn't need to ask for the time, but can be told what time it is?

An observation is that many systems like this only need to know the time at a few places in the code, typically at the source of events. Get the time when a message is received. Get the time on user input. Get the time when receiving a signal. Get the time when a timer fires.

Then, all actions that come as a result of these events, are passed the time stamp.

Passing a chrono time_point is cheap (it's typically a 64-bit value passed in a register).

Now tests become easy. All tests of code that needs to know the time are given time points as input, controlled in full by the test code.

An additional advantage is that you can (should!) instrument your timer code so that for every timer that fires, you keep track of how late (or early!) it fired. This is typically much more interesting than to measure CPU-load.

The disadvantage is a loss in precision of time. Some cycles will pass between getting the time stamp at the event source and the logic decision that depends on the time. The programs that I have experience in writing are not bothered with that loss of precision. Your program may be different, in which case one of the other approaches may be a better choice.

Which to choose?

If you can live with the loss of precision from Approach 5, pass time stamps, I think that is the preferred way. It makes everything so much easier. If the loss of precision is unacceptable, Alternative 2, template specialization access is probably the best option.

Tuesday, December 12, 2023

When private member function?

 I've seen this a few times too many recently, and need to get it off my chest.

Ponder a class that has a private member function. The function does not touch any member variables nor does it call any member functions.

In my opinion, there are three possible situations here:

  1. The function is completely generic. Nothing it does is specific to the class.
  2. The function is specific to the problem domain of the class, but it not otherwise dependent on it.
  3. The function uses types that are private to the class.

In case of 1,talk to your colleagues. Say that you need this generic function. Fight^H^H^H^H^HArgue about its name, what it does, and where it belongs. That's good. It increases understanding in your team.  Do not make it a member.

In case of 2, make it a free function, in anonymous namespace, in the .cpp file for the class. It's a detail of how things are done, and no user of the class needs no know.

For 3, it may be the right thing to make it a private static member, but it may also be right to make it a function template. Discuss with your colleagues.

These are not do or die issues, but small things that makes the daily a little better for yourself and your colleagues.

If the function is completely generic, making it available for others saves them the work of replicating it. This, of course, requires that they know about its existence, hence the discussion about where it belongs, what it should be called, and what it's signature should be.

If the function can be squirreled away in the .cpp file, it reduces the visibility. At the very least it shortens the build times, since users of the class won't have to spend the build time parsing the signature, but more importantly it's a compartmentalization of the signature. If you need to make a change to how the function is called, it's an internal matter for the implementation of that class, no other files will need to be recompiled.

In my experience the 3rd case is much rarer, but also much more worthy of discussion. Make use of the opportunity to share with, and learn from, your colleagues.

And, most important of all, go through this list above before writing the function. There is no need to write the private member function first and then argue that it shouldn't be there.

Friday, January 6, 2023

A Linux C++ programmers adventure in improving Windows CI on GitHub actions

A Linux C++ programmers adventure in improving Windows CI on GitHub actions


TL;DR;

  • Ninja is available directly in the windows images
  • Use ilammy/msvc-dev-cmd@v1 to set up the MSVC environment for building with Ninja.
  • You need nested quotes for CXX flags from CMake CLI options.

How it started:

It has been very frustrating, on and off, for a few days now, and I thought that if I write down my findings someone else may perhaps find them and be saved the frustration.

The background. I have a few open source C++ libraries hosted on GitHub. They are well covered for Linux builds with both clang and gcc, and have had poor, if any, coverage for Microsoft's MSVC. I should say right away that the frustrations were about build environments, and not about peculiarities of MSVC, which Linux C++ programmers are otherwise well known to be tripped by.

A few days ago, I decided to split the single header file template library strong_type, into several small headers, and split the unit test sources similarly.

This had the effect that build times for Windows went from around one minute to around 4 minutes. On Linux I used Ninja for parallel builds, and it was not much affected by the change.

First attempt: --parallel

cmake --build build -t self_test --parallel

As far as I can tell, the --parallel flag did bugger all on Windows.

Second attempt: ninja

Search for how to build with Ninja on Windows in github actions. Unfortunately this search lead me to several actions for installing Ninja on the github actions marketplace. I tried a few. They all involve an extra step in the pipeline, but it only adds a few seconds, but the clincher is that they change the compiler to gcc instead of cl.exe.


More searching found several explanations for why this is exactly what you want (what???), but that if you insist, you just explicitly set the compiler to 'cl' and all will be fine. Except that CMake now said it couldn't find 'cl'.

 

At this point I decided to shelve ninja and instead direct my attention to strengthening the warning levels.

Third attempt: Add CMAKE_CXX_FLAGS

I want to adhere to strict C++ and clean builds at high levels of warnings, so I added -DCMAKE_CXX_FLAGS="/permissive- /EHsc /W4 /WX" to the CMake command line. This  caused all types of problems, because apparently the quote isn't carried through and CMake made a complete hash of all its arguments.


This, of course, lead to some advanced quotology, i.e. randomly changing quote characters, nesting quotes, escaping quotes, order of quotes, until suddenly it worked with -DCMAKE_CXX_FLAGS="'/permissive- /EHsc /W4 /WX'". As with any CI problem, each tiny change is a commit/push/wait/observe cycle.

Fourth attempt: Back to Ninja again

By this time I had stumbled upon the information that Ninja is available in the build images already, so no need to add it as a separate step. I also learned that there's a tool named 'vcvarsall.bat' that sets up the environment. Unfortunately I also learned that the location of that script depends on the version of Visual Studio installed, and since I'm not a regular windows developer, I'd have to go through the frustration of finding out the path whenever there's a new visual studio. Fortunately, it turns out that GitHub user Oleksii Lozovskyi (@ilammy) seems to have taken upon themselves to carry that frustration for us, so that we don't have to, by publishing the github action ilammy/msvc-dev-cmd@v1. Just use it as a step prior to running CMake. This now works, with "-G Ninja". Woohoo!

 

One last frustration is that the the path to the built program differs when building with Ninja. It changed from ./build/test/Debug/self_test.exe to ./build/test/self_test.exe.

Result: Around 90 seconds for Windows builds instead of 4 minutes, and better warnings.

Here's an excerpt from the ci.yml file:

   name: "Windows C++${{matrix.config.std}}"
   steps:
     - uses: actions/checkout@v3

     - name: Configure MSVC console (Windows)
       uses: ilammy/msvc-dev-cmd@v1

     - name: "setup"
       shell: bash
       run: |
         cmake \
           -S . \
           -B build \
           -DCMAKE_CXX_COMPILER=cl \
           -DCMAKE_C_COMPILER=cl \
           -DCMAKE_CXX_STANDARD=${{matrix.config.std}} \
           -DCMAKE_CXX_FLAGS="'/permissive- /EHsc /W4 /WX'" \
           -DCMAKE_BUILD_TYPE=Debug \
           -DSTRONG_TYPE_UNIT_TEST=yes \
           -G Ninja

     - name: "build"
       shell: bash
       run: |
         cmake --build build --target self_test

     - name: "test"
       shell: bash
       run: |
         ./build/test/self_test.exe


90s, with huge variability I might add, is not as fast as I'd like it to be, but it's a huge improvement over 4-ish minutes.