Good lord. Not another unit test framework!

Why muTest

There are tons of C/C++ unit test frameworks out there, so why write another one? Well, I didn't mean to. :) I was unhappy with the very weighty CppUnit, CUnit, and CuTest because I ended up wrestling with the unit test framework instead of writing tests. So I started mutest (then called zutest) in 2005 for some internal projects. After sporadic development, it has turned into an amazingly flexible, reasonably capable framework.

Advantages

  • Can turn on verbose mode and see exactly what tests are being run and with what values. This helps a lot when someone asks, "what do your unit tests cover?"
  • Output is similar to GCC so many IDEs can parse it directly. (include directions on how to use this in Eclipse).
  • CONTROVERSY DESIGN: Runs tester and test in the same process. This is slightly less reliable than the alternative but far easier to write and debug. In practice, since most tests will fail in assertion rather than crashing, the drawback is minimal but the advanatage is huge.
  • Can dynamically disable tests at runtime.
  • Straight C. It won't confuse your IDE or screw up code analysis tools (unlike, say,
  • Totally debuggable. Step through your test suites, set breakpoints deep in test code, it all works just like your actual app.
  • Light weight enough to ship unit tests in your final executable, flexible enough to not. Your choice. When tracing strange or intermittent failures on live systems, it helps to still be able to run the unit tests. Adding unit tests tends to increase the size of your app by only a few tens of K. To omit the unit tests anyway, simply define NDEBUG or NO_UNIT_TESTS when compiling.
  • Does not require C++. Can be used to test both C and C++ programs.
  • Does not dictate where tests may reside. Many test frameworks require tests to live in FILE_test.c and compiled separately from the unit that they are testing. Personally, I find that it's a LOT easier to keep tests up to date when they live in the same file as the code that they are testing.
  • Includes a few test dialects (TODO: link to this). If you don't like the included dialects, it's easy to write your own.

test_eq, test_equal, etc. test(a op b), test_op(a, op, b) must_be_equal(a, b), must_not_be_equal(a, b)

Perceived Disadvantages

  • No boilerplate like http://www.lastcraft.com/cgreen.php#mock. This is an example of a feature that is harder to use than it is just to stub out yourself when you need it.
  • Doesn't fork. This means that there's a small chance that a wayward test will stomp all over muTest's data structures. However, the chances of this causing a failing test to appear to succeed are miniscule, yet it is noticeably more complex and ruins your chances of easily attaching GDB. It's a bad tradeoff.
  • Doesn't statically declare tests. This makes it much easier to write tests and debug them -- just write functions!

Disadvantages

  • TODO: should interpret argc/argv. First, optional --test. If --test exists, -l --list will list unit tests. -s --skip will skip some, -r --run will run some, -f --fail-on-error will cause it to fail on error, -v --verbose?. We should do this ourselves and return if we don't see
  • TODO: Signals cause it to bomb out completely. Especially trap int/fp divide by zero and bus errors (both ptr deref and funcptr call).
  • TODO: standardize the NO_UNIT_TESTS macro and test it.
  • TODO: support subtests?
  • TODO: Add assertion flavors.
  • TODO: Add the ability to explicitly run or skip tests.
  • TODO: Add the ability to halt testing on the first test failure.
  • TODO: Add the ability to time a test out.
  • No graphical front-ends. Maybe convert Eclipse's cppunit plugin to call mutest?

How do I use it with my program?

If you're using Subversion, you should use its externals feature to keep muTest up-to-date. Otherwise, just use a recent tarball release.

Installing via Subversion

If your project uses Subversion, it's easiest to pull mutest in using externals. You can be sure that mutest's api will never be broken in a release branch:

   $ svn propset svn:externals "mutest http://svn.u32.net/mutest/branches/releases/rel_1.x" .

(right now mutest doesn't have any releases. Hopefully that changes)

To ensure that mutest never changes even if a bugfix is released, first find the specific revision that you'd like to use. Here is how to discover the latest revision on the trunk:

   $ svn info http://svn.u32.net/mutest/trunk
   ...
   Last Changed Rev: 121
   Last Changed Date: 2007-02-28 13:52:43 -0800 (Wed, 28 Feb 2007)

Now specify the revision when setting up the externals property:

   $ svn propset svn:externals "mutest -r121 http://svn.u32.net/mutest/trunk" .

If you ever want to update mutest, set update svn:externals property to the new value and then run "svn up" to pull down the changes.

   $ svn propset svn:externals "mutest -r155 http://svn.u32.net/mutest/trunk" .
   $ svn up

If you make changes to mutest, please submit the patch!

TODO: also maybe include a tutorial on how to integrate with git?

Installing via Tarball

Adding Tests to your App

Test Coverage

TODO: http://check.sourceforge.net/doc/check.html/Determining-Test-Coverage.html#Determining-Test-Coverage

If you want to check the coverage of only your units, create a standalone unit test application. Your units should be tested to within a few tenths of 100%.

Checking the coverage of your entire application is possible but, for most real world apps, unit tests can only achieve coverage of around 60%. You need to use functional tests to exercise the rest.

FAQ

When compiling mutest, my compiler complains about a missing __func__ macro!

I guess you're not using GCC. Most modern compilers expand __func__ to the name of the current function. If your compiler does not support this, you can add -D__func__='"test"' to the compiler's command line so that __func__ expands to "test" in your source code. Since you still have __FILE__ and __LINE__ macros, you'll still have all the information you need to debug failing tests.

Other C/C++ Unit Testers

  • MinUnit -- Very cool. muTest is just a more useful MinUnit.
  • CUnit -- Decent, lots of code, poor assertions.
  • CppUnit -- C++ only, very heavyweight.
  • CuTest -- Decent but included lots of unnecessary code.
  • cUnit -- dead. Also had some issues with test hierarchies.
  • Cut -- stillborn.
  • Gnu Autounit -- dead. Too tied to autoconf.
  • I wanted to name this project "munit" (mini unit tests) but one already exists.

There is an awesome list at http://www.opensourcetesting.org/unit_c.php