Proposal: mocking support

[PavelVozenilek]

Definition from Wikipedia:

In a unit test, mock objects can simulate the behavior of complex, real objects and are therefore useful when a real object is impractical or impossible to incorporate into a unit test.

When one tries to do mocking in C++ he usually needs to:

1. Create abstract base classes (interfaces) for everything that may be mocked.
2. Place all mockable functionality into subclasses and virtual functions.
3. Create mechamism to replace mockable parts with mocks (e.g. via dependency injection).

Something what on first sight looks rather simple turns into nightmare. Design needs changes to allow mocking, internal details have to be made public, complex frameworks are invented to support mocking.

---

My proposal: if a test needs to mock something just redefine that thing inside the test.

test "with mock" 
{
  // "re-define" is context dependent keyword
  re-define fn fopen(name : string) -> int { ... } // dummy replacements
  re-define fn fclose(int handle) { ... }

  // test code opening and closing files, doesn't actually touch the disk
}

Whenever compiler sees a test containing redefinitions it acts as if original functions do not exist anymore and uses redefinitions instead. For code outside this specific test nothing changes.

(I suppose that names are unique in Zig, so no redefinition will be ambiguous.)

---

Original functionality may still be accessed:

test "with mock" 
{
  re-define fn fopen(name : string) -> int { 
    ...
    var id = ::fopen(...); // leading :: means original definition in this context
    ...
    return id;
  }
  ...
}

Redefined functions may access test local variables as "globals", to communicate intent and results. I believe access to parent's locals is planned feature for inner functions, this is similar.

test "with mock" 
{
  var expected result : int;
  ...
  re-define fn fopen(name : string) -> int { 
    return expected result;
  }
  ...
}

Unit test may need imports to do the redefinitions:

test "with mock" 
{
  import "whatever needed"
  re-define const some_datatype = struct { ... }
  re-define fn some_function(val : some_datatype) { ... }
  ...
}

Anything could be mocked, e.g. constants, struct methods, types ...

test "with mock" 
{
  re-define const TIMEOUT_MILLIS = 0.1;
  ...
}

---

Implementation: I am not sure whether such feature could be easily implemented. One (not very scalable) way is to do complete project recompilation for every single test with redefinitions, doing replacement on source code level.

---

Situations for which suggested feature is not appropriate: very complicated mocks that would require lot of redefinitions. These situations would be better handled in traditional way, with design changes.

[hasenj]

I think the whole concept of mocking is an artifact of the OOP philosophy which seems to me at odds with the spirit of what zig is trying to be (please correct me if I'm wrong).

If you are writing low level code I would think you are mostly thinking about data and how to manipulate it.

So if you want to make sure your code is testable and reliable, structure it so that the data manipulation is "pure" in that it's independent of any input/output from the environment.

When you are dealing with pure data, there's nothing to "mock". You just create sample data and feed it to the function to test its output.

For example, if you have to parse csv files and need to test your parser, write it so that it receives a string representing the content of the csv file or a line from it, not a file name that it needs to open. Then you can easily write a test on the function because you just pass it a string and there's no Input/Output to mock.

[PavelVozenilek]

@hasenj: I don't think mocking is only OOP thing, it just happened that OOP turned testing (also) into giant monster. Secondly, it is not always possible (or desirable) have pure functional design which would (mostly) not need such a feature.

Say you have the most primitive webserver (could be done in 50 lines of C) and want to test it. Instead of adding many lines which hijack functions like bind/accept etc and duplicate their hidden state you redefine the (well documented) socket API (~6 functions).

What you get:

1. The 50 lines webserver is not polluted by any test supporting logic. In this case it could dwarf the "real" code.
2. The checks inside redefined API are for single specific test, thus simple, with hardcoded literals etc. One is not tempted to turn it into general purpose tool.
3. The test doesn't touch the "real TCP system". (Sockets can get easily exhausted in the system, with funny consequences for web browsers etc.) And this is done without recreating the TCP/IP stack functionality.

[Ilariel]

I'm not so sure whether we would need a separate mocking support instead of just having more advanced facilities for compile time type manipulation. Basically after initial implementation for your mockable type builder function you could just redefine an interface as you see fit. This of course has some implications such as having to know what you might want to mock and you would have to program against that interface.

However as a possible improvement to @PavelVozenilek's suggestion. Instead of using a re-define keyword we'd use the test keyword to redefine functions in test scopes.

test "with mock"  {
  test fn fopen(name : string) -> int { ... } // dummy replacements
  test fn fclose(int handle) { ... }
  test const TIMEOUT_MILLIS = 0.1;

  // test code opening and closing files, doesn't actually touch the disk
}

[PavelVozenilek]

Here's idea how it could be implemented:

1. Compiler finds out all functions that are mocked.
2. It generates code for these functions, but every call to them anywhere in the application is indirect call, through pointer to function.
3. When a test with mock starts these function pointers are updated with new implementation. Afterwards the original pointer is restored.

Notes:

1. Mocked inlined functions won't be inlined.
2. Mocked exported functions will export a helper which calls indirectly the expected code.
3. One cannot easily mock functions in object files and static/dynamic libraries. It may be done via some assembler trickery, but perhaps it isn't worth the trouble.

[hasenj]

If a user needs to mock a function, can't they implement this indirection themselves?

Instead of calling fopen, you can call my_fopen which would check a global flag, and if it's set, call my_mock_fopen, else call the default/builtin fopen .. or something of this sort.

[PavelVozenilek]

@hasenj: that's complicating the design to support testing. In extreme this leads to complex mock frameworks, everything being interface, everything being publicly accessible.

Imagine testing hundred of functions. That would be hundred of flags and three hundred of new names (my_open, my_mock_open, the flag).

With mock proposal above it would be 0 flags, 0 new names and no design changes. You delete a test and no other fix is needed.

[kyle-github]

@PavelVozenilek, I certainly have used some mocking frameworks in C in the past, but they were very, very limited in scope and used ugly macro tricks. It has been many years so I do not remember details. Much of it relies on LD_LIBRARY_PATH and other hacks.

Since Zig is compiling everything from scratch, I am not sure that you need to do the patching.

@hasenj, @Ilariel, I would see this as possibly extending beyond the use of mocking for tests. The LD_LIBRARY_PATH trick allows some very interesting (if somewhat delicate) replacement to happen. Perhaps something like:

fn override fopen(name : string) -> int { ... }
fn override fclose(int handle) { ... }

Here I used a new keyword override to show that the function definition is being replaced. The requirement would be that a replacement function MUST have exactly the same signature as the function it is replacing. If you allow calling the previous definition within the replacement function, then you can use this to inject all kinds of debugging etc.

Any kind of mocking or function replacement is fairly useless if you need to be able to modify the original code to make it work.

Perhaps there is a way to leverage compile time to get most of this?

comptime const fopen = if(debug == true) { debug_fopen } else { fopen };

Then in the rest of the code in that module when you use fopen you get the debug version (or not if you are in release mode). While this appears possible today, it does not allow for replacing the function everywhere it is called, not just in the current module.

[Ilariel]

@kyle-github, I have to agree that conditional compiling with compile time constants is good enough for most cases and it has always been good enough for my usage as I usually need proper logging data and small tests aren't exactly good enough for that big things.

As a disclaimer I also have to say that I haven't looked at the compiler code or written any tests in Zig.

However given that tests should be their own isolated cases of code which should fail when result isn't what it should be and they aren't exactly something that you export as library code or as real executables I think function/constant replacement in the scope of the tests with test keyword allowing you provide testable behavior with the same signature/type could be reasonable so that we don't add yet another keyword and you could create tests without having to actually modify the actual source code or prepare it for testing since the beginning. As silly as it sounds I think this could be a nice to have feature that isn't a must.