I really enjoy writing automated tests(yes, really) for basically anything i am working on, i sometimes go test-first if something is clear how it’s going to end or at least clear enough
to see a possible outcome as you go.But what i don’t do is mocking!(sometimes unavoidable, but still) I usually use a set of fixtures with real data(at least as real as a predefined piece of information can be), letting my tests hit real endpoints and databases and actually thinning the barrier between unit and integration testing.
So, i had a thought, what if i could have my program run its tests using the actual real data, inputted from real users in a real state environment? How would that turn out? Would that be
feasible(ignoring runtime overhead and implementation details) and actually useful? I am still not sure but i went ahead and hacked an implementation in order to explore it.
Note: in the following section i’ll use actual code with LiveUnit but i am interested in the idea, the theory if you will, not the details.
For a simple program like :
A live test could look like :
The test_calculate will run automatically after the calculate method completes and will notify us of a failure.
Now let’s see what our test say, pretending we see something like that in the real world :
“I should return something > 400 when my instance’s @myvar == 15”.
Imagine @myvar was @has_access == true|false and the method returns the result of a record editing operation.Could that test spot a loophole, an edge case in our program? Yes.
A normal unit test would have of course spotted this as well but maybe we would have mocked @has_access instead of letting our login system set the session variable in a real live case.
I can see the appeal to that, testing every possible edge case against every possible global state can be hard.Imagine deploying your application to a development server, turn live-testing on
and use it for a while like a normal user would and then examine the report for any fails.
In a few words it’s just “After each unit finishes execute a callback that runs its test using current real state data and log the results”.
I see this as a higher level of logging or just another way for reporting.
The implementation is bad but fits its purpose and should work for simple ruby programs, i haven’t tested more complex things like rack applications or anything that actually matters.
Anyway, it was a fun little experiment which i still can’t figure if is of any use what so ever, if someone is interested go to the github repo, look at the readme and give it a spin.