Uncommon Things for Common Tests
As far as the Erlang testing goes, here are some snippets I've collected over the years with the great help of my teammates (shout out to the Regulators crew!).
First things first
At least if you're using rebar everyone is so keen about, you can put your
helper libs in the CT test directory. I usually make one called test_helpers and
alias it with ?th macro.
Run a single Common Test case via Makefile
It greatly improves your workflow, especially if the whole local build takes up
to 30 minutes. Here's a make target ready to use:
ct-single:
@mkdir -p ct_log
@if [ -z "$(suite)" ] || [ -z "$(case)" ]; then \
echo "Provide args. e.g. suite=moo case=boo"; exit 1; \
else true; fi
@ct_run -noshell -pa deps/*/ebin -pa apps/*/ebin -include apps/*/include \
-include deps/*/include -sname test -logdir ct_log -ct_hooks cth_surefire \
-sasl sasl_error_logger false \
-dir $(dir $(shell find . -name $(suite)_SUITE.erl)) -suite $(suite)_SUITE -case $(case)
Of course this won't work if you're using CT groups. That can be tweaked with the -group
switch.
Common fixtures directory
The CT-enforced convention is to have one data directory per suite. We were fed up with that; reusing fixtures is usually the bread and butter of medium/large systems.
set_fixt_dir(Test, Config) ->
case code:which(Test) of
Filename when is_list(Filename) ->
CommonDir = filename:dirname(Filename) ++ "/fixtures/",
[{common_data_dir, CommonDir}|Config]
end.
load_fixt(Config, Filename) ->
F = filename:join(?config(common_data_dir, Config), Filename),
{ok, Fixture} = file:read_file(F),
Fixture.
So the way you call it in your suites is:
init_per_suite(Config) ->
% fiddling with application:env and all the mocking tyranny...
Config2 = ?th:set_fixt_dir(?MODULE, Config),
Config2.
whereas:
t_do_what_i_tell_you(Config) ->
BinData = ?th:load_fixt(Config, "data.raw"),
% ...
Exporting all the tests
Have you noticed the t_NAME convention for a test case I used in the previous
paragraph? This comes in handy when you choose not to use groups.
all_tests(Module) ->
lists:filtermap(
fun({F, _}) ->
Name = atom_to_list(F),
case Name of
[$t,$_|_] -> {true, F};
_ -> false
end
end, Module:module_info(exports)).
And then, in CT all/0 callback:
all() ->
?th:all_tests(?MODULE).
This way you don't need to needlessly repeat yourself in all/0 and
-export/0. Just export the tests you wish to run, and the compiler will warn
you about the unexported ones.
Debug calls
Let's assume, for the purpose of the following example, that your system
has become too hard to reason about, especially when it comes to testing async routines.
First of all, dbg works here too.
start_debug_modules(Mods) when is_list(Mods) ->
dbg:start(),
dbg:tracer(),
lists:foreach(fun(M) -> dbg:tp(M, '_', '_', []) end, Mods),
dbg:p(all, c).
stop_debug_modules() ->
dbg:stop_clear().
That's all you need in your helpers library. Call that on setup/teardown with the modules of interest/hate as an input to get things moving.
Sleep is the cousin of death
When testing e2e async routines, people often use timer:sleep/1 and similar
to "wait for an output". This is wrong due to the fact that any kind of sleep
is non deterministic. In other words it will behave incalculably on different
test environments - example: the Bongo DB on your ssd omgmacbook is way faster
than the one on the CI box. If you really need to wait for an output,
please keep trying to wait instead. Consider the following functions in your
helpers module:
keep_trying(Match, F, Sleep, Tries) ->
try
case F() of
Match -> ok;
Unexpected -> throw({unexpected, Unexpected})
end
catch
_:_=E ->
timer:sleep(Sleep),
case Tries-1 of
0 -> error(E);
N -> keep_trying(Match, F, Sleep, N)
end
end.
keep_trying_receive(Match, Sleep, Tries) ->
keep_trying(Match, fun() -> receive Msg -> Msg
after Sleep -> error
end
end,
0, Tries).
Comparing JSON dudes
Dure to different circumstance I have never tried maps. We often need to put some JSON
somewhere and check if the output matches a JSON elsewhere. We have written a function
that tries to deeply compare two lists of tuples at least as far as jsx
is concerned.
assert_deep_love(Expected, Got) ->
compare(lists:sort(Expected), lists:sort(Got)).
compare(X, X) ->
true;
compare([], []) ->
true;
compare(Obj1, []) ->
{false, {obj1, Obj1}};
compare([], Obj2) ->
{false, {obj2, Obj2}};
compare([{K, V}|Obj1], [{K, V}|Obj2]) ->
compare(Obj1, Obj2);
compare([{K, V1}|Obj1] , [{K, V2}|Obj2]) ->
case compare_values(V1, V2) of
true ->
compare(Obj1, Obj2);
{false, E} ->
{false, {key, K, E}}
end;
compare([V1|L1], [V2|L2]) ->
case compare_values(V1, V2) of
true ->
compare(L1, L2);
{false, E} ->
{false, {list, E}}
end.
compare_values([{_,_}|_] = Obj1, [{_,_}|_] = Obj2) ->
assert_deep_love(Obj1, Obj2);
compare_values(L1, L2)
when is_list(L1) and is_list(L2) ->
assert_deep_love(L1, L2);
compare_values(V1, V2) ->
{false, { {expected, V1}, {got, V2} }}.
Put that in your helpers module and benefit from it.
That's all folks
Ping me if you need me!