Announcing beta of a new tests runner: elm-test-rs

Show and Tell
2

Complementary note on portability, architecture and performances

If you are interested in the some of the design decisions, architecture choices and implications on performances of elm-test-rs, here are some complementary notes.

The portability requirement of elm-test-rs implies that a few things had to be done differently, impacting the performance profile of elm-test-rs. In particular there are two important steps that are done differently in elm-test-rs and in node-test-runner@0.19.1-revision4.

  1. Generation of an elm.json for the main program of the test runner, combining dependencies of the user and of the test runner.
  2. Detecting exposed tests to build the main of the test runner.

For (1), node-test-runner@0.19.1-revision4 is using custom JS code, that is not a full dependency resolution algorithm, and thus causing few dependency issues in some edge cases. To solve that, node-test-runner@master now leverages elm-json as a binary, which makes it a bit longer to npm install but overall brings correct dependency solving at almost no runtime cost since elm-json improved its offline mode. In elm-test-rs, we leverage the pubgrub dependency solver that I presented here few weeks ago, meaning there is no additional installation cost as it’s embedded in the elm-test-rs executable.

For (2), node-test-runner@0.19.1-revision4 is using the elmi-to-json binary, which is a tool to convert the .elmi interface files generated by the elm compiler. This means that an additional preliminary compilation step is required to generate the .elmi files, and then it uses elmi-to-json to extract the exposed tests. Both in node-test-runner@master and in elm-test-rs, we are detecting exposed tests with a new approach. First we parse the tests files to extract “potential” tests values, then, in the generated JavaScript, we embed a kernel-written function check : a -> Maybe Test used to filter all potential tests values with code looking like this:

main : Program Flags Model Msg
main =
    [ {{ potential_tests }} ]
        |> List.filterMap check
        |> Test.concat
        |> ...

In practice, this means that we avoid both the preliminary call to the elm compiler, and the call to the elmi-to-json binary. As a result, both node-test-runner@master and elm-test-rs have very similar timing profiles, faster than node-test-runner@0.19.1-revision4.

On my machine, and with my network (which is impacting the elm compiler), here are a few timings obtained for node-test-runner@0.19.1-revision4, node-test-runner@master and elm-test-rs on successive calls. Beware that the first run with a cleared elm-stuff will take additional time with all three tools, due to more things happening within the elm compiler.

test runner empty package empty app elm-color (8963 tests)
node-test-runner@0.19.1-revision4 1.8s 1.4s 3.4s
node-test-runner@master 0.7s 0.7s 2.3s
elm-test-rs@0.4.0 0.6s 0.6s 2.8s

As you can see, both the upcoming node-test-runner and elm-test-rs are swift to run when there is a small amount of tests. For elm-test-rs, the 600ms of the empty package are roughly divided in 10ms for the preparation Rust code, 300ms for the elm compiler, 300ms to spawn the node supervisor and the worker for the one test present.

The preparation code isn’t the only architectural design difference with node-test-runner. In elm-test-rs, I implemented the simplest system to distribute tests to the runner workers and gather tests results. The supervisor gives a test to a worker. When this one is done with it, it informs the supervisor of the result, and the supervisor gives it a new test to run. In node-test-runner however, the tests are split in as many groups as workers, and each one runs all the tests of its group and then informs the supervisor of the results. So node-test-runner has a lot less communication going between the supervisor and the runners for work distribution. I believe that’s why it is faster than elm-test-rs when there are plenty of unit tests, such as with elm-color which has 8963 tests. But one advantage of the very simple model of work distribution of elm-test-rs, is that some functionalities like capturing debug logs are almost a one-liner:

let logs = []; // logs is flushed for each test run
process.stdout.write = (str) => logs.push(str);
10 Likes