I’m an Elm enthusiast and I’ve been wondering about the development process of the Elm compiler. The codebase is really big but I haven’t found any test suite in the repository.
How does Evan verify that what he’s implemented works as expected and make sure there are no regressions? I know that Haskell eliminates lots of verifications that would be otherwise necessary in other languages, but still, you need some way to verify that what you wrote really does what it’s supposed to, don’t you?
I have found that testing the compiler gives a relatively uncommon set of tradeoffs. It is a project where things may not compile for a week or a month at a time. it is also a case where you usually implement a subset of what you want, gradually expanding to cover all the scenarios.
So in the actual process of development, I have not found that having tests that run on every single build give significant benefits. For example, if I start rewriting the parser, the tests will say “hey, it doesn’t work!” But of course it does not work, this is step 1 of a 20 step process. And then on step 2, “hey, it doesn’t work!” Etc.
So the time where testing is important is in the lead up to a public release, where new binaries are being created. The thing is that there are a great deal of things written in Elm, so the process of going through things that exist to make them work actually finds pretty much everything. At least, it has been my perception so far that releases have not tended to have big deal regressions.
So I think it could be beneficial to have a set of “weird programs” that recorded things that have been bugs in the past, but I also try to prioritize my work on “do I absolutely need to do this?” So I am not personally able to collect and maintain such a list given my existing tasks and the experiential data, but it might be cool for someone in the community to organize that. From there, when we have binaries to test, they can see how it goes with all that. For me, ideally it would be decoupled enough to just allow tests to be run as bash test.sh dist/build/elm/elm where you tell it which binary you want it to try out.
By coincidence it is something that I have been thinking about. I would call these tests “characterisation tests”; their purposes is not so much to work like normal unit tests that you evolve and keep up to date with the code on every commit, but as you say to test releases and provide an easy way of comparing the ‘character’ of the implementation from one release to the next.
I once wrote such a set of tests for my Prolog compiler. They were grouped into tests around different areas of the language, each version of the compiler would be shipped with the test pack that was relevant to it. That is to say that the test pack does not have to be a fixed target against which the compiler was built, its features can still evolve by moving the target - but it gives me something to compare and document-by-test with. Here is an example to give a flavour of what that looks like:
A test pack for a version of the compiler:
An example of some tests in the test pack:
I picked the unifier tests as an example, as unification is closely related to type checking/inference in a language such as Elm. The idea is to write tests to explore as many different cases as you can - both positive and negative examples.
Such a set of compiler tests for Elm could benefit:
Future releases of the compiler - summarize in a table what has changed.
Runtime behaviour of code in different environments - does it behave differently in different browsers, or when run under node.js or another javascript engine such as Java Nashorn.
Elm ports, for example the Elm syntax front-end to Elixir.
Any possible major compiler re-write to support a WASM back-end.
Experimental portings of the compiler such as investigations into WASM etc.
I am way too snowed under with the day-job to take this on. Perhaps by next summer I might have more time. Could this be done as a Google Summer of Code project?