A test suite for Elm

Show and Tell
1

Through working on the eco compiler I have become quite familiar with many aspects of testing Elm and its quirks. I have also created a huge number of tests. Part of eco’s test suite is a set of what I call the E2E tests, which are real Elm programs designed to test Elm, and the elm/* packages. I have extracted that test suite into the above linked project, and run it over Elm 0.19.1, 0.19.2, 0.19.3 and Lambera 1.5.0 (build on 0.19.3).

Here are the current results:

  ┌───────────────┬────────┬────────┐
  │   Compiler    │ Passed │ Failed │
  ├───────────────┼────────┼────────┤
  │ Elm 0.19.1    │ 1,375  │ 2      │
  ├───────────────┼────────┼────────┤
  │ Elm 0.19.2    │ 1,377  │ 0      │
  ├───────────────┼────────┼────────┤
  │ Elm 0.19.3    │ 1,371  │ 6      │
  ├───────────────┼────────┼────────┤
  │ Lamdera 1.5.0 │ 1,372  │ 5      │
  └───────────────┴────────┴────────┘

Eight tests have a different outcome on at least one compiler:

  ┌──────────────────────────────────────────────────────┬────────┬────────┬──────────────────┬──────────────────┐
  │                         Test                         │ 0.19.1 │ 0.19.2 │      0.19.3      │  Lamdera 1.5.0   │
  ├──────────────────────────────────────────────────────┼────────┼────────┼──────────────────┼──────────────────┤
  │ elm-bug-fixes/CaseCharEscapedDuplicateTest           │ fail   │ pass   │ pass             │ pass             │
  ├──────────────────────────────────────────────────────┼────────┼────────┼──────────────────┼──────────────────┤
  │ elm-bug-fixes/ParseErrorCasePatternUnfinishedTest    │ fail   │ pass   │ pass             │ pass             │
  ├──────────────────────────────────────────────────────┼────────┼────────┼──────────────────┼──────────────────┤
  │ AliasProblemExtensibleRecordUnitTupleAndFunctionTest │ pass   │ pass   │ hang (60 s       │ internal crash   │
  │                                                      │        │        │ timeout)         │                  │
  ├──────────────────────────────────────────────────────┼────────┼────────┼──────────────────┼──────────────────┤
  │ AliasProblemQualifiedTypeTest                        │ pass   │ pass   │ hang (60 s       │ internal crash   │
  │                                                      │        │        │ timeout)         │                  │
  ├──────────────────────────────────────────────────────┼────────┼────────┼──────────────────┼──────────────────┤
  │ AliasProblemRecursiveAliasWithCommentedParensTest    │ pass   │ pass   │ SIGSEGV          │ internal crash   │
  ├──────────────────────────────────────────────────────┼────────┼────────┼──────────────────┼──────────────────┤
  │ CyclicDefinitionATopLevelValueDefinedByItselfTest    │ pass   │ pass   │ SIGSEGV          │ internal crash   │
  ├──────────────────────────────────────────────────────┼────────┼────────┼──────────────────┼──────────────────┤
  │ CyclicValueALetValueDefinedByItselfTest              │ pass   │ pass   │ SIGSEGV          │ pass             │
  ├──────────────────────────────────────────────────────┼────────┼────────┼──────────────────┼──────────────────┤
  │ elm-conformance/DebugToStringTest                    │ pass   │ pass   │ fail (record     │ fail (record     │
  │                                                      │        │        │ field order)     │ field order)     │
  └──────────────────────────────────────────────────────┴────────┴────────┴──────────────────┴──────────────────┘

The reason you see 2 fails on 0.19.1 and 0 on 0.19.2, is that the changes made around those tests in 0.19.2 look like bug fixes, not regressions, so I put them into a bugfix category.

The test suite is likely to have very high coverage of code branches in the compiler, but not 100%. The reason for this is that I ran a coverage driven loop, generating more test cases to try and hit more branches in the compiler - effectively the same compiler as the guida port, which is 1:1 with the Haskell - therefore likely to be getting good coverage against the Haskel. But this has not been measured (perhaps it could be if anyone knows about Haskell coverage tools ?).

I am also aware that I have obviously not captured all known bugs in Elm in this test suite yet. Some “bugs” may be pinned by tests in the elm-conformance category. The elm-conformance categeory is where most of the quirky behaviour, that we might consider buggy (or not!) sits currently.

I have the elm-janitor bug sheet too, and could run a scan over all the github issues to try and extract and categorize more bugs as failing tests.

To give an example, I don’t have the type rank = 2 thing exposed as a test fail yet, or crashing on modBy 0, but I think it would be good to have those recorded in this suite as bugs.

This is not a criticism of Elm and the hard work spent creating it - I understand the volume of work and time involved and that its impossible to do everything. I have also had the benefit of coming at this in the age of AI; I did not write a single one of these tests by hand. Also worth noting that creating them was not effort free or slop, my effort has gone into curating and shaping this test suite over the development of eco itself.

I am happy to act as maintainer for this test suite. Rather than forking it into many fragmented opinions on what is the “correct” definition of Elm - raise an issue or PR against this project and lets refine it together.

12 Likes
2

Debug printing order changed in 0.19.3 - probably should not be relying on that and enforcing it with a test. I put it in the elm-conformance quirks pile for now.

4

Recently shared this on Elm Slack, but would be relevant here as well: each published Elm package can also act as a test - does the tested compiler compile the exposed modules of the latest version of the package correctly?

There are enough weird code patterns eg. in elm-css that exercise edge cases in the compiler.

A list of packages lives eg. at https://package.elm-lang.org/search.json .

EDIT: just realized you also wrote and the elm/* packages, so you’re already doing this to some extent! :+1:

5

That’s the approach I took with Alm. It worked quite well I believe. But hard to tell fully without having people try it on their proprietary code bases of course.