My company has quite a large Elm app (100k lines) and we have some unit tests for decoders and for nice testable algorithmic functions. We have one or two end-to-end tests. We don’t have any Test.Html tests.
I feel like I should write more end-to-end tests but then I feel like they take a long time to write and I should be maximising on what elm-test can offer to write faster, neater tests and therefore should be trying to use Test.Html. (Aside, I realise I should have more tests in general.)
But when I look at the module help, I’m not really sure what to test for. And I guess I’m torn. On the one hand some of my unit tests are for super basic for things that appear to be obvious from the code and so almost seem crazy to test for and, on the other hand, when I look at what sort of Html tests I might write they seem kind of super basic and obvious from the code and crazy to test for so I find myself not writing them.
Can anyone help me by sharing the kind of Html tests they write and the sort of things that they prove to be useful for?
Bad idea : Taking snapshots of the dom
I was working on a team where the tests would render the html, and then diff it against some known good html we had saved to a file. If there were any differences the tests would fail. The problem was that dom changes arent problems. Almost 100% of the time the dom changes, its because the developer is deliberately changing the dom as part of whatever they are working on.
Good idea : Verifying particular dom transitions
Suppose you want the UI to transition to a loading state after you click “save”, and you dont want this to break unexpectedly. You can test for this by having a computer automatically click that save button, and then automatically scan the the dom for some kind of loading spinner html element. If its missing, then the test fails.
(Experimental?) Good idea : Custom types
View functions go from state to html, like this.
view : Model -> Html Msg
And you cant really test Html Msg; both because Html Msg is opaque, and because Html Msg changing isnt a failure case. However, if you generalize your views into some finite set, like this:
type View
= Ready
| Loading
| Done
and then have functions like these:
view : Model -> View
toHtml : View -> Html Msg
Then you can test that certain Msg and Model lead to certain View, without worrying about what actual dom nodes are supposed to appear. This should let you develop and change the Ready view, without causing the test to fail, while still verifying big important questions, like that the Ready view is what is shown when it should be shown.
My understanding is that that would be a end-to-end testing set up? Rather than using the Test.Html Elm module? Or is Gherkin more a way to phrase a certain kind of test and you still implement them with Test.Html?
I’m really interested to hear a vote against snapshot testing. It seems like it is getting quite popular but I agree with your argument that it perhaps isn’t the right level of granularity to test it. There is a bit of discussion on this elm-test ticket too: https://github.com/elm-community/elm-test/issues/209 (including a previous hopeful inquiry from me!)
Is the second example something that can be achieved with Test.Html? Perhaps it isn’t and perhaps I wasn’t as clear as I should have been in the question. I’m partially interested in what sort of tests people write with Test.Html. Perhaps that is manageable but it feels more like a full view & update & view test? Is that possible with the Elm Test modules rather than a browser based testing framework? If people aren’t using Test.Html though, I’m also curious what people do use and what sort of tests they write.
You’re third example seems to explicitly be trying to avoid using Test.Html which is interesting. Perhaps Test.Html does provide enough introspection tools to achieve what you’re interested in? Good to see as an example though!
Yes, that’s the whole idea. You just test the app like a user does. No Test.html. You write your tests in a special syntax (Gherkin), which looks like this:
Scenario: Resetting password
Given I login for the first time
And I logout
When I fill in "Email" with "my email address"
And I follow "Forgot password?"
Then I should see "Password reset instructions will be sent"
And the "Email" field should contain "my email address"
When I press "Email new password"
Then I should see "Please click on the link in the email to reset your password."
And a new email with subject containing "Replacement login information" has arrived for "my email address"
When I press "OK"
Then I should see "Forgot password?"
And the "Password" field should contain ""
When I visit "the first link in the last email"
And I fill in "New password" with "abc123"
And I fill in "Re-enter password" with "abc123"
And I press "Update password"
Then I should see "Hi Mrs, let's measure your impact"
I’ve read a article recently on Gherkin but I’ve not made an effort to learn it and use it in a project yet. You have used this development style with an Elm project with good results? And is this on top of any unit testing?
Have used Behat for years. I can’t say I’m really a strict BDD guy, although I’m strict that for every bug report I create a test that reproduces it, and only then fix the bug.
Have used Elm for about 1 year, and have used Behat for every Elm project I’ve done. I do continuous delivery, so many live app updates a day, and it’s extremely rare to introduce a bug.
I was looking for an end to end testing solution, and came across Cypress which I’ve been using successfully for a couple of months now.
I found it to be simple to set up, and comes with some neat features like time travel, automatic reloads when you change your tests, and automatic waiting for elements to load so you don’t need to add waits to your tests.
It will also take snapshots and videos of the test runs.
You can use the test runner locally, and the dashboard service to integrate with Github and CI/CD provider.
(I also just started using Percy for visual testing, which integrates nicely with Cypress, Github, CI/CD)