Elmer: An Elm Testing Tool

eeue56 2017-12-13 19:24:52 UTC #6

The unique things that Elmer has are that it lets you completely mock the outcome of effects (unlike elm-testable, which tries to help you simulate the outcome of effects), it provides more general spies which lets you do isolated testing in a way that’s not otherwise possible in Elm, and it tries a different API for mocking HTTP requests.

Right, cool! I’d love to see a comparison of the two approaches side by side in order to better say look at the different attempts to see what works and what doesn’t. An illustrated example comparison goes so far in figuring out what’s useful, especially when there’s a lot of docs to read.

I’m not sure why there’s such a resistant response here. The current combination of elm-test, elm-html-test, and elm-testable has a lot of problems at the moment and doesn’t yet account for all the needs of TDD. I’m happy to see attempts at improving testing in Elm, and I hope it will be useful in understanding and solving the problems with the currently recommended tools.

There is not a resistance response – like I stated, I could not tell from the examples the unique offering provided by Elmer. The README is rather huge and I’ve read it a couple of times and come away unsure of the differences. The only resistance is to the approach – I believe much more could’ve been done through a conversation! So we are in agreement, both you and I. We can always do better and make things better, and putting heads together is more productive than re-inventing a wheel that doesn’t need re-inventing, in the case of the underlying code.

Edit: updated my previous post to hopefully better reflect this!

eeue56 2017-12-15 00:25:12 UTC #7

I decided to take another look, focusing on the spy section. It’s interesting!

I have some questions though:

parseTest : Test
parseTest =
  describe "when the string is submitted"
  [ test "it passes it to the parsing module" <|
    \() ->
      let
        spy = Elmer.Spy.create "parser-spy" (\_ -> MyParserModule.parse)
      in
        Elmer.given App.defaultModel App.view App.update
          |> Elmer.Spy.use [ spy ]
          |> Elmer.Html.target "input[type='text']"
          |> Elmer.Html.Event.input "A string to be parsed"
          |> Elmer.Spy.expect "parser-spy" (wasCalled 1)
  ]

In this example, it seems like parser-spy couples your test code to the implementation detail of your view. What’s the benefit here of making sure the function is called rather than checking the end result? I’m familiar with spys in other languages: but in those languages, they generally focus on providing a way of inspecting something with mutation. In this case, wouldn’t the just testing the view/model be enough? Or some other unit tests?

The most interesting part, to me, is actually the Http section! Elmer lets you provide some stubbed responses by doing:

Elmer.Spy.use [ Elmer.Http.serve [ stubbedResponse ] ]

It’s a great idea and I really dig it! I think this API could be implemented via the elm-testable route, so that’s great too!

I’m not too sure about faking the view functions:

thingsViewSpy = Elmer.Spy.create "things-view" (\() -> ThingsModule.view)
    |> Elmer.Spy.andCallFake (\_ ->
            Html.div [ Html.Attributes.id "thingsView" ] []
    )

What is the benefit of faking the view function in this way? Or rather, what’s the use cause for this? If you haven’t yet implemented things-view, isn’t it simpler to just implement things-view than faking it?

Either way, cool stuff! Some of it should be implementable without any kernel code. I have some ideas about a safer long-term way of implementing spies, without need for kernel code, which you should be able to do now. Would love to discuss in #testing on Slack!

brian-watkins 2017-12-16 19:36:15 UTC #8

Three things:

(1) I think Aaron did a nice job highlighting some of the differences between elmer and elm-testable – Thanks! Here’s a link to a medium article I wrote that gives another example of test-driving an elm application:

(2) It’s a great suggestion to use elm-html-test for the low-level html stuff in elmer; I will definitely look at that. Reducing the amount of native code in elmer is, of course, something I’m very interested to do.

(3) For spying on functions: Yes, if you start spying on functions during tests, you are starting to expose some implementation details, so you’ll want to do this with care.

It might be helpful here to distinguish two types of details: the ‘inner workings’ of a module and its ‘exposed interface’. Tests will probably need to know about the exposed interfaces of software components, but should try not to know about the inner workings of those components.

For example, when you’re writing ‘end-to-end’ style tests with elmer (edit: not a good use of ‘end-to-end’ here; I mean unit tests that describe the public behavior of a model, view function, and update function in interaction with each other), these tests will need to know about the interfaces of the app’s high-level dependencies – for example, a function that produces a command to make an http request. Usually, you can just spy on and stub functions that produce the relevant commands and subscriptions to simulate the side effects necessary for describing the behavior of your app under specific conditions. So your tests know some details about these interfaces but not about the inner workings of the app.

As an app becomes large, you might also find a need to spy on functions that represent exposed interfaces between different parts of the app. I’m interested in agile architecture design patterns like hexagonal architecture and clean architecture, and I’m a big fan of the SOLID principles. Those kinds of strategies recommend dividing your application into smaller, decoupled components. Once you do that, you might want to divide your tests along similar lines, describing the behavior of each component independently. Elmer’s capacity to spy on and stub functions can be useful when testing such components in isolation.

For example, you might want the UI portion of your app (with its own model, update, view) to be decoupled from the mechanisms by which the application gathers data. This is a good thing, since these two portions of your app will probably change for different reasons, and separating them makes each easier to understand and change. But then, your tests should probably respect this decoupling. You don’t want the tests of your UI to presuppose any particular mechanism for gathering data, since if you change that mechanism you’ll then need to update those tests. So, in the tests of your UI, you might spy on and stub the functions the represent the exposed interface by which data is gathered, and vice-versa. This lets you ensure that each component respects the interface between them and more easily simulate conditions to help you describe specific behaviors of the component under test.

So, yes, to spy on a function during a test you’ll need to know about some implementation choices. If you follow the SOLID principles or other agile architecture patterms, you’ll want to give your code a high-level structure composed of modules that are decoupled and easy to change. In that case, the tests may need to know about the interfaces exposed by those components, but they should know as little as possible about the inner workings of each component. And, I should stress, I think these kinds of patterns are something code needs to adopt only when an application starts to become big. For simple elm apps, just spying to stub commands and subscriptions is enough, and I think elmer lets you do that without needing to know too much about the inner workings of the app.

If there are ways to accomplish the kind of spying that elmer enables without the use of native code, I’d be happy to hear it.

rtfeldman 2017-12-17 02:41:14 UTC #9

It never occurred to me to push for spies in elm-test because to be totally honest, I eventually came to regret the tests I wrote using spies in other languages.

Can you share some Elm code you’ve written that uses spies to good effect? I’d like to discuss in the context of real-world code!

brian-watkins 2017-12-18 01:45:53 UTC #10

You can check out the medium article I referred to above for a straightforward example. One test spies on the WebSocket.send function so the test can make expectations about its arguments. This kind of test ensures that our code is using the WebSocket.send function in the way we expect. Another test uses a spy for the WebSocket.listen function to provide a fake implementation that allows us to simulate receiving data over the websocket during the test.

Here’s another example. Suppose we write a magic eight ball app. You ask it a question and it returns some answer, like “The future is hazy”, “Definitely”, etc. We might structure the app so that the user interface is decoupled from the module that actually determines the answer. We could even apply the dependency inversion principle and provide the update function with a reference to the relevant function from the module that fetches the answer. That way, our UI module doesn’t have to know how an answer is fetched, it only needs to worry about managing the UI – gathering the user input, passing it to the function, and displaying the result.

When we write the test to drive out this code, we’ll use a spy to represent the module that fetches an answer. We could just use a normal function, but a spy allows us to easily assert that the user interface passes the right arguments to this function. Let’s say that function looks like (Result () EightBallAnswer -> msg) -> String -> Cmd msg, where the string is the question text. Here’s the test:

module Web.EightBallViewTests exposing (..)

import Test exposing (..)
import Expect
import Elmer
import Elmer.Html as Markup
import Elmer.Html.Matchers exposing (element, hasText)
import Elmer.Html.Event as Event
import Elmer.Spy as Spy exposing (Spy)
import Elmer.Spy.Matchers exposing (wasCalledWith, anyArg, stringArg)
import Elmer.Platform.Command as Command
import Web.UI as App
import Types exposing (EightBallAnswer(..))

requestAnswerCommandSpy : Result () EightBallAnswer -> Spy
requestAnswerCommandSpy answer =
  Spy.createWith "request-answer-command" (\tagger _ ->
    tagger answer
      |> Command.fake
  )

answerTests : Test
answerTests =
  describe "when a question is given" <|
    let
      state =
        Elmer.given App.defaultModel App.view (App.update <| Spy.callable "request-answer-command")
          |> Spy.use [ requestAnswerCommandSpy <| Ok Hazy ]
          |> Markup.target "#question"
          |> Event.input "Will I eat pizza soon?"
          |> Markup.target "#submit-question"
          |> Event.click
    in
      [ test "it sends the question to the question service" <|
        \() ->
          state
            |> Spy.expect "request-answer-command" (
              wasCalledWith [ anyArg, stringArg "Will I eat pizza soon?" ]
            )
      , test "it presents the answer" <|
        \() ->
          state
            |> Markup.target "#answer"
            |> Markup.expect (element <| hasText "The future looks hazy.")
      ]

Here, Spy.createWith provides the function to spy on. Spy.callable provides a reference to that function.

So, in general, I think spies help you describe the behavior of the code under test with respect to its collaborators; we can assert that the code under test calls those collaborators with the right arguments at the right time. We can also provide fake implementations to functions we spy on, and this lets us stub data to simulate conditions that help us describe the behavior of the module we’re testing.

Not every app needs to use spies in this way (and the example I’ve given is certainly contrived). But once an app gets large, it can be a good idea to decouple parts of the software system, and in that case spies can be handy as a means to make sure the parts interface with each other in the right way.

rtfeldman 2017-12-19 05:27:49 UTC #11

Thanks for the examples! ️

As I see it, tests are useful for two reasons:

Helping guide implementation (test-first development)
Identifying regressions early

As Spies rely on implementation details, it’s not possible for them to help guide implementation, so we can rule out benefit 1 categorically. What about Spies a way to catch regressions?

The answerTests suite has two tests. One checks that "Will I eat pizza soon?" is sent to the question service. The other checks that the correct answer was presented to the user.

From a regression-catching perspective, the Spy test is redundant. If the wrong arguments get sent to the service, the service won’t (can’t) respond with the correct answer, so both tests will fail. This means the Spy test neither helped guide our implementation nor helped us catch regressions we wouldn’t have already caught otherwise! It seems to me that the Spy test does not benefit us, but does have a maintenance cost, so our test suite would be improved if we deleted it.

The medium article tests that "Hello!" was sent to "ws://testserver.com". I can see the utility of testing that, but using a Spy for that seems unnecessarily brittle.

The regression we want to catch is if clicking the button no longer sends Hello! to a websocket connected to "ws://testserver.com". Testing this with a Spy creates two undesirable scenarios:

The implementation of WebSocket.send changes such that it no longer does what we expect. In other words, it no longer sends "Hello!" to the expected websocket, even when we continue passing it the same arguments. We wanted the test to fail in this scenario, but it continued to pass.
WebSocket.send gets renamed, we update our implementation, but the test still fails even though our implementation works - meaning we have to spend time updating the test for no benefit. We wanted the test to pass, but instead it failed.

It seems to me that it would be strictly better to write a test which actually verifies that the data gets sent to the Websocket. It would catch all the relevant regressions the Spy-based approach would catch, but it would not have either of these problems!

Granted, this would require creating a Websocket-specific testing API to do this (the way elm-html-test does for testing Html), but I think that is a better goal to pursue than Spies. I think these two examples generalize - specifically to the claim that for any Spy test, there exists a better test that doesn’t use Spies - and that we should follow the example of elm-html-test rather than creating a system for Spying.

That way we can catch the same regressions without the downsides!

marick 2017-12-19 18:56:00 UTC #12

Growing Object-Oriented Software, Guided by Tests is a good book on the mock/spy approach.

I was a reviewer on the original mock objects paper, and totally missed the point. It took me about eight years and a fair amount of experimentation before I understood and became fond of mocks.

I find that thinking of mocks/stubs/spies in terms of implementation details is misleading. I’ve come to think of them as having the same relationship to product code as lemmas do to proofs. Here’s an explanation from my clojure testing tool’s documentation.

I have a hunch that, under this interpretation, mocking fits better with FP than with OO. Hard to know, though.

brian-watkins 2017-12-23 19:04:35 UTC #13

Let me explain a bit more how the test I gave in the post above describes behavior and drives out implementation. First, you write that test. It will fail to compile. You do the simplest thing to make it compile. Then you run the test and see that it fails in the way you expect. Finally, you do the simplest, reasonable thing to make that test pass. If you follow this process, you’ve done a few cool things:

You’ve described the behavior of part of your software system. In this case, we’ve described how the UI takes input from the user and passes it to a collaborator (another part of the software system) that fetches the eight ball answer.
In writing the test, we’ve made decisions about the interface we expect from that collaborator. This is great since when we go to write the tests for that collaborator, we won’t be tempted to build things that won’t be used.
To make these tests pass, we have to do some work. That work is the implementation driven out by the test. The test doesn’t know what this implementation is – and that’s a good thing since it gives us freedom to refactor later, if we need to.
Because we used a test double (more on that below) to stand in for the real collaborator, we’ve effectively decoupled the part of our software we’re describing with these tests (the UI) from the part that fetches the eight ball answer. That’s really good, since in implementing the UI I don’t have to worry about how the data is fetched. As a result, my UI module has a chance to be cleaner and easier to understand.

The tests in my post above are, I take it, a fairly straightforward example of the style of TDD known as the London School or ‘mockist’ TDD. (I’m not advocating for this school over any other style of TDD, just providing extra context). Here’s a good blog post that discusses the ‘mockist’ style in relation to other approaches:

@marick also provides a good explanation of this style of testing in his link – programming by wishful thinking is a nice way to describe it.

As I suggested in my earlier post, it’s not necessary to use an elmer Spy in this test. We just need a test double to stand in for the collaborator during the test. A spy is a type of test double that records the calls made to it. This is a great blog post on the different types of test doubles:

Using a spy simply makes it easier to write these tests. An earlier version of elmer did not have spies at all, and you could still write the same tests – in your code you’d just need to inject any functions that produce commands or subscriptions so they could be replaced during a test with functions that returned fake commands or subscriptions. I added spies because it facilitates certain testing strategies without necessarily requiring the code to inject all these dependencies. Dependency inversion seems to me to be one of those patterns that really pays dividends only in larger apps. So it’s good, I think, that elmer doesn’t force you into that pattern unless you really want to adopt it.

@rtfeldman suggests two other things. First, I believe he’s suggesting that end-to-end/integration tests give you more confidence that the software works than unit tests. That makes sense to me. However, I wouldn’t want my test suite to contain only integration tests – in my experience these tend to be slower, sometimes flakey, and more cumbersome to set up for testing edge cases. I’d advocate combining a few high-level, happy path integration tests with many more unit tests that describe the behavior of each part of the software system in isolation. And for those unit tests, you will need test doubles to stand in for collaborators.

I take it that the other point @rtfeldman gets at has to do with the extent to which there should be library-specific testing apis. To me, there are two challenges in writing tests for elm apps: managing side effects during the test and inspecting (when necessary) values represented by opaque types. To my mind, it should be the test framework that helps to manage side effects during the test, since this problem is (for the most part) the same across libraries – that’s what elm-testable does (I take it) and that’s what elmer does (not with spies, of course, but with the ability to send and process fake commands and fake subscriptions during a test). Some library-specific testing tools may still be necessary to manage side effects, though – elm-lang/html in particular complicates things since apps that use it subscribe to dom events without using Platform.Sub.

I think library-specific testing tools are necessary when you need to inspect opaque types used by those libraries. For example, in elm-lang/http, it’s not possible to inspect one of the core values – the request record. So, in writing elmer I had to add some native code that looks into that opaque type and makes the values hidden by it inspectable during the test. Now, it would be great if library authors took testability into account in designing their apis, adding functions that made it possible to inspect important values otherwise hidden by opaque types. Since, for various reasons, that might not always happen, I’m looking into ways to make elmer more extensible so that it’s easy to write library-specific plugins that help with making expectations about values hidden behind opaque types.

rtfeldman 2017-12-24 00:07:49 UTC #14

Thanks again for the thoughtful reply @brian-watkins!

The approach Abelman and Sussman called “programming by wishful thinking” is not about testing.

Suppose I’m writing a function and I think “I really wish I could call a depth : Tree -> Int function right about here” - I can follow “programming by wishful thinking” like so:

depth : Tree -> Int
depth tree =
    Debug.crash "TODO implement"

My function can call this, and I can continue on my way. (In Elm 0.19, Debug.crash won’t be allowed outside --debug mode, so I can’t forget to implement this function for real, even if I never write a single test.)

Like using a Spy, having a Debug.crash in the depth function will cause the test to fail when my original function calls it. However, now when I finish everything up, my original test won’t be brittle to implementation detail changes such as my refactoring to no longer rely on the depth function.

This makes Debug.crash a better choice for “programming by wishful thinking” than Spies would be. That means introducing Spies increases API size but does not introduce a better way to do things.

in my experience these tend to be slower, sometimes flakey, and more cumbersome to set up for testing edge cases

That’s been my experience with tests where race conditions are possible, e.g. Selenium tests in browsers.

However, elm-html-test tests are exclusively Elm code inspecting exclusively Elm data structures. They require no more work to set up than any other elm-test test, they run just as fast, and they are 100% deterministic; I’m not sure how they could even theoretically flake.

My point is that we can do the same thing for testing the opaque value Cmd as we’ve done for testing the opaque value Html. (This is one of the benefits of representing effects as data!) Granted, we haven’t done that work yet, but I claim that once we have, those Cmd tests will be just as flake-proof, easy to set up, and fast to execute as any typical elm-test test.

In a world where we have elm-html-test-esque ways to test commands, I think the claim “for any test written using a Spy, a better test could be written that doesn’t use a Spy” becomes true. Debug.crash is better for guiding implementation where the function you want to call doesn’t exist yet, and Cmd tests are better for detecting regressions.

As API designers, I think it’s just as important for us to decide what goes in the toolbox as it is for us to decide what doesn’t. I think the toolbox that will lead to the best test suites is one that contains a full menu of of Cmd tests, and no Spies.

brian-watkins 2017-12-27 16:25:29 UTC #15

(1) When I mentioned slow, flakey, cumbersome to set up tests, I was specifically referring to “end-to-end/integration tests” and @rtfeldman’s comment, “It seems to me that it would be strictly better to write a test which actually verifies that the data gets sent to the Websocket,” made me think of that – I’d call a test where the app connects to a websocket an integration test, and such tests, while still being useful in certain cases, in my experience can be slow, flakey, and cumbersome to set up. Obviously, elm-html-test allows you to write different kinds of tests that do not have this problem. And so does elmer.

(2) @rtfeldman suggests that a good approach to dealing with side effects in tests is to follow the pattern of elm-html-test and find ways to inspect the opaque value Cmd. I tried something like this in the early days of working on elmer and I ran into a few challenges. First, I believe, unlike the opaque value Html, libraries can choose to store values in a Cmd however they like. So, if you compare the Cmd generated by WebSocket.send vs that generated by Http.send, you’ll see that these have different structures for the data stored within them. Each library that uses a Cmd will thus probably need its own specific testing tools, but maybe that’s ok. The second, deeper, problem is that some libraries might store relevant data in a Cmd in a way that is inaccessible. For example, Http.send from elm-lang/http stores a function that captures the values it needs (the details about the request). That could be the right decision as far as that library is concerned, but I’m pretty sure it means it’s not possible to inspect the details of the request (details I might like to test), based solely on the Cmd.

For these reasons, elmer takes a different approach. Elmer manages effects for you during a test, passing messages from commands or subscriptions to the update function as necessary. Test writers don’t test the update or view functions in isolation; instead, test writers describe the behavior associated with a model, view function, and update function working together. To manage effects during the test, elmer just knows how to process fake commands (and subscriptions). For example, test writers can pass a tagged value to Elmer.Platform.Command.fake : msg -> Cmd msg to generate a fake command. When elmer sees this kind of command, it knows to get the tagged value and pass it to the update function. When you test a part of your app that deals with a Cmd in some way, you need to do one of two things: structure the code so that during the test you can pass a fake command instead of the real one (ie, follow the dependency inversion principle with respect to cmd-generating functions) or use a Spy to replace the cmd-generating function during the test with a function that returns a fake command. (Again, spies are not necessary; they just make writing certain kinds of tests easier.)

There are certainly other approaches one could take, but I’ve found this one to be quite nice. I don’t have to worry about how some particular library has decided to structure data in a Cmd. My tests describe the high-level behavior of my code – ie that it passes the correct arguments to the cmd-generating function (if that’s necessary to test) or that it does the correct thing when the command is processed. My tests are all pure functions, so they’re super fast and flake-free. I am completely faking out Cmd and Sub, though, so, of course, as I mentioned earlier, it’s a good idea to supplement elmer tests with a few high-level, happy path, end-to-end/integration tests to provide confidence that the system works as a whole.

rtfeldman 2017-12-27 19:33:39 UTC #16

Yep, I agree. Fortunately, there are a fixed, small number of Cmd values in the language!

Oh, for sure - it’s impossible to inspect a function, any Task can contain a function if Task.andThen was used to create it, and many Cmd values come from tasks. So there are lots of ways this can happen.

Hm. If I understand this right, here’s the sequence of events:

Test author comes up with a Msg value
Test author passes that Msg value to fake to generate a fake Cmd
The test library takes the Msg value provided by the author, and passes it to update
Test author runs assertions on the result of update

What value is step 3 adding? I can call update myself!

That way my workflow is strictly shorter:

Test author comes up with a Msg value
Test author passes that Msg value to update
Test author runs assertions on the result of update

What regressions would I expect to be caught by adding the layer of indirection of having the library call update instead of calling it myself?

A third option: offer an API for running expectations on that command and simulating its output.

For example, here’s a rough sketch of how such an API might look for HTTP:

cmd
    |> Expect.Http.fromCmd
    -- Verify things about the HTTP request in this command
    |> Expect.Http.header "X-Requested-With" "XMLHttpRequest"
    |> Expect.Http.url "/foo"
    -- Convert to an Expectation when finished verifying
    |> Expect.Http.toExpectation

You could read this as “Expect that this command contains an HTTP request which has this particular header and this particular URL.” The Expect.Http module would support checking all the various ways you can construct a HTTP request.

That module could also include APIs for running expectations through Task.andThen chains by specifying what the server sent back as a response, e.g.:

cmd
    |> Expect.Http.fromCmd
    |> Expect.Http.header "X-Requested-With" "XMLHttpRequest"
    |> Expect.Http.url "/foo"
    -- Expect this HTTP request to chain into a second HTTP request
    |> Expect.Http.toTask expectedResponse
    |> Expect.Task.andThen
    |> Expect.Http.fromTaskExpectation
    -- Run checks on the second HTTP request
    |> Expect.Http.header "X-Requested-With" "XMLHttpRequest"
    |> Expect.Http.url "/bar/baz"
    |> Expect.Http.toExpectation

This style of API would permit:

Testing Cmd values which wrap HTTP requests
Testing HTTP requests chained together with Task.andThen
Testing how HTTP responses get converted into commands (not shown in example)
Doing all of these without needing a browser environment or actually running any effects

These are the things I really want in a Cmd testing solution. I want to verify that when a certain Msg gets passed to update, the result includes a HTTP request being sent to a particular URL. If that doesn’t happen, regardless of implementation details, I know I have a regression!

You can imagine modules like Expect.Websocket and Expect.Dom for the other entries in the fixed set of Cmd values in Elm.

This approach for testing commands really appeals to me. It can catch any regression a Spy can, but without the false positives or false negatives!

turboMaCk 2017-12-30 16:23:57 UTC #17

I agree with this very much. And this is the same advice I always give when someone brings testing. That being said, I think it’s more about flaws of whole TDD approach than about this specific package. If the point of this tool is to improve TDD experience in elm I see it as being a beneficial contribution to elm ecosystem.

mgold 2017-12-30 19:13:21 UTC #18

From the Fowler blog post:

With the classic approach, however, any tests of client objects can also fail, which leads to failures where the buggy object is used as a collaborator in another object’s test. As a result a failure in a highly used object causes a ripple of failing tests all across the system.

In OOP, you have highly interconnected systems that can be brought down like this. In FP, everything acts only by its return value. This kind of torpedoes the “state/behavior validation” dichotomy. There’s no state to validate unless its returned, which is the only thing you can validate. Behavior validation is less than useless: whether or not a function calls another one is an implementation detail. Such a call can’t set state somewhere else, so the only thing it can do is ease computation of the return value.

The other thing that strikes me about Fowler’s quote is how abstract and speculative it is. He goes on to describe how mockists and classicists can’t agree on whether this is a problem – a sure sign, to me, that hard evidence is lacking. Elm has a philosophy of gathering evidence and being data-driven, rather than worrying about everything that can go wrong up front. I would summarize it as, “it’s not a problem until it becomes a problem”. I’d say a testing concern “becomes a problem” when either you can’t be confident in the correctness of your app, or you can’t guide your implementation with tests. (This connects to Richard’s two reasons for testing in the 11th post in the thread.)

All of that is to say that purity and immutability dramatically change the approach and philosophy of testing. We don’t have the “gorilla holding the banana in the jungle” problem, so we don’t need mock jungles and gorilla spies (guerrillas?). But we do need to pass arguments to functions, and elm-test takes the most salient idea from prior art FP testing – fuzz tests – and makes them super easy to use. And once we’ve called the function under test (not a whole system, just a function!), we need to validate the return value. Even a command is just a value that can theoretically be validated, but for various reasons those values are opaque and cannot be inspected.

That’s where Richard’s Expect.Http and similar idea comes in. I think that, in broad strokes, it’s a good idea since it makes unit testing a function that returns Http.Request look like any other unit test. The specific API Richard proposes requires a fairly large number of function in Expect.Http, all of which require kernel/native code right? It also means a single expectation and test includes an unlimited number of validations. I wonder if a better API would be to convert the request to a record, and then use existing expectations on its fields?

cmd
    |> Expect.Http.fromCmd
    |> .header
    |> Dict.get "X-Requested-With"
    |> Expect.equal (Just "XMLHttpRequest")

I mean, maybe not, but this would take a very direct approach to turning an opaque value into an inspectable one.

describe "my HTTP Request"
let
   req = Expect.Http.fromCmd cmd
in
   [ test "has the correct X-Requested-With header" <| \_ ->
      req.header
      |> Dict.get "X-Requested-With"
      |> Expect.equal (Just "XMLHttpRequest")
    -- , other tests here
   ]

brian-watkins 2018-01-07 22:55:46 UTC #19

I don’t think I did a good job in my earlier post explaining one should use Elmer.Platform.Command.fake, and @rtfeldman’s question about the value of step 3 brings this out.

As I’ve said before, to practice good TDD, I want to write tests that know as little as possible about implementation details. Why? Because as I add new features or realize better ways to structure my code, I’ll have to change the details of my implementation. When I change those details I want to run my tests; if those tests pass then I know my software still has all the behavior I’ve described with those tests. In this way, my tests provide me with confidence to change my code. But if my tests know a lot about implementation details, chances are that when I change how my software is implemented, I won’t be able to run those tests – I’ll have to spend some time changing them so they’re in sync with the new details of the implementation. But then my tests won’t be able to give me that confidence anymore, that fast feedback as to whether my software still has all its expected behavior. And if that’s true, then the value of those tests has been significantly diminished.

When it comes to writing tests for an Elm application, I want my tests to know as little as possible about the shape of the model, the messages passed around by the application, the particular functions that are called (besides view and update), and other implementation details. If I can write tests like that, then I’ll have a lot of freedom to refactor my code, and I’ll do so with confidence provided by running those tests.

This gets me back to @rtfeldman’s question: Of course I could write a test that called the update function myself – but that’s precisely what I want to avoid. To call update from a test, the test would need to know details about my model and about the messages passed around my app. If my tests know those things, then they will have to change should I refactor my code in a way that modifies these details; and if that’s so, then my tests won’t be able to give me confidence to change my code.

Take a look back at the examples I’ve provided in this thread. You’ll see that whenever Elmer.Platform.Command.fake or Elmer.Platform.Subscription.fake is used, there is no reference to any particular Msg value or tagging function. What Elmer test writers do is inject a command (or subscription) generating function into a test that then calls Elmer.Platform.Command.fake or Elmer.Platform.Subscription.fake with a reference to the Msg value used by the code. For example, consider WebSocket.listen. During the test, you would replace this function (by injecting it or by using a Spy) with something like this:

fakeWebSocketListen : String -> (String -> msg) -> Cmd msg
fakeWebSocketListen host tagger =
  Elmer.Platform.Subscription.fake "fake-websocket-listen" tagger

My test does know that I am using WebSocket.listen – but it doesn’t have to know about the details associated with how my app uses it. The test doesn’t care what the tagger is, it just passes the reference to Subscription.fake so that Elmer can process the subscription data just as the Elm runtime normally would. In this way, I can describe the high-level behavior of my app while my tests know as little as possible about the low-level details.

If you’re not a fan of TDD then Elmer might not seem worthwhile. This project has been a fun way for me to learn what the kind of TDD practice I follow might look like in Elm.

rtfeldman 2018-01-08 02:50:57 UTC #20

Thanks for clarifying! I think we agree that it’s desirable to be able to test commands and subscriptions without having to couple the tests to update, Model, or Msg.

I want to write tests that know as little as possible about implementation details.

This paragraph describes precisely what Spies do - testing internal implementation details such as which other functions are being called - and then argues forcefully that doing this is a mistake.

I don’t know how to read this as anything other than a full-throated condemnation of Spies, so I’m confused why they still seem like a good idea to include in the API. Don’t these supporting arguments point to the opposite conclusion?

brian-watkins 2018-01-13 03:36:40 UTC #21

I’m not sure we agree. When I’m using Elmer to test-drive an application, I’m not testing real commands and subscriptions directly at all. Instead, my tests describe how my code interacts with the functions that produce commands (like ‘Http.send’ or ‘WebSocket.send’ provided by effect manager libraries). That’s a lot different from taking a Cmd value and making expectations about it, as the API @rtfeldman describes above does.

On the other point @rtfeldman makes: I said I want my tests to know as little about implementation details as possible, but that’s not to say that they should know nothing about the unit under test. How could they? My tests at least need to know how to exercise that unit in code, after all. In general, unit tests need to know this much: the public interface of the unit I’m testing (it’s exposed functions) and the public interfaces of those other units it depends upon. The implementation details I don’t want my tests to know about are the choices I make ‘behind’ the unit under test’s public interface to give it the behavior I’ve described with my tests.

When I’m writing a test with Elmer, I’m describing the behavior of a unit (usually a model, view function, and update function working together) in isolation from the other units it depends upon (like the elm-lang/http module). To accomplish that, during the test, I’ll need to provide stand-ins (test doubles) for any other collaborating units. A spy is just a special kind of test double that records calls made to it. I’ll use spies to help me describe and verify the behavior of the unit under test with respect to those other units it depends upon. Writing good tests involves making good (sometimes difficult) decisions about where and how to draw the lines between units, what to test about those interfaces, and so on. For more on this, see post 8 above, where I addressed a similar point.

rtfeldman 2018-01-13 15:26:08 UTC #22

As you noted earlier, “I want my tests to know as little about implementation details as possible” - and it is possible to completely exercise the behavior of functions using unit tests without knowing the public interfaces of other functions they depend on.

Certainly unit tests need to know the public interface of the unit they’re testing.

Why should they need to know the public interfaces of the other functions they depend on? It’s an implementation detail that’s not required to completely test their behavior.

brian-watkins 2018-01-20 15:44:11 UTC #23

Let me give you two examples to hopefully clarify my point:

Suppose you’re testing a particular function: myFunction: CoolRecord -> AwesomeRecord -> String. In order to test this function you’ll need to pass in a CoolRecord and an AwesomeRecord and you’ll need to know how to construct these. When I say that a unit test needs to be aware of the dependencies of the unit under test, I’m thinking that CoolRecord and AwesomeRecord are things the test would need to know about. Now myFunction could call all sorts of functions in the process of generating a String given a CoolRecord and an AwesomeRecord. My test doesn’t need to know about those functions called by myFunction. Like I said above, the test only needs to know about other units that are dependencies of the unit under test.

Elmer allows me to adopt a higher-level testing strategy, where I can often treat ‘the unit under test’ as some unit of application behavior that a user would encounter. So, if I’m writing an RSS Feed reader, I might structure my tests around ‘viewing a list of feeds’, ‘reading a particular feed’, ‘reading a particular feed item’ etc, with particular tests describing the expected behavior in each of those cases. In order to write these tests, Elmer needs to know about an initial model, the view function, and update function of my app. I’ll then need to do some stuff in my test setup to get me to the right place of the app to exercise the behavior (click on a button or input text etc). Then I can make expectations about the behavior. However, in most cases, there are additional dependencies that need to be in place to simulate the behavior I want to describe. For example, if I’m writing a test that describes the behavior of viewing a list of feeds, I’ll need to stub the http request that fetches the list of feeds. I do that by treating Http.send as a dependency that I need to provide, because (1) I don’t want to make http requests during my unit tests and (2) Elmer doesn’t know how to process Http.send. Instead, with Elmer I can inject a fake command to replace Http.send during the test (there are various strategies to do this which I’ve talked about above), which allows me to make sure the proper request is being sent (that’s part of my application’s behavior when a user attempts to view a list of feeds) and which allows me to stub various return values (successes, errors) so I can describe the expected behavior under various conditions.

Given this kind of test setup, I have tons of freedom to refactor my code and to experiment with different approaches for structuring the app. My tests do expect there to be an http request made, and they need a reference to a default model, a view function, and an update function. But that’s about it; the tests don’t need to know about any other functions called. It’s been fun to learn Elm alongside writing tests with Elmer, because as I learn more about how to structure Elm apps, I can refactor the code and just re-run my tests to be confident my app still has all the behavior I expect.

rtfeldman 2018-01-20 23:18:32 UTC #24

Sure! My only point is that spies shouldn’t be one of them, because the alternatives lead to strictly better tests.

I think we’ve now each made our positions clear, and we’ve gone several posts without making any progress, so I don’t suppose there’s much point in belaboring the thread.

Thanks for politely engaging on this topic! ️

brian-watkins 2018-01-23 11:42:59 UTC #25

I encourage people to keep an open mind. I know as I’ve learned more about TDD and testing strategies in general, I’ve found that test doubles, including spies, can be used in both bad ways and good. When used correctly, test doubles can make some kinds of tests easier to write, and help you test elements of a software system in isolation.

I’m happy so far with the kinds of tests I can write today with Elmer, but I would be happier still if better approaches present themselves in the future.