What (not) to unit test?

Learn
1

Lately I’ve been having a bit of a crisis when it comes to unit testing in a typed functional language like Elm.

Thing is types already cover a lot of ground. Also, the fact that functions tend to get really small and declarative encourages me not to cover them with tests at all. As a matter of fact, I’m not sure what testing a small declarative function would get me besides making me think twice by writing the same logic twice: once in production code and once in test code. Especially because that would come at the cost of having to maintain the tests afterwards.

I guess this issue is connected with the fact that I’ve been doing type driven development and not test driven development in Elm. In other words, I have to justify to myself what’s worth covering with tests when the production code is in place (and likely works since it compiles).

Since small declarative functions seem not worth testing, I’ve considered testing bigger units (more functions together). But then I would have to setup more stuff to be able to assert. And again, I have the feeling that it’s not worth it.

To conclude, I don’t feel I need to test drive Elm cause I can type drive. I don’t feel I need to cover small declarative functions because they are straightforward. I’m not sure covering bigger units pays off. Also, I don’t want to end up with a lot of tests that don’t really provide value and I need to maintain.

I’d love if you could share your reasons for testing in Elm and how you do that?

4 Likes
2

I had a very similar experience. Elm’s types are so helpful, that there rarely is a bug, even without tests.

You mentioned that you felt like you were implementing everything twice. I’ve come to the conclusion that that is definitely a bad thing to do. Your tests should never mirror your implementation. Why? Because if they mirror your implementation, every change in the functions needs to be reflected in the tests, or they break. I. e., you need to make the same change in two places. The whole reason we want tests is to enable us to confidently refactor, but this is not the case when every change is redundant.

As you said, in Elm you really feel that burden, since the types to almost all the work, and tests rarely offer additional benefits. There are some instances, though, when I’m not quite sure whether I have implemented a complicated function correctly. Sometimes the types don’t help. For example, I created a simple DateTime, but the day’s validity depends on the month and year. There, the types didn’t help, and I wasn’t sure whether the implementation was really correct, so I decided to test that constructor function.

To prevent me from copying the implementation, I always treat the functions under tests as black boxes, just as any other user would. I just put something in, and then I check whether I get the right thing out. For the DateTime, I could have input a bunch of interesting days. But there was an even better way to test.

I already new another, slightly more complex DateTime implementation. (I didn’t use that because of some details.) So I just decided to check whether my implementation returned the same result as that one. Instead of manually writing out a lot of values, I used fuzz testing, which is really awesome in Elm. I actually came to realize, that since I stepped away from unit testing everything and relying more on types, the tests I wanted to write tended to be fuzz tests. This is a greate sign, since fuzz testing virtually has to be black box testing, so there is little risk that I copy the implementation. (In case you’re interested, here’s the fuzz test I explained above.)

To summarize, I totally feel with you! But it actually feels great not to need to write tests for trivial stuff that can be taken care of with good typing. Instead, I have more time to think about the more difficult parts of the program, and those I can often test very nicely with fuzz testing.

1 Like
3

Testing is also a form of documentation and encoding assumptions.

I really like how the Rust book introduces the testing section (emphasis mine):

In his 1972 essay “The Humble Programmer,” Edsger W. Dijkstra said that “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.” That doesn’t mean we shouldn’t try to test as much as we can!

Correctness in our programs is the extent to which our code does what we intend it to do. Rust is designed with a high degree of concern about the correctness of programs, but correctness is complex and not easy to prove. Rust’s type system shoulders a huge part of this burden, but the type system cannot catch every kind of incorrectness. As such, Rust includes support for writing automated software tests within the language.

As an example, say we write a function called add_two that adds 2 to whatever number is passed to it. This function’s signature accepts an integer as a parameter and returns an integer as a result. When we implement and compile that function, Rust does all the type checking and borrow checking that you’ve learned so far to ensure that, for instance, we aren’t passing a String value or an invalid reference to this function. But Rust can’t check that this function will do precisely what we intend, which is return the parameter plus 2 rather than, say, the parameter plus 10 or the parameter minus 50! That’s where tests come in.

We can write tests that assert, for example, that when we pass 3 to the add_two function, the returned value is 5 . We can run these tests whenever we make changes to our code to make sure any existing correct behavior has not changed.

Testing is a complex skill, […]

The commentary applies to Elm as well, in my opinion.

If unit testing doesn’t seem useful for a situation, consider other forms of testing, like integration or golden master, whatever suits your verification needs.

I personally find now that for anything that is not a toy project, if I don’t add tests and encode my assumptions, when I come back I usually get lost and break things really easily. The same applies for any time you work with other people and colleagues. Maybe even more so. In the case of Elm, the effect is less prominent, if you actually added all of the type annotations (which I sometimes don’t do).

Learn each tool, and use it where it makes sense. Unit tests shine at some things, and are worthless at others. Also, something usually ignored is that the definition of a unit is flexible. Maybe a unit for something is a function. Maybe it is a data structure. Maybe it is a module. Quoting Wikipedia:

unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use.

3 Likes
4

Thanks a lot for chiming in @Y0hy0h and @joakin! I really appreciate the quotes and examples!

I see what you mean when you say that types can’t prevent all bugs. I agree with testing complex logic. At the same time, I’m not sure if simple functions should be tested.

For example the addTwo i = i + 2 function is something that I prolly would not test (even with a fuzz test). The types would not prevent me from writing addTwo i = i + 100. But is a test needed?

In the addTwo case I guess the test could save me from distraction. But I guess writing the correct implementation is easy. Also, what the function does can be deducted from the name and the types.

Kent Beck in Extreme Programming says

It is impossible to test absolutely everything, without the tests being as complicated and error-prone as the code. It is suicide to test nothing (in this sense of isolated, automatic tests). So, of all the things you can imagine testing, what should you test?

You should test things that might break. If code is so simple that it can’t possibly break, and you measure that the code in question doesn’t actually break in practice, then you shouldn’t write a test for it…

Testing is a bet. The bet pays off when your expectations are violated [when a test that you expect to pass fails, or when a test that you expect to fail passes]… So, if you could, you would only write those tests that pay off. Since you can’t know which tests would pay off (if you did, then you would already know and you wouldn’t be learning anything), you write tests that might pay off. As you test, you reflect on which kinds of tests tend to pay off and which don’t, and you write more of the ones that do pay off, and fewer of the ones that don’t.

That’s why I’m wondering if simple stuff should be tested. If not, what is simple and what not in Elm in your opinion?

3 Likes
5

As somebody that always uses BDD in other languages this is definitely something that I’ve pondered. My current thinking is that tests and types are two different mechanisms for constraining the behaviour of your system.

With Elm I favour adding constraints via types as the compiler applies those constraints everywhere, not just where I think to assert specific behaviour. The compiler is also far faster so for instance when I’m refactoring I get a shorter feedback loop than using a test suite. There are two places where I regularly switch to unit tests though, when I’m writing encoders/decoders and when I have logic that can’t easily be encoded in the type system.

You didn’t mention integration/acceptance testing but I find I lean on those far more for testing against business requirements. Full stack tests give me confidence that the whole system is working (vs FE unit tests) and make it easy to refactor the frontend without having to rework the tests.

I’m still figuring out the right balance so I’m interested to hear other people’s views.

3 Likes
6

For your addTwo example, I wouldn’t test it. Now in general, I am very aware of how even supposedly simple calculations can fail (off by one, sign error, etc). But the function addTwo is so trivial, that I fully understand it by reading the code. There is nothing that I’m unsure about. You don’t need to test constants, and I think similarly you don’t need to test code that feels trivial.

So my current approach is to work on the types first and foremost (as others have also mentioned). Then, if there are cases where I’m not entirely sure whether something works, I will write a test for that.

1 Like
7

Do you approach unit tests differently than integration/e2e tests? I always have trouble with that distinction, because it just feels like testing something more or less complex. But I wouldn’t approach it differently, i. e. I would still aim for black box testing from the user’s perspective.

Do you have a different experience?

8

When I am working on a project, I make extensive use of elm repl, testing functions that I am writing as I go. Sometimes, for a function whose body has many pieces, I will comment out the type signature (which I almost always write first), as a I incrementally add steps the body and test each addition: a |> b |> c |> d |> etc.

There is one big exception to this: tricky code. One example is the Latex parser-renderer which I wrote. The parser is very tricky and small changes in one place can have unexpected effects – the functions in the parser are a kind of mutually recursive package. Another examples is some math code where it is easy to mis-transcribe the formulas.

All this said, testing is a subject I worry about from time to time, and would love to hear more about it.

PS. One of the many wonderful things about Elm is how easy it is to do even major refactors, e.g. when you make a change in an important data structure. The fact that after minutes to hours of work following and fixing compiler errors one comes out of the tunnel with a working program is nothing short of amazing.

1 Like
9

Have you looked into fuzz testing in elm-test (I see that @Y0hy0h has mentioned it as well)? My preference is to use property-based/fuzz testing instead of plain old unit tests. These tests are more robust to implementation changes and I think they take care of your concern about writing the same logic twice.

These tests aren’t useful for everything, and can be tricky to write, but they’re definitely worth looking into. They work well for things like testing algorithms and complex data structures. For more straightforward code, I also find that type-driven development works well.

1 Like
10

Hey James! So you write tests only for what you call “tricky code” and leave the rest untested because the Elm type system gives you enough confidence. Did I get it correctly?

By the way, I loved the Minilatex presentations. Thanks for chiming in!

11

Hey Alex! I believe that fuzz testing is a great tool. At the same time I’m not sure what’s the return on investment when applied to simple, small, declarative functions.

For example I could fuzz the following functions but I’m not sure that would add much value.

type MyType = { one : Int, two : Int }

getOne : MyType -> Int
getOne { one } = one

updateOne : Int -> MyType -> MyType
updateOne i myValue = { myValue | one = i }

That’s a simplistic example but I don’t find my Elm code getting much more complex than that. When it does I do believe unit testing is a great tool. When it doesn’t maybe it’s better to reach for other techniques (e.g. code inspection)?

What do you think about that? Do you have a specific strategies when testing Elm code?

12

If the functions are really that simple, there’s really no need for testing.

My strategy is roughly:

  • Make impossible states impossible (as per Richard’s talk: https://www.youtube.com/watch?v=IcgmSRJHu_8)
  • Skip testing simple functions
  • Use unit tests and property tests for complex functions
  • Mostly manual tests for UI/integration (I think automating this becomes cost-effective a lot later than most people seem to think).

(Note that even in JavaScript, I wouldn’t test really simple functions despite the lack of static types. I got far more value out of writing FP-style JavaScript.)

4 Likes
13

Thanks for the list, I love it!

14

My hints for accepting part of code as a test unit:

  • it’s hard to predict, function has multiple outcomes. It may be because of some algorithm doing lots of calculation or data reasoning, but that may be also just too loosy types like multiple Maybes
  • I fear or I know I will fear of changing some functions
  • sequence of actions for data, i.e. multiple transformations, like parsing data or generation of some document/code/report/long string/whatever
  • correctness in very important parts, like business logic, moneyz and so on

If you’re doing mostly CRUD then yeah, type system has your back.

15

To give a concrete example, one thing I test outside the type system is the symmetry of JSON encoders and decoders. I set up a test with a set of values to test encoding and decoding roundtrip.

The constraint I am modeling, in this case, is: The combined encoder and decoder functions should always return the same value which is given as argument.
I have not yet found a way to model this in the Elm type system.

On a more general level, I have not found a way to model all constraints or invariants in my programs in the Elm type system, the encoder-decoder pairs are just one example.
I found many such cases when modeling game mechanics for in video games.

16

What is the result of addTwo if the parameter is the maximum value of integers?

Even a deceivable simple function can carry huge problems.

With that said, I am in the same mindset as the OP. Mainly my tests are for Json decoders.

1 Like
17

Thanks for sharing the list @Namek!

18

That’s a nice symmetrical property to test. Thanks for sharing @Viir!

19

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.