I released a new elm-review package: jfmengels/elm-review-test-values
I have explained the why and how of the technique in the blog post below. I hope you’ll find it useful 
5 Likes
Nicely written post, it’s very clear and thorough!
Regarding defining test cases directly in the module, I’m of the same opinion that it allows for testing low-level details (not the public interface).
As you say in the post, it still feels brittle and imperfect to expose those values, even with the added help of the elm-review rule (if it gets renamed, you lose that safety that it’s not being used outside of tests).
Another approach
I thought of an approach that I believe gives stronger guarantees, and doesn’t require you to define the test case in the module. You can define the test data in the module, like a test fixture or test factory approach.
The trick is to use a Fuzzer.
Why a Fuzzer, isn’t that just for random values? Well, I’m glad you asked! Fuzzers are great for the guarantee they provide. The only place you can get a value from a Fuzzer is by running it through elm-test.
So that gives us the guarantee that we aren’t accidentally exposing the Admin role outside of tests. And otherwise we get to define our test case as usual (it’s just some helpers for building test data that we need to create Fuzzers for).
Another benefit of this approach is that we can extend this approach to more general ways of building up test data, including part of the data being random (for example, generating random names for the Admins, or random IDs). We can define these additional parts of our test fixtures in the opaque module if it needs access to private data, and otherwise we can define it in our test folder.
An example of the Fuzzer test fixture approach
module RoleTests exposing (suite)
import Fuzz
import Http
import Json.Decode as Decode exposing (Decoder)
type Role
= User
| Admin
adminFuzzer : Fuzz.Fuzzer Role
adminFuzzer =
Fuzz.constant Admin
-- ... rest of the code from the blog post
Then in the test file, we get access to that private Role.
module TennisTest exposing (suite)
import Expect
import Role
import Test exposing (..)
suite : Test
suite =
describe "Role"
[ Test.fuzz Role.adminFuzzer "admin can delete database" <|
\adminRole ->
adminRole
|> Role.canDeleteDatabase
|> Expect.equal True
]
Again, note that we can only get the value from the Fuzzer by running it through a test case like that.
I’d be curious to hear what you think about this alternative!
4 Likes
Oh, that’s a very good usage too! Very interesting usage of fuzzers!
I guess this still adds elm-explorations/test as a dependency, but it would be removed in production, so no issue there. elm-test also doesn’t need to be configured in any way. So yeah, a pretty good solution!
The downside I can see, which is like really small, is that unless you have other fuzzers, you will likely want to do Test.fuzzWith { runs = 1 } in order not to run the same exact test 100 times 
Or maybe that’s already handled by default by elm-test? If not, I wonder if combining fuzz tests with a constant fuzzer increases the likelyhood of running the same test several times 
Another solution I just thought of is to have something wrapped for tests, and only tests can use that function.
module TestOnly exposing (wrap, unwrap)
type TestOnly a = TestOnly a
wrap = TestOnly
unwrap (TestOnly a) = a
-- + functions to combine them?
module Role exposing (Role, testOnlyAdmin, testOnlyUser)
testOnlyAdmin = TestOnly.wrap Admin
testOnlyUser name role = TestOnly.wrap (\name role -> User name role)
suite : Test
suite =
describe "Role"
[ Test.test "admin can delete database" <|
\() ->
Role.testOnlyAdmin
|> TestOnly.unwrap
|> Role.canDeleteDatabase
|> Expect.equal True
]
Then we prevent the function TestOnly.unwrap to be used anywhere in the production-facing code.
It’s very similar to the idea of wrapping as a Fuzzer, but it will not have the side-effects of causing hundreds of tests for no additional value. elm-test itself could provide such a wrapper in order not to have this overhead, or if it notices that the only fuzzer is a Fuzz.constant, then it can only run the thing once (which it may already do, that’d be great!), and I think that would be perfect. And as you said, if you can generate more sensible values, then a fuzzer is a win-win anyway.
Wrapping a value inside a type with TestOnly or with Fuzzer is actually just a different, and probably better way of tagging a value, rather than by name 
(sudden (re-?)realization for me). It makes it harder for tools like elm-review, but it will get to the point where it can detect things by type 
Either techniques (this new one or the original one) can be used for other/similar situations, like limiting the usage of something to only specific parts of the codebase, like a “storybook” for instance.
Re-NoUnused.Exports: I mentioned the problem of having test-only values be reported by the rule. I now see that it can be a problem even without this setup. I can’t come up with a good way to not report test-only values, unless by tagging the functions and/or adding exceptions to fuzzers, which doesn’t seem like a great solution. So I’ll have to hold out on that one . Good thing I’m aware of this now!
2 Likes
BTW you can get values out of fuzzers outside of tests:
getValue: Fuzzer a -> Maybe a
getValue fuzzer =
Test.Runner.fuzz fuzzer
|> Result.toMaybe
|> Maybe.map (\val ->
Generator.initialSeed 4 -- chosen by fair dice role
|> Generator.step val
|> Tuple.first
|> Tuple.first
)
Admittedly, this function is going to stick out a bit in code review…
1 Like
Ah yes, good point. I glanced in the docs to check for a low-level utility like that and missed that.
But yeah, as you say it will stick out, and generally I’m pretty comfortable with a few reasonable gaps like this. For example, if a module exposes a Decoder OpaqueType, it seems like enough of a guarantee, even though you could always call the exposed decoder with a manually built string to get that generated type. Either way I think using a Fuzzer provides stronger guarantees than checking for a naming convention with elm-review. In the strict sense of the word “guarantees” of course they’re both not actually guarantees at all. But we’re assuming developers with good intent, so it’s more a question of making the desired choice easy, and making it clear that the undesired code is not the right approach.
Interesting discussion!
I thought I’d share one other approach. If you were to use elm-spec then you could write a test that describes the behavior of this module without needing to expose the values of the opaque Role type.
I created a gist that shows what this could look like: https://gist.github.com/brian-watkins/bec6351780ee985134609411b233b852
The test scenario calls the requestRole function, sends the command to the Elm runtime, processes the command with a stubbed HTTP response (which ends up determining the Role value for the user), observes the Role value produced by processing the command, and expects that canDeleteDatabase returns True when it is called with that Role value.
The key is just that elm-spec allows you to simulate interactions with the world outside the Elm program (like HTTP requests). So, in this case, we can use that fact to help us generate a particular Role value during the test without needing to expose that type.
I like this approach because (1) the Role type remains opaque, even within the test, so the test uses functions from the module just as they would be used within the actual program, (2) the full behavior of the module can be exercised in the test (eg, sending and parsing the HTTP response), and (3) the test documents the connection between a particular server response and permission for a user to delete a database in the program.
Happy to discuss further!
1 Like
I’ve worked around this situation once or twice before - I used this strategy, based on your example:
module Role exposing (Role, adminToBool, userToBool)
type Role = Admin | User
adminToBool: (Role -> Bool) -> Bool
userToBool : (Role -> Bool) -> Bool
This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.