- the ability to make explicit assumptions about fuzzer inputs (and report that a certain number of cases have been skipped because of an assumption)
Filtering fuzzed values is a very bad idea. Any filter has the potential to filter out all the values, or even enough of them that your test runs very slowly. We’ve considered workarounds but it’s been pretty ugly, and we removed them. This is true of other fuzz libraries in other languages I’ve tried that offer a filter – sooner or later, you run in to trouble.
- the ability to label test cases so as to get statistics about the values
Sounds reasonable, at least in isolation. You could pass an
a -> String alongside the
Fuzzer a into the test.
- the ability to assert coverage statistics
This is really just codifying what distribution of labels are acceptable. I think the progression is actually add labels => assert distribution => filter fuzzed values to make the distribution acceptable. And again, that last part isn’t viable. So maybe we should rethink the seemingly innocuous labeling feature if it’s building towards something we can’t have?
Ian’s bounding box test is a pretty weak test.
intersection box1 box2 = Nothing would cause it to pass. While seeing the distribution of expectations would tell you something isn’t right, it wouldn’t tell you how to fix it. Most tests invoke every expectation every run (either there’s one expectation or there’s an
Expect.all). Putting expectations in branches almost inherently means that you don’t have your test value nailed down enough.
The elm-test README offers this piece of advice (courtesy of yours truly):
If you find yourself inspecting the fuzzed input and making different expectations based on it, split each code path into its own test with a fuzzer that makes only the right kind of values.
So to use Ian’s bounding box example, don’t fuzz two bounding boxes independently. Fuzz a pair of disjoint boxes and ensure there’s no intersection, and fuzz a pair of intersecting boxes and ensure the intersection is what you expect. That’s two separate fuzz tests, so you know that you’re testing 100 of each.
Of course, this does mean that you’re putting more work into the fuzzers, but I think that pays off. The solution to “how do I filter a fuzzer” is to only generate the values you want in the first place. (And yes, fuzzers can get as complicated as you like. There’s no
Fuzz.andThen, but there’s
Random.andThen, and if you’re constructing something that complicated you should write your own shrinker anyway.)
Oh by the way, there’s a lot of breaking changes on
master with no set release date (including renaming
Simplifier and making them much easier to write). So if we do want to add a feature, now is the time!