So, I watched the talk (like I should have before replying!) and I’m a lot more sold. (Hughes knows how to present – I felt like I was thinking what he was going to say right before he said it.)
I think there’s a dependency chain here (not a tree, thankfully, just a chain):
- The ability to label values generated by the fuzzer
1b. The ability to see the distribution of labels on a test - The ability to see shrunken/simplified examples of each label, to qualify your label function
- The ability to encode advisory coverage requirements for each label
- The ability to enforce mandatory coverage requirements with “military grade statistics”
1 isn’t useful without 1b, hence the numbering. The part of the talk describing 2 was super fun and quite convincing. I’m thinking that invoking the label explanation functionality would skip all tests and print the explanation, and then have a “yellow” test run similar to Test.only so you don’t commit it.
Even if we don’t get to 3 and 4 immediately (4 being the severe implementation challenge), it’s worth thinking about the API. (A quick glance finds some union types that might need extra cases.) The obvious way to write a labeler an a -> String function with an if or case ... of expression, but that can’t tell you how many different cases to expect. Using the style of adding one case at a time is essential to 3 and 4, but also helpful for 2. One downside of this style is that the labelers might not be disjoint or complete; presumably the most recent one wins, and an unclassified value shows up in its own category, and fails 4?
Fuzz.intRange tests the boundaries too. (Perhaps 10% for each boundary is too low, if Hughes recommends 1/3?)
Yes, it’s pretty clear that writing fuzzers for each case of overlapping rectangles is going to balloon exactly as he’s describing. Opinion changed!
If you’re transforming between two different types, that won’t compile, but I digress.
It is, I’ve used it!