Here’s a small tool I wrote so you can test backwards compatability.
Test your currently checked out code against the tests for any previous versions. Use when publishing a new minor or patch version of your package to make sure you’re not making a breaking change.
This is an idea I’ve had in the back of my head for a long time. I don’t even publish packages, but I just wanted to make it. Hope this is helpful (or at least interesting) for package authors!
This is pretty interesting. I’m curious what use-cases you were trying to have this find though.
The one that comes to mind is to figure out in which version a bug was introduced, but that’s kind of the backwards approach (test previous versions’ code with the current tests, after introducing a new test)
Use when publishing a new minor or patch version of your package to make sure you’re not making a breaking change.
Usually, tests are the standard to determine whether something is correct. Testing with old versions’ tests is only useful if tests were over time either removed or modified. But tests are usually modified either to make the test more complete, or to reflect a behavior change (often for bug fixes).
Testing with old versions’ tests is only useful if tests were over time either removed or modified. But tests are usually modified either to make the test more complete, or to reflect a behavior change (often for bug fixes).
Pretty much this kind of thing. Usually tests are being updated so that the next version’s tests are a superset of the previous version’s, but there’s nothing to guarantee that. The two main examples that I can think of are when tests are refactored or public code behavior gets changed.
Tests have been refactorted: let’s say v1.0.0 is stable, v1.1.0 refactors the tests that unintentionally leave a code path untested. v1.2.0 introduces a bug in this code path and the author isn’t aware they’ve broken backwards compatibility because the tests all pass. But if you run this version’s code against the tests from v1.0.0, you’ll catch it.
Public code gets changed: I wouldn’t worry too much about bug fixes, since a previous version probably doesn’t have a test that asserts on this buggy output. But there’s nothing stopping someone from changing their tests at the same they change some behavior. For example, some function f : Int -> Int that previously operated on all inputs now handles negative numbers differently.
If this testing strategy were enforced for all packages, everyone would just have that much more peace of mind when it comes to bumping minor and patch versions.
All that being said, this is probably more of an argument for everyone to have good test coverage for their own code - if a minor/patch version bump changes in a way that affects your code, you ideally will catch it pretty quickly.
The one that comes to mind is to figure out in which version a bug was introduced, but that’s kind of the backwards approach (test previous versions’ code with the current tests, after introducing a new test)
Yeah, exactly, this tool won’t help you at all here. (a clever git rebase and git bisect --run <test command> can already automate this though)
Strikes me that it could also be useful for package maintainers.
When publishing a new minor or patch version, could compare test results with the previous version to compare. It might help to check you haven’t made a mistake, and could also help when writing the documentation, if it flags up some things that you really ought to mention in the release notes.
