I am positive that I have this working, because I run tests like this and they do in fact break when things go wrong.
So, I do still have to have a tests/MainTests.elm file. But that file doesnt do anything; it looks just like what I posted before. MainTests.elm merely has import Main at the top. I just read through the tests we have at work, and I notice modules expose their tests tests : Test, so I think maybe that is key.
But, they arent explicitly organized into a heirarchy like with describe "Tests" [ Main.tests ]. I think elm-test really does climb the module heirarchy and run any exposed Test values.