I have an upcoming project to investigate improving our automated testing accross various browser/OS combinations. We already are building up a set of regression tests using elm-test on our domain model, and application logic (the update functions).
What resources and techniques are available for testing Elm code running in a real browser, or simulated browser? Is it possible to write these tests in Elm, or is it better to use a different language that has access to more testing tools? For example, I could write Selenium/WebDriver tests in JS (or Java, C#, Python, etc…) - an Elm web app is just a web app after all, the tests don’t really have to care that it is written in Elm.
I also found these packages which may be relevant, I am just not sure how mature they are, or whether they will enable me to test on cloud hosted device testing services such as BrowserStack. Generally, it seems like Selenium/WebDriver is they way to go for working with these type of services?
elm-webdriver 1.0.2 - Seems like it lets you create WebDriver from Elm. Is it 100% complete? Does not offer a way to introspect the Elm program state?
elm-spec 3.3.2 - Looks quite comprehensive and I think it lets you test the DOM as well as introspect the Elm program state? But no WebDriver, so unlikely to work with online device testing services, or is there a way?
Ideally I would write tests that can check the state of the DOM, state of the Model, are fully automated, support Fuzz tests, and can make use of device testing services to cover many browser/OS combinations.
I would be interested to hear your thoughts, experiences or recommendations of tools to consider.
We tried Cypress for some time and while it has a lot of nice features in the end we decided that actual end-to-end testing is was not worth the afford for us but I’d still recommend the product.
Seems like brian-watkins/elm-spec is aiming to be a Cypress for Elm. Runs in browser, rather than as a separate process controlling the browser through WebDriver, and lets you stub Http and ports etc.
@rupert if you are interested in contributing elm-test-rs, we have already a deno runtime and I believe providing a simple web runtime (without watch mode) would be fairly doable. I’ve listed the technical details that were needed to port the runners from node to deno on an issue and much of it is relevant for a web browser.
EDIT, maybe I misread. I don’t think this would help you with actual dom testing.
Yeah, as I understand it elm-test-rs is a drop in replacement for running elm-test tests? It may certainly be a better way to run our existing unit tests. I also think we should be trying to do as much of our testing via unit tests as we can - it is going to be less brittle than end to end testing.
In this case though, I need to look beyond what we can do with unit testing and do some actual against-a-real-browser-testing. This is because we use things like contenteditable, Selection API, touch events, the intersection of SVG and CSS - all areas known to be a bit poorly specified or buggy between the various implementations. Ideally, I would like to be able to write a test that confirms the same user action works on a desktop running Chrome as it does on an iPad.
yep, elm-test-rs is there to be able to run all your Test expressions. It won’t solve any need that goes beyond what can be expressed with elm-explorations/test. I did not interpret your needs correctly at first, and thought you wanted to run your test code in browsers (sorry read too quick).
Cypress currently supports Firefox and Chrome-family browsers (including Edge and Electron) .
I need to test on Safari on Mac or IOs, which could be a problem…?
That could be an advantage of elm-spec, as it seems it should be able to run anywhere our Elm code does? Not 100% sure on this though, I will need to try it out in more depth to figure out exactly what it can do.
I have hovered around these topics a few times and researched them. I haven’t executed on the research yet.
Here are some of my thoughts:
The language that the test scripts are written in doesn’t matter too much to me. The tests are usually short, relying on libraries / framework to do most of the work. Given that preference, I look across ecosystems and tried to find the most feature rich, well liked testing tools. If I really want the scripts to use a language that I like, I would probably create ReScript bindings for a js testing library.
Overall, java or python with selenium/webdriver seems like the most used historically, but cypress seems to be growing the fastest in usage and playwright sounds interesting. There are some articles that compare the dozen or so options in node/js.
Since selenium has been around longer, I think one is more like to find companies that provide environments for running tests across browsers for you remotely, such as sauce labs Both Sauce labs and browserstack support a variety of frameworks. I don’t think general purpose CI tools like circle and gha will produce as fast and feature rich reporting as special purpose browser testing CI tools, although I imagine the browser testing tools have integrations with both circleci and gha. I would just want an alert message posted to a PR, so circleci and gha integration would not be important to me.
I am also curious about combining e2e tests with tests detecting regressions in design/layout such as Percy and Backstop.
Some open questions:
What is the daily workflow like? Commit to draft PR, observe results from remote execution, and run any failing tests locally? Any way to speed up this workflow?
Do you mock out the back end when running the tests?
These things are more where we aspire to get to than where we are currently at - testing so far is all pure unit tests with elm-test, or manual testing through the complete UI.
Across all of our teams we use cypress (with few experiments on playwright) to test e2e. We write tests in javascript with imported ts types via jsdoc comments to streamline development and build workflow.
Cypress development mode is a godsend for troubleshooting bug reports.
Of course for each project we have one constant instance of database (sometimes with thin backend) so we can write assertions for specific values presented on the screen.
Additionally, we use visual regression. It is historically written with puppeteer and pixelmatch.
Our applications use urls to store global state and we treat every other state as ephemeral so it is easy to setup e2e tests and regression snapshots by just specifying urls.
As for workflow, our tools run on CI. If you get an error in visual regression, report is generated with two different images and description of case (description always explain how to achieve this specific view). If we get error in e2e, developer runs locally cypress with focused failed test case and can explore what is wrong inside cypress developer mode browser.
I haven’t implemented them myself, but I have browsed a large react codebase that had unit tests, unit tests w/dom (enzyme/react-testing-library), and browser tests. I haven’t looked into the elm equivalent of react-testing-library. It sounds like you want all three levels of test suites. The browser tests don’t need to use anything elm specific, as you and others have noted. The react-testing-library style tests can run in any generic CI system.
One other either downside or upside, is the node based frameworks go through the usual rate of npm churn, so you get bleeding edge features, but even as we discuss, I am sure there is someone trying to topple playwright as the next up and coming browser framework. On the other hand, tests using python or java frameworks using selenium can probably be maintained with little attention to upgrading or switching the framework.
Thanks for all the ideas and thoughts particularly on cypress vs selenium.
I am going to try out elm-spec, and see how it goes as I am curious about it.
But I also think this is basically right, and I should mostly put my efforts into unit testing with elm-test. Have a rule like - test everything through unit tests, and only move up to integration tests if a unit test will not do, and only move up to end-to-end tests if an integration test will not do.
elm-spec tests can be run in a real browser in a few different ways.
elm-spec-runner is a program that runs elm-spec tests from the command line. By default, it runs tests in JSDOM, but you can specify that it run tests in a real web browser – it relies on Playwright behind the scenes to accomplish this. Just specify chromium, firefox, or webkit for the --browser flag to run tests in the relevant browser. The version of the browser depends on the version of Playwright installed.
elm-spec-runner also allows you to run elm-spec tests in any browser via the --browser remote flag. This provides a URL; just load that URL in a web browser and your tests will be executed in that browser, with the output printed to the command line.
karma-elm-spec-framework allows you to run elm-spec tests in a real browser via karma. This also allows you to take advantage of other karma tools while running elm-spec tests.
I’m a big fan of testing web applications in real browsers – since that’s where they will run – and so when I use elm-spec I typically use option 1 to run my tests in chrome.
As far as using online browser testing tools with elm-spec, I think there are a few options to explore.
Tools like Lambdatest (and Browserstack I think) can set up a tunnel between a remote browser and the local machine. Once this is established, you should be able to run elm-spec tests in the remote browser by leveraging option (2) above – ie, use the --browser remote flag to get a URL that you can load in the remote browser (via the tunnel) to run the tests.
I believe Browserstack has a karma plugin and so it may be possible to use that with karma-elm-spec-framework to run tests in the browserstack environment. I haven’t tried this myself because my free Browserstack account expired some time ago.
I’m happy to talk about this further; if elm-spec doesn’t satisfy this use case today, I’m happy to consider adding support. If you like, you could raise an issue on the elm-spec github page so we could continue the discussion there.
Super helpful - thanks brian. Option 2 elm-spec-runner --browser remote sounds like it would enable me to do some scripted browser/device testing for Safari/Andriod, even if it does not get all the way to full automation. I can explore other options later.
I was able to run elm-spec on Browserstack with karma:
elm-spec-spike@1.0.0 test /home/rupert/sc/gitlab.com/gameboard-tester/elm-spec-spike
karma start
Success!
MainSpec ───> /tmp/2022527-25498-12e1nmk.f5rr.js
27 06 2022 21:40:09.151:WARN [filelist]: Pattern "/home/rupert/sc/gitlab.com/gameboard-tester/elm-spec-spike/src/elm/*.elm" does not match any file.
27 06 2022 21:40:09.160:INFO [karma-server]: Karma v6.4.0 server started at http://localhost:9876/
27 06 2022 21:40:09.160:INFO [launcher]: Launching browsers bs_chrome_windows with concurrency 1
27 06 2022 21:40:09.183:INFO [launcher]: Starting browser chrome 102.0 (Windows 11) on BrowserStack
27 06 2022 21:40:12.452:INFO [launcher.browserstack]: chrome 102.0 (Windows 11) session at https://automate.browserstack.com/builds/9eb4f406ad43056715b28288aae8993016d53021/sessions/d9ef9aee64b0b0a7f411c1615a596444bb56eb69
27 06 2022 21:40:22.261:INFO [Chrome 102.0.5005.63 (Windows 10)]: Connected on socket AYqGrkHQQVC6gd-KAAAB with id 17875640
Running specs: .
Accepted: 1
So far I have managed to test succesfully against Chrome, Firefox, Edge and Safari on Windows11 and MacOs Catalina - pretty nice!
Running locally via elm-spec-runner or Karma with one of its local runners for Firefox I get a syntax error. I will try to figure out what is going on with that.
Slightly off-topic, but if one of your problems is dealing with browser/environment inconsistencies, then maybe introducing this tool into your workflow could help?
You can proxy your existing dev server and replicate scrolling, clicking links, etc. all from one device.
It won’t help detecting bugs after the fact of course, and it’s not really testing. But maybe it could help dealing rapidly with obvious issues early on, while working on a feature (if developing with multiple screens is doable).
It’s my understanding that Playwright is more “capable” than Cypress and probably all around “better”. With Playwright, you can for instance test running multiple tabs, which is impossible to do in Cypress. Also, I found the Cypress API to be rather quirky compared to Playwright’s but that’s more of a nitpick
Thanks, Browsersync is a good suggestion. It also occurs to me that I do not necessarily have to have the physical devices to use it, I can sync up a cloud hosted device with it too - presumably its just a question of giving the device the correct URL. I shall try it out.
Again, looking at the testing pyramid and thinking about the value/effort involved in testing, and the fact that we are a small development team, we will certainly continue doing manual testing when the effort to automate is too high. So browsersync seems like it could be a useful force multiplier there.
