Back in October 2021 I joined Realia, a startup aiming to help Swedes find good realtors when selling their home. Existing competitors don’t do much other than provide a form for you to fill out and then they come back with some realtors they picked. We wanted the process to be more interactive and provide more control to the customer to choose the realtors they like.
While the domain is interesting, I joined because the founder there let me start from a clean slate. This was an opportunity to use Lamdera professionally!
This is what it looks like https://realia.se/
Now 1.5 years later, I no longer work at Realia. I think some of what came out of this project might be useful for the rest of the Elm community, so the rest of this post will be about things I learned and built while at Realia.
Table of contents
- Lamdera
- Creating a map widget in pure Elm
- Visual regression testing
- Implementing a session viewer
- Okay so why don’t I work there anymore? (this is last to build suspense)
Lamdera
Type safe server/client communication, deploys that take a minute while also being type-safe across multiple versions of the app- look most of you know about Lamdera already and don’t need me to repeat all of its selling points. If you haven’t heard of it, the docs here will explain it better than I can anyway.
Lets talk about the challenges instead!
- We have a few GB of realtor data. Lamdera keeps everything in RAM so it wouldn’t work well to store it there. Instead it needed to stay in a postgres database and http requests would be made to it the old fashioned way. That said, even if it could fit into Lamdera, I think it would have stayed as a separate database because-
- My boss didn’t want us to fully rely on Lamdera. He envisioned Lamdera as a tool for quickly getting us to an MVP but long term we’d switch over to a more traditional setup with the Elm backend code being hosted on Google Cloud. For him, Lamdera isn’t mature enough to be trusted for with storing data. I spoke with Mario about this and he’s put together a draft describing what data guarantees Lamdera provides. Hopefully this can address these concerns in the future.
- Tooling support for Lamdera is mixed right now
- The Elm plugin for Intellij can fail to load lamdera projects and the package cache can end up in a weird state (
lamdera resetusually fixes these issues) - elm-json doesn’t know about Lamdera packages
- elm-dependencies-analyzer had the same issue but I got a PR merged that adds support
- Elm-review fully supports Lamdera!
- The Elm plugin for Intellij can fail to load lamdera projects and the package cache can end up in a weird state (
- Kernel types (i.e.
Fileinelm/fileandTextureinelm-explorations/webgl) can cause some problems in more advanced use cases (more on that later)
Despite these challenges, most things went smoothly and Lamdera is still my top choice for any new project.
Creating a map widget in pure Elm
An important part of the app is displaying an interactive map with markers to show where realtors have previously sold properties.
Originally we tried implementing this in Google maps. While I think Google maps is fine from an end user perspective, it was just awful to try integrating into our app. So we decided to switch to Mapbox instead. It’s easier to work with and there’s also gampleman/elm-mapbox. While definitely an improvement, it also had problems:
- The Mapbox JS added 891kB (uncompressed) to our bundle size
- Mapbox has a JSON based configuration language that is very difficult to work with. @gampleman partially addressed with another tool that converts this JSON config into Elm code but with the drawback that it’s 2.8k lines of Elm code (enough to have an impact on bundle size). Also Mapbox had made breaking changes to their format so the generated code required manual fixing whenever our UI designer wanted to change the map styling.
- The map widget has lots of internal state that gets lost if I’m not careful about how the view function changes the VDOM
Despite the criticism I want to thank @gampleman. The problems stem from how Mapbox is designed, not his package. Things would have been worse without it.
After a few months using Mapbox then, I decided to make my own map widget in order to solve these problems for good. What you see in the app is the result of that work. It still uses the Mapbox server for requesting map data but otherwise it’s purely Elm code with rendering done via elm-explorations/webgl.
My boss allowed me to open source it so here’s the repo. Unfortunately it’s currently not quite up to date with the latest app version and a lot of documentation needs to be written before it’s ready to release as a package.
Overall the pure-Elm version solves almost all of the problems I had with the Mapbox map widget. The only major drawback is that it lags when zooming in rapidly or loading realtor markers. I’m hoping that once Lamdera integrates elm-optimize-level-2 this will be less of a problem.
Visual regression testing
Before I started at Realia I had been working on lamdera/program-test in my free time. It’s based on avh4/program-test but does away with the programmer needing to define a Effect type. Instead you just swap out all of your cmd modules with Effect.* module equivalents (a more in depth explanation here). Once this is done it’s possible to write end-to-end tests that simulate the frontend and backend within elm-test. I have a presentation demoing this.
In addition to this, I had been working on adding Visual Regression Testing (VRT) support. VRT is where you generate screenshots of your app and then compare them to earlier versions to see what has changed. This can be very effective at catching unwanted visual changes.
Here’s some example code:
myTest =
Effect.Test.start (config httpRequestHandler) "Happy path"
|> Effect.Test.connectFrontend
sessionId0
domain
{ width = 320, height = 568 }
(\( instructions, client ) ->
instructions
|> shortPause
|> client.inputText HomePage.addressInputId "Elm Stree"
|> shortPause
|> client.snapshotView { name = "Unfinished address" }
|> client.clickButton (Autocomplete.autocompleteRowId 0)
|> shortPause
|> client.snapshotView { name = "Autocompleted address" }
...
)
shortPause =
Effect.Test.simulateTime (Duration.milliseconds 100)
Each client.snapshotView stores the HTML for the frontend at the point in time. When running elm-test this doesn’t do anything. But with a VRT runner I made, it collects all those snapshots, converts the HTML to strings, and uploads it to https://percy.io/ which handles generating screenshots and diffing them.
After that’s done I can just view all the screenshots in Percy and verify that the diffs (red highlights) are all intentional.
So how did end-to-end testing and VRT work out at Realia?
Positives:
- On several occasions it caught subtle visual mistakes that I would have never noticed otherwise
- The end-to-end tests required very little upkeep. Since they are on the level of simulating button presses, it didn’t matter if I made large changes to the underlying Elm implementation
- This spared me from having to use Cypress for simulating user input and uploading snapshots
Negatives:
- Percy has issues with rendering consistently. Sometimes the same HTML will generate two slightly different images which means I have to spend more time checking diffs. Some of this isn’t Percy’s fault such as Safari’s rendering engine not being deterministic. But sometimes even Chrome and Firefox would change the layout on me and all I could guess was that Percy swapped which browser version it was using between the old and new screenshots.
- VRT is slow. It would take 5 minutes for my VRT runner to finish uploading all the DOM snapshots and then it would take another 5 minutes for Percy to generate the diffs.
- After a while, elm-test started getting really slow and would use so much RAM that my computer would lock up. I was running each end-to-end test 3 times for different window sizes so I lowered it to 2 and things sped up. I’m not sure what the root problem was though.
- Since the DOM is converted to a string and uploaded to Percy, DOM nodes with internal state like a custom element or canvas would appear blank. For Realia this meant that the map and map markers wouldn’t appear.
Overall I’m happy with end-to-end testing and VRT. I’ve already started using VRT in Meetdown as well. It’s not well documented but if anyone wants to try it out, here’s the VRT runner. I’m happy to answer any setup questions!
Implementing a session viewer
I added an admin page to the Realia app for viewing logs, handling customer submissions, and viewing customer sessions.
For customer sessions, when a customer loads the app, the msgs generated on the frontend are sent to the backend and stored as a session. Then in the admin panel I can replay those sessions to see what the customer did. This helped give us a better picture of what customers were doing in a way that isn’t as obvious when only storing more coarse grained events like “customer reached page x” or “customer clicked button y”. Also since Lamdera automatically encodes and decodes types, no dev time was spent serializing FrontendMsg.
Unfortunately there’s a lot of drawbacks to this:
- Conceptually replaying a session is simple, just save the initial model and msgs and then fold over them using the update function. But there’s a variety of ways it can get a lot more complicated
- Are any of your msgs large? In our case, realtor requests could be several Mb and there could potentially be dozens of them for a single session. The work-around is to have a dummy msg that just stores enough info to redo the request and then the session viewer needs to make those requests and turn the dummy msg back into the real msg before replaying the session.
- Does any of your msgs contain
elm-explorations/webglTextureorelm/fileFile? ForTexturethe solution is the same as above with dummy msgs. ForFile… I don’t think there is a good solution. You either avoid File and use ports when dealing with files, or you make the person viewing the session open a file from their computer in order to create aFileinstance (there is no other way to create it within Elm). Fortunately for Realia, onlyTexturewas needed.
- Anything not represented in a msg won’t appear in the session replay. For example, scrolling probably won’t be replayed. Also, anything outside of Elm. So custom elements won’t be initialized. In Realia’s case this was an issue when we were still using the Mapbox map widget, but it’s no longer a problem with the pure-Elm version.
- Implementing all the UI for the session viewer can be quite time consuming
- Maybe there are GDPR concerns? Anything the user typed will get saved in the session. My boss decided that in our case this wouldn’t be an issue (and I got that in writing in case he’s wrong
)
- Another solution is to replace sensitive data with dummy data. It has to be done carefully though to avoid changing what happens in the session replay.
Okay so why don’t I work there anymore?
I mentioned earlier my boss wanted to move away from Lamdera in the long term. Maybe you guessed that’s why I no longer work at Realia? Well we never reached “long term” unfortunately! From a technical perspective things were going smoothly. But from a business perspective, we failed to attract customers. We had a very high bounce rate. Most people never reached the map screen I had put so much time into! So after 1.5 years my boss decided to let the app continue running but stop development on it.
While I would have preferred that Realia was successful, hopefully some of the things that came out of it can be useful to others in the community. In the meantime, I’m looking for a new Elm job, preferably with Lamdera!
