Two experiences with Elm

[rtfeldman]

The Elm leadership need to understand that if they go around promoting Elm, talking about it as if it is a production ready language and getting people excited, and then intentionally make the compiler refuse to compile users’ code simply because they disagree with what features people are choosing to use, and simply dismiss these users as “not many people”, they cannot expect to retain good will from the community, and don’t deserve to either.

(Emphasis mine.)

Evan was open about the fact that the 0.19 design is the design he intended from the beginning, back in 2011. Is it fair to call unintended, undesirable behavior (aka a bug) a “feature people are choosing to use?” To me, fixing a bug before even more people come to rely on its unintended behavior seems like the most responsible thing to do.

Is Elm a production ready language? This is objectively true. There are too many examples of Elm in production success stories for it to be false.

As for the notion that Native is necessary, as opposed to expedient, so far that hasn’t held up under scrutiny. You can see threads on this forum (and others) where people described their use cases, and others came up with solutions that didn’t use Native. I’ve seen this happen at work, where we’ve used Native in the past out of expedience, and have since found that we can do everything we were previously doing with Native using a mix of ports, custom elements, and polyfills. Turned out we didn’t need it either.

As Evan noted on the other post:

"Native Code" in 0.19

For folks who are really anxious about this, I really encourage you to talk to folks about it in a friendly way. I have not yet heard a case with no answer, and we have lots of time to work through things in a good way.

I don’t know all the details of your scenario, but if you want to run a performance-critical JS algorithm in response to user actions, one way you could do that is by writing some JS to attach a custom event handler to a DOM node managed by Elm, which runs the calculation and then fires a custom event containing the result of the calculation. Elm can listen for that event instead of the original event, and decode the result as normal. Should work, yeah?

(That said, solutions are probably better discussed on a different thread, assuming you would like to discuss them. I think if you make a separate post outlining your particular scenario, you might be pleasantly surprised what solutions people come up with!)

[pdamoc]

However, this blog post also contains info about the specific problems I have with the changes to native modules in 0.19 that would block me from upgrading.

I can empathize with some of the things you mentioned in your post. Some of them do point out to things that I too have experienced as frustrating.

The tone of the article is also familiar as I think I wrote comments on these topics in the same tone.

Lately I have become aware that I was feelings a lot of resentment and entitlement and both of these things are toxic.

Both of these feeling sometimes masqueraded as “constructive criticism” in certain things I’ve said.

Now I try to be mindful of them and just delete the comments I write where I detect them. I have probably wasted hours writing, rewriting and finally deciding not to publish comments of this kind. I know that I will not be able to avoid them every time but I try to do my best.

When I started focusing on the solutions, I started feeling better. I started seeing what others have tried and found successful in dealing with the issues.

In closing I would like to point out two things:

1. As someone (I think Richard or Luke) pointed out on reddit recently, everything Evan does is made public, published as Open Source under a very permissive license.
2. The whole idea of Open Source is that you have the freedom to take the source code and modify it the way you see fit in order to make it do what you want. In this respect I recommend this article by André Staltz. Criticize with code. This way you will have skin in the game. You will both receive the good karma of an author of a solution and you will also have to deal with the brokenness of your solution.

[madasebrof]

Had a super long reply, deleted it. Didn’t really add much vs. what has already been said.

The one part I didn’t really get in your post is the part about state bugs in Elm.

“Finally, there are certain types of bugs that I found happened more with Elm than with other languages, including a certain category of state bugs.”

The reason for my reply is not to be nit-picking on one sentence in a huge post, but rather because I think proper state mangement is the single greatest reason to use Elm in the first place. Also, I think state mangement is the hardest thing to do correctly in a front-end framework, which is probably why most frameworks kick the can down the road (e.g. React -> Redux/MobX, Vue -> Vuex, etc.), and the reason for things likes Cerebral.js. Of course, most of the these are explicitly based on Elm and TEA.

https://cerebraljs.com/

“I often tripped up over the pattern of calculating some new state (e.g. in an update function) and then finding that within the same function I was still using the old state for another calculation, resulting in some head scratching and debugging sessions (usually well before deployment, but it is still a real cost, sometimes an hour sunk). I will probably get better at this with time, but it is something that a type system doesn’t save you from, and requires a discipline about how you write and modify functions.”

I have never had a bug related to state in Elm, whereas that is the first thing to break in other frameworks/languages and really does lead to endless debugging.

You can see this problem fairly regularly when you use things like Twilio, Squarespace, Upwork, etc. Periodically you have to reload the entire page as the state gets out of whack. (I have no idea what tools/frameworks etc those companies use.)

Here is the pattern we use for update functions. Are you doing something similar?

type alias Model =
    { unitPrice : Float
    , totalCost : Float
    , unitCost : Float 
    , etc...
    }

type Msg
    = UpdatePriceChangeFactor Float
    | MoreUpdateMessagesGoHere MessagePayloadType
    | Etc.

update : Msg -> Model -> ( Model, Cmd Msg )
update msg model =
    case msg of
        UpdatePriceChangeFactor changeFactor ->
            -- Make all of your state change functions in the 'let' section.
            --   a 'model' is passed into the 'update' function
            --   and refers to the old state of the model, prior to any updates.
            --
            -- 'let' is where all of the heavy lifting happens!
            --
            let
                newUnitPrice =
                    -- whatever function you like goes here...
                    model.unitPrice * changeFactor

                newTotalCost =
                    -- and here...
                    newUnitPrice * model.unitCost
            in
            -- Then update the records of the model in the 'in' section
            -- with all of the state change functions from the 'let' section
            -- and return a new ( Model, Cmd Msg ) with the updated model records.
            ( { model 
                | unitPrice = newUnitPrice
                , totalCost = newTotalCost 
               }
            , Cmd.none 
            )

        etc...

The same logic applies to updating sub-records, dicts, lists, etc. Get the thing you want to update in the let section, modify the results via a new function combined with the message payload (e.g. assign a new variable, so to speak), then apply the changes in the in section, returning a new model and whatever command you need to run.

Hope this helps if you continue on your Elm journey!

PS I have no dog in this hunt. I just have found Elm really productive, and personally think the tradeoffs between verbosity and robustness are a worthwhile exchange.

Rock on,

Adam

[spookylukey]

Hi Richard, thanks so much your reply.

Evan was open about the fact that the 0.19 design is the design he intended from the beginning, back in 2011. Is it fair to call unintended, undesirable behavior (aka a bug) a “feature people are choosing to use?” To me, fixing a bug before even more people come to rely on its unintended behavior seems like the most responsible thing to do.

I find it very hard to characterize this as fixing a bug, when it makes the compiler simply less functional than it was before. Even if the original behaviour was unintended, everyone was well aware that people were making use of it, and were doing more in Elm than they would have been able to otherwise. It might not have fitted Evan’s vision for how Elm projects should work, but sometimes our accidents are better than our plans.

I’m a Django core developer, and I’m well aware of things like users making use of internal APIs. For example, Django’s ORM had an internal API called _meta (the underscore by itself indicates it is internal, plus it was completely undocumented, and we make it clear that we document all intended features). However, we became aware that people were using it, and finding it useful. So, we 1) cleaned it up, 2) documented it, 3) included a full migration guide for the changes we had made (even though it was only ever internal before) and 4) used the normal 2-version deprecation cycle for removing the old version.

True, normally we wouldn’t go to such much effort for an internal that no-one was supposed to be using anyway, and I wouldn’t expect that of others. But the last thing to ever cross our minds was “deliberately change this to make it harder for people to use”. In my mind, that kind of attitude is included in what I mean by production ready - it’s not just the current state of the software, it’s an implicit commitment for the future that at the very least you won’t go out of your way to make it harder for your users’ code to continue working.

I don’t know all the details of your scenario, but if you want to run a performance-critical JS algorithm in response to user actions, one way you could do that is by writing some JS to attach a custom event handler to a DOM node managed by Elm, which runs the calculation and then fires a custom event containing the result of the calculation. Elm can listen for that event instead of the original event, and decode the result as normal. Should work, yeah?

I’m sure your solution would work, and I’m sure there would be other solutions - I could even use ports, possibly, with other hacks necessary if I went that route. I have already had to use several elaborate mechanisms like that to work around some issues I hit (as I mentioned in my blog post), because I found no other way. The question is, would my code be better and more robust for that kind of change?

To put it another way, if I could magically wave a wand and get a pure Elm function that did what my current Javascript function does, with all the performance as well as exhaustive testing etc, of the original, would anyone then suggest using any of these solutions or ways of constructing the code? These things would obviously make the code more convoluted and fragile. They would be poor engineering solutions.

But I already have that magic wand (i.e. since I trust this Javascript code not to throw exceptions, I can simply write a native module that gives me a pure Elm function that is in every respect equal to other ‘pure’ builtins such as ceiling etc.). So those solutions remain inferior engineering solutions. Why would I choose them? And why would Elm attempt to shut the door to a superior engineering solution? That’s the bit I don’t understand.

I understand about the plans to move to WebAssembly someday etc., but for the reasons I posted in my blog I imagine that I, and other people using performance sensitive code, would have less problems with that than, and the breaking change would be for a positive, necessary reason.

Thanks for reading again - you must be a very busy person with all that you put into the community. I appreciate it, and understand if you don’t have time for a reply.

[robin.heggelund]

Blockquote
Why would I choose them? And why would Elm attempt to shut the door to a superior engineering solution?

There are a couple of considerations to make here.

1. Native/Kernel code is not stable API. It has changed entirely for 0.19 from what I can see, and it can (and probably will) in the future. Internal representation of things like records, ADTs, Lists, and so one might change based on if the debug flag is set, leading to subtle breakage. From what I’ve read of the code for 0.19, the Elm compiler takes liberties based on the assumption that no one uses native code, and it wouldn’t surprise me if more of that will happen in the future. Ports, on the other hand, is stable and will continue to work just fine.

2. The more people that rely on some solution, the more difficult it is to change. If people are doing something that isn’t intended possible, taking it away before “everyone” makes use of it makes the most sense. The case you mention about an internal api seems like there wasn’t any hurdles to really make it publicly available (like you wanting to remove it, or change it to make it more performant). Elm, however, is not done, and would like the freedom to be able to change how code is generated from release to release. This is harder if such changes break peoples production code. And so, there should be incentive not to use such brittle constructs.

3. Why would anyone bother to create pure Elm packages if one can just rely on the native code crutch? It’s better overall to have incentive to make good Elm code that is portable and more stable. This doesn’t necessarily apply to your case, however.

That’s the way I see it at least, hope it makes sense.

[rtfeldman]

The question is, would my code be better and more robust for that kind of change?

Absolutely, but I can see why it wouldn’t appear that way at first!

To build on what Robin said, the key point from Evan’s post is this:

"Native Code" in 0.19

The fact that generated code is not pinned to some specific format means that I can change it dramatically without the community paying a big upgrade cost. In fact, I have done that a few times already! This has allowed me to (1) make Elm code a bunch faster overall and (2) shrink the size compiled output quite significantly.

The “custom event” approach relies only on public browser APIs, whose extreme emphasis on backwards compatibility has sometimes had hilarious consequences. However unfamiliar that appraoch might feel the first time you try it, if it works today you can expect it to work tomorrow.

In contrast, the Native approach relies on a secret API that is designed to break in brutally unforgiving ways without notice at any time—including changing formats (0.17 → 0.18), output languages (Web Assembly), or even disappearing altogether (0.18 → 0.19).

Native has never been a robust way to do JS interop from Elm. Closing the door on it for good creates a pit of success where it becomes clearer that the more robust solution is to rely on stable browser APIs instead!

[KasMA1990]

I’ve been struggling to reconcile two parts of the Elm “experience” lately, that I think I only really understand now after this discussion. On one hand, Elm is all about being predictable (compiler led development, no run-time errors, if it compiles it works, good performance in general, etc.), but when a user needs to write code that Elm cannot reasonably express (as is the case with the performance issue here), we get some responses like this:

I think you would be happier and better served by developing with PureScript.

This should not be interpreted as, “if you don’t like Elm you can just leave,” but it’s very hard to convey earnestness through text […]

I do believe you are completely earnest @luke, but I think the problem with this attitude is that all of a sudden you are saying that the given developer should have predicted that this would be an issue before they chose Elm, and chosen something else instead. In other words, Elm’s claim of predictability begins to break down in the eyes of the developer. And instead of providing an escape hatch, with 0.19 it will be even more important that the developer get this prediction right (and humans are of course notoriously bad at predicting the future). So now instead of removing pressure from the shoulders of developers, Elm becomes more about redistributing the pressure, and making developers hope they won’t ever run into a corner where Elm doesn’t cut it.

Now, I’m not trying to argue that native modules should stay; I don’t personally have any skin in that game, and I really just want to see where Elm goes. I also appreciate that the community wants to help and in many cases a good Elm solution can probably be found instead of going with native code. And though I can see how Richard’s solution would probably work, I also agree with @spookylukey that this seems like a hack instead of just calling a function.

Maybe the lesson here is that it should be easier for developers to predict if they want to do something that Elm doesn’t support very well?

[robin.heggelund]

The thing is that Elm really only has one way to do interop, and that is ports. “Native” code has never been documented, or advertised, by Evan. It’s never been a part of the language, just a badly kept secret.

Elm will change in the future, and it can do so in a way that breaks all code written using an unsafe and undocumented API. That isn’t very predictable, so it’s better that it becomes harder to do.

[KasMA1990]

And like I said, I’m not actually arguing for having native code. But I do think the upsides of removing native code have been widely discussed, without much good discussion on the downsides of removing it; the downsides being that you put pressure on the developer choosing Elm to predict up front if they will need an escape hatch (as this post makes the case for). It’s easy enough when removing some language features to prove that they are in fact unnecessary, but other features are much more complicated to make convincing arguments for, including this one (and whether the feature is actually a public feature or just a badly kept secret doesn’t change much, once people are actually using it).

[robin.heggelund]

Blockquote
without much good discussion on the downsides of removing it;

But we have mentioned many downsides:

1. The API is unstable and can change considerably from release to release.
2. The code and data-structures generated by the compiler can change based on whether the --debug flag is set, meaning code can work in development, but break in production.
3. Portability is compromised
4. Having an easy way out makes it less likely that people will write better solutions that work well with Elm.

Changing things which are not documented or meant to be used is not uncommon. Java removed unsafe APIs in JDK 9. Clojure changed it’s hashing function, making a lot of code break because people relied on the output of the previous function.

Calling JS directly from Elm was not meant to be possible. The fact that people do it anyway doesn’t change this. The API is undocumented for a reason. This is not something unique to Elm. If you use unstable and undocumented API, you’re taking a risk that things just might not work tomorrow.

[opvasger]

But we have mentioned many downsides:

• The API is unstable and can change considerably from release to release.
• The code and data-structures generated by the compiler can change based on whether the --debug flag is set, meaning code can work in development, but break in production.
• Portability is compromised
• Having an easy way out makes it less likely that people will write better solutions that work well with Elm.

These are downsides of using native-code, not removing it

But I’m pretty sure the downsides of removing it doesn’t go beyond that fact that some of us made the technical decision to built on top of implementation-details; A decision that now makes them anxious about the 0.19 upgrade…

But where are those scary examples that cannot be rewritten using ports?

If people are rightfully anxious about 0.19 without native-code, they would have code they could copy-paste into the conversation to demonstrate the problem with Evans decision? Or am I missing something?

[KasMA1990]

Okay, just to make it crystal clear: I completely agree with Evan’s decision. Again, I’m not trying to argue that we should have native code in Elm; it has many downsides, and removing it is a completely fair call. What I’m trying to discuss is the downsides of removing native code.
All the points you listed are downsides to having native code, and hence they are also the upsides of removing it, which is the opposite of what I’m trying to discuss. Just because there are many good reasons to remove something from the language does not mean there aren’t also downsides that are worth addressing.

[KasMA1990]

If people are rightfully anxious about 0.19 without native-code, they would have code they could copy-paste into the conversation to demonstrate the problem with Evans decision? Or am I missing something?

It’s not quite that simple in my mind. Elm is famous for being “limited” on purpose, which means some developers will be hesitant about running up against those limits. Even before they start a project, they will have to evaluate Elm and consider if those limits will be a problem. Now, if the language has an escape hatch, the risk of making the wrong choice with Elm becomes substantially smaller, because you have the ability to fill the gap if you need to. So when you don’t have an escape hatch, naturally you put more pressure on developers to make sure they predict correctly whether all their future use cases can be fulfilled with Elm.

So what would it take for new developers evaluating Elm to sleep easy, knowing they made a solid call? Do we need more examples of big projects using Elm successfully? More documentation of where Elm doesn’t shine?

[opvasger]

It’s not quite that simple in my mind

Well, you can send any old JSON value in/out of your Elm program using ports; What ports takes away is fact that you cannot tie reads to writes when communicating across; Something that adds complexity when your reads are tied to your writes. But how does some added complexity become a show-stopper for getting shit done? piggy-backing on JS-libraries isn’t helpful to the evolution of the language, yet this seems to be the only case where the complexity of using ports might hurt sufficiently?

Again, I can’t think of any good counter-examples, so people should provide the cases that are making them anxious so we can help each other out

I fail to see how access to native-code makes a project more technically feasible with Elm when it compromises the reason for using the language in the first place

[DanAbrams]

But where are those scary examples that cannot be rewritten using ports?

If people are rightfully anxious about 0.19 without native-code, they would have code they could copy-paste into the conversation to demonstrate the problem with Evans decision? Or am I missing something?

To me, the biggest issue is reusability with both native code and ports. I agree 100% with the native code decision, but I think there’s a big problem with regard to certain web platform APIs that aren’t yet wrapped that require javascript currently.

The one I’ve been working on is the Media API. Just to do play, pause, load, etc, requires javascript currently. And to do more complicated things, like canPlayType or decode TimeRanges also requires javascript.

If you’re using the media API to write something like a podcast player, that’s fine. But the media API is used extremely widely on the internet, yet most web developers never touch it. The reason is that people who specialize in it, like myself, write reusable media players that they plug into their applications or pages.

This is a great use case for a reusable view, and in fact, I’ve written a prototype media API wrapper and a reusable view player (Shameless plug: here and here). I want people to be able to grab a reusable view audio or video player, and use it in their applications. My dream is for Elm-town’s website, for instance, to use my library to show us the audio.

Wiring up the number of ports required for a basic player would be pretty tedious. Doing it for one that supported more advanced features, especially media source extensions, tedious and difficult.

I don’t think Media API is alone in this. What about a reusable view Color Picker that requires canvas? I don’t much about FileReader and LocalStorage, but that’s the one I hear all the time. I’m sure others can think of other examples of reusable views that would be really wonderful to have, but require native/ports and some level of expertise. How about a Midi player written in Web Audio? (WHICH WOULD BE AWESOME)

I think this would be much less of a problem if more of the web platform were already wrapped. And some of the API’s that aren’t already wrapped lend themselves well to ports…that is, they don’t really tend to be reused. I think storage falls into this category, but I’m not sure.

But there’s another aspect to it that I think is causing a lot of the hurt feelings and strong opinions over this, which is that blocking native code without wrapping the whole web platform first creates winners and losers…that is, it advantages some use cases and disadvantages others.

Obviously we’re all using HTML, but from my perspective as a media specialist, it’s absolutely crazy that WebGL, which is not a widely used API from my perspective is wrapped, but localStorage and Media aren’t. I’m sure if I were a game developer I would feel differently. I love that SVG is wrapped, but I’m sure many never touch it, and would rather have web audio or canvas completed.

Ableton was one of the biggest companies doing elm stuff, for a while, am I wrong? I’m sure they needed to do web audio through ports, and that worked fine for them. But imagine how much better it would be for all of us if they had been able to refactor some of what they were doing into reusable packages on the elm-package?

I’m going to spend part of my day refactoring what I’ve done with Elm-Media into a package that uses ports, to help someone who specifically requested an easy to use audio player on slack. It’s still going to be complex for him to set up, as he needs to edit the package to use ports and also add the ports on the javascript side. I’m honestly not sure it’s feasibly achievable yet.

[KasMA1990]

I think this blog post is a good example personally: code that is simple to apply synchronously, but cannot be written in Elm, so it must become asynchronous because of ports. What if you have a project that ends up relying on this a lot, making the flow difficult to understand?

I’m focusing on imaginary code here intentionally, because that’s where I imagine that not having an escape hatch will be an issue for some. Because when you don’t have an escape hatch, you have to predict if you are going to write, just as one example, performance critical code that lends itself well to an async flow, or if the synchronous flow is so important to their code that Elm will be a complicated mess.

I still think many will choose Elm anyway, even if they are worried about these issues. But can’t we do something to their decision easier? Maybe by giving good examples of code that relies on synchronous JS code or similar, just to make it easier to understand how such code will need to work if it is needed.

[opvasger]

To me, the biggest issue is reusability with both native code and ports

Well, reusability for who? You can reuse your own code, you can publish (anything) to npm, you could roll your own package manager at work or use npm against a private git server and probably a myriad of other options I can’t think of.

You can’t publish to elm-package, but that’s definitely not a showstopper for reusability; and down the line I think we will be greatful for that constraint

I’m on my phone at the moment, so I’ll wait until tonight (when I get back on my laptop) to go through all your point!

[DanAbrams]

You can reuse your own code, you can publish (anything) to npm, you could roll your own package manager at work or u

Reusability for other users: color picker, audio player, video player, these are examples of reusable views that are widely required for many web apps.

Theoretically, elm-markdown could do it’s markdown parsing through ports, but it doesn’t, because it would be cludgy. It’s a reusable package that many people want to use and don’t want to have to write from scratch, or wire up ports for. Ditto with Elm-lang/dom. It’s a bit crazy to imagine people hooking up a port just because they want to focus and blur.

Honestly, it may be as simple as figuring out where the biggest gaps in the Web Platform are and committing to getting those in. I suspect that with implementations for like 6-7, 70% of the concerns would go away. Some of those API’s are particularly hard to design elmic APIs around, but not all are.

[opvasger]

Reusability for other users

Then put it on NPM and Github? Why does it have to be the Elm package-manager?

color picker, audio player, video player, these are examples of reusable views that are widely required for many web apps.

What do you mean by reusable views? Do you mean view as in view/model/update?

I don’t think Media API is alone in this. What about a reusable view Color Picker that requires canvas? I don’t much about FileReader and LocalStorage, but that’s the one I hear all the time. I’m sure others can think of other examples of reusable views that would be really wonderful to have, but require native/ports and some level of expertise.

I don’t understand how LocalStorage requires any level of expertise; it seems fairly straight-forward to just wire up ports, decoders and encoders, no?

Obviously we’re all using HTML, but from my perspective as a media specialist, it’s absolutely crazy that WebGL, which is not a widely used API from my perspective is wrapped, but localStorage and Media aren’t.

This hangout is part of the effort that was made to support WebGL in Elm, and as per the work that was done in collaboration with Evan, WebGL was integrated and got pretty good support; This community-effort was, as far as I’ve understood, the moving force behind getting it done, which I don’t think is crazy at all.

I can’t speak to what efforts will go into covering more of the Web API’s, but I do know how collaboration happens, so perhaps this is how you should pursue year dreams of better media-API support

[opvasger]

I think this blog post is a good example personally

I don’t - it’s a big discussion (on multiple topics, at this point) and doesn’t illustrate anything concrete about what is and what isn’t. Real cases are important to get anywhere.

I’m focusing on imaginary code here intentionally, because that’s where I imagine that not having an escape hatch will be an issue for some. Because when you don’t have an escape hatch, you have to predict if you are going to write

Regardless of my opinion of imaginary examples, work on Elm is based on the real issues people encountering while using the language