Afterthoughts around about Kernel code and custom operators complaints

Request Feedback
41

I’ve actually been starting to think that much of the griping (or actual grievance) has less to do with the inadequacy of ports or privileged nature of kernel code and more to do with just implementation of virtual dom that make it impossible to go down to that lower level and wrap the APIs of various html elements in such a way that they’re more usable in a declarative environment.

Hence, for dom interop, the obvious refrain is “use a custom element.”

I wonder if there’s a more obvious way around this or set of tooling that could improve the custom element story from within elm.

42

Like a code generator that will create an Elm module and the JS side of a custom element, and keep them in sync? Along the lines of these ones for Elm to typescript interop:

2 Likes
43

Maybe. I think those are both cool but I think there’s also a bit of a problem of developers not wanting to just use the ports/web components but also redistribute.

Evan once said he wanted you to be able to do an elm install and then npm install and then have thing redistributable that way. But of course, then you have to create the port and so there’s some configuration work to do.

I think web components might even be easier in this sense because You just have to install and include them. Maybe it’s a tool that installs the elm package and the web component simultaneously.

1 Like
44

There is one downside of using web components that, in my opinion at least, is not discussed enough.

Web Components can be, and often are, stateful or/and perform effects. This is sort of what makes them useful. With pure elm view code, there is also some state under the hood. User can type values into input which without onInput won’t send messages but updates its value. Text cursor in input is also a state that is not available etc. Anyway, these things are states of native UI elements mostly and they don’t alter the DOM tree (except for things like ::focus and other state flags). For practical purposes it’s IMO fair to say that elm views are pure functions though.

By embedding web-components into view one always gives this purity away. Adding web component is putting more mutable state and other effects into what would otherwise be pure API. I know it’s cool that web-components allow us to do many things we would not be able to do otherwise but that doesn’t mean they are some sort of free ticket to do them. In fact, web-component spec is mediocre API and web-components are inferior design pattern. The reason why some things can’t be done in elm is by large also caused by the fact that cleaning all the mess around imperative web APIs is ton of work and that takes time.

I’m not saying that we should all stop using web-components from now on. But just because something is possible to be done using web-components today doesn’t mean it’s solved or that it can be done in core language just by wrapping some API.

I personally did use web-components to solve several problems. But I’m well aware that I’m making deal with the devil. There are always multiple solutions to a problem and they all have they trade-offs. I would for instance rather use ports for translations than web-component. Send information about selected language out and send new object with strings in. One has to pass some sort of Locale type around anyway to make these functions pure. That’s not that different from passing around record of strings.

EDIT:

Also I want to add that if you need to do a lot of effects in a view code for your app, then maybe using PureScript would be a better choice. PS has monadic IO system and main lets you do more things. It’s more suited for wrapping effectful patterns in view. Of course this means you will loose some simplicity, that’s the cost of having effects.

9 Likes
45

This is true of built-in HTML elements too. Custom elements are one way to add a declarative api to the elements.

2 Likes
46

HTML elements are not stateful, DOM is. Bindings provided by VirtualDom are not stateful even though they might result in producing stateful DOM (which I’ve described in post above precisely because I expected reaction like this). With any vdom implementation, a lot of effort goes into tuning diffing algorithm in a way that these states won’t mess up on user.

Example 1: Global State

Now have a look at this https://ellie-app.com/8DhgqBYnXvfa1 and try to increment and decrement the counter. This shows how purity can be broken with web component. Call to pure function Html.node "x-state" mutates global state every time diffing algorithm adds this node into the DOM.

Example 2: Node State Loss

Survival of local state of a component depends on diffing algorithm. This means that if diffing algorithm decides that it will remove the node and insert the new instance, the state of the component is lost.
This is demonstrated by this example https://ellie-app.com/8DhvpDkLdMKa1.
Even inputs and other things suffer from this issue. Anyway I don’t think it’s wise to make this issue in your app even worse by adding tons of additional state using web-components api.

2 Likes
47

Yes, I could see that working. I also think that tool would work with ports too - install an Elm package giving the port type, and an npm package giving the JS side of the port. Obviously the application developer still has to declare the actual port, just as they still have to use the web component.

Perhaps it could work by having an additional config file next to elm.json - say elm-ext.json. And define a package format for these Elm extensions that specified how Elm and JS code is to be layed out in a package, basically an intersection of the Elm and npm package formats. There could then be a new package site that holds meta-data for these extended packages, and the tool to install them.

48

I’m sorry, but some built-in HTML elements ARE stateful—that is they update their state without any user interaction. You don’t need to create a web component to have this issue, and a web component can solve it.

<audio src=“whatever.mp3” autoplay muted></audio> is a stateful and changing element.

In the virtual dom, this element would be created as a dom node, and without any intervention by either the user or the developer, properties on it would change regularly.

There are built-in elements in HTML5 that are designed explicitly to work with function calls to update state, particularly audio and video, but also canvas. How would one create a video conferencing tool or a YouTube clone without interacting with stateful elements in some way that involves either a web component or reaching into the dom via some sort of query?

Declarative is very nice. Not all Built-in html elements can be used appropriately in a purely declarative way. Web components are one way we can build a declarative api around these cases, and I would argue a better one than document.querySelector(), given the current state of the virtualDOM and the language.

UPDATE: @turboMaCk you talk about state flags like ::focus, my point above that there are other cases like focus in built-in elements that have not been wrapped by Elm natively.

50

I’m sorry, but some built-in HTML elements ARE stateful

I guess the part of problem is terminology. What I mean by html element is declarative HTML markup <audio src=...>. The stateful part is DOM node which is created after this HTML is parsed and translated to DOM. Anyway that’s just to clarify something I said above.

Yes indeed audio is one of the elements I didn’t even thought about but it’s really nice example. As I said above even input has this hidden state. For instance inputs loose focus and position of text cursor in situation demonstrated in example 2 above. But it’s a good point that audio is even more problematic. For input we at least have Dom.focus to at least restore the focus (we can’t restore position as far as I know though). I guess audio node would in example above also be broken.

You’re definitely right that these binding exposed by Html module are as problematic as web components are. Thanks for the example :+1:

1 Like
51

So I used to make my living working with audio and video tags, and to the small extent I’ve done frontend development in the past year it’s made heavy use of them.

It’s a daily reminder for me, and has been for four years, that these elements are broken in Elm.

If the answer is, well you shouldn’t use Elm, use Reason or PureScript instead, because these elements are broken, then, to answer the original question, this bothers me. It makes me feel excluded, it makes me feel unvalued by Elm’s design, and it’s frustrating and at times depressing.

When I’m also told don’t use one of the two, fairly limited outlets provided by the language because it violates purity, it frustrates me tremendously.

It’s great that Elm goes for this purity. It’s part of the magic of the language. But there are parts of the underlying platform where some impurity is required, like the dom, like focus and blur, like WebGL, where either the core team has come up with a solution or allowed third-party to via an elm-exploration.

Elm-explorations was supposed to be a solution to this very problem, but as far as I can tell, the only elm-explorations are the ones that were in the initial batch almost two years ago. And some of them are great: people are creating incredible abstractions on top of Elm-WebGL that make our whole community richer. That’s not realistically going to happen with ports or Web Component solutions.

Maybe this is becoming my own “Why I’m leaving Elm letter,” I don’t know. After four years of banging my head on this, trying to get Evan’s attention, trying to contribute a good design to help others, using the alternatives we have in my own work, I really am fed up. I love this community and my friends within it, and I appreciate the beauty of Elm and Evan’s vision for it, but it’s depressing, frustrating, annoying and painful, and it feels like some Browser APIs are just more important here than others, and ergo, some people’s work is more important than others.

Some of us are frustrated not because we think it should be impure, not because we want to wrap some third party library or implement jQuery, but because we have APIs we use, we care about, and we are expert in and which are going scant attention by Elm, while we watch other APIs, like file, bytes, webGL get implemented.

6 Likes
52

TLDR I’ve been in this community for four years and witnessed the same argument happen over and over.

I think the reason is more complex than there are bad actors muckraking. It stems from the good design decisions which necessary exclude some things.

Some of us have APIs we care about that don’t have native wrappers and we wish they did, so that abstractions could be built and distributed on top of them. Some of us have the APIs we care about wrapped already and therefore don’t fully understand the frustration of those who don’t.

It’s not a gap in expectations, it’s a gap in inclusion.

We don’t really have any communications outlet to resolve this gulf, and elm-explorations, which was supposed to provide an outlet for us to experiment with these APIs but the barrier to entry there has been enormous. Without some sort of outlet, some people get very upset.

Even if I disagree with the articles arguments and his need to write it, I sympathize with the pain he felt, because I’ve felt it too, and it does seem like that pain is not fully felt by everyone in the community, which is isolating.

6 Likes
53

that these elements are broken in Elm.

I agree. They’re broken in my opinion too.

shouldn’t use Elm, use Reason or PureScript instead

Well I think if you really need to work with similar things a lot I think you might be better off using one of these languages instead. At least I would use them myself. I don’t mean to exclude you though. In my view at least I’m trying to do quite opposite. I’m afraid to recommend Elm to anyone with use-case in which it might be painful to use. That leads to frustration a potential conflicts which will lead to feeling even more excluded and therefore frustrated (see the recursion). I understand some people need to deal with these problems and I want them to succeed.

Elm-explorations was supposed to be a solution to this very problem, but as far as I can tell, the only elm-explorations are the ones that were in the initial batch almost two years ago.

I myself agree with this part of criticism. I belive elm would be better off if cmmunity would be allowed to do a bit more to help. For instance one of the motivations for locking kernel code was possible future in which elm would compile to native on mobile. Meanwhile while there is nothing like that in elm, purescript (which is now basically controled by community) already has backends to golang and C++. I think that’s empirical prove that while it is risky to hand more control to community, great things can happen as a result of that.

Maybe this is becoming my own “Why I’m leaving Elm letter,”

Even if you decide to go with other language for these project it doesn’t need to be about leaving the language. I’m for instance always looking for opportunity to find a nice project to do in other language with different set of trade-offs. I did projects in both PureScript and Reason and even GHCjs and never thought about it as about leaving the Elm.

we care about, and we are expert in and which are going scant attention by Elm

I checked your github account and have seen elm-media. I think your design is good and aligns pretty well with what I feel (without being expert myself) correct solution should look like.

For what it’s worth I think your and mine arguments are the same. Web components (and broken stuff from elm/html) are not optimal solution and have own problems. I also think it would be better to make elm-exporations more open and community controlled. I just didn’t express that in first post because I wanted to focus solely on explaining why I don’t consider web-components to be the optimal solution. Anyway I don’t want to feed your or anyone’s frustration. I have no influence on openness of elm-explorations and it probably deserves it’s own discussion.

I feel I’ve said everything I wanted. I’m switching back to the role of silent observer.
image

6 Likes
54

Strongly agree.

The idea of it being called ‘explorations’ in my mind suggests that it would be perfectly ok for experimental, even bad, ideas to be tried out there. It should be a proving ground to shape up good APIs that could become part of the kernel and the barrier to entry should be knowable and achievable. Sad to see that Dan is keen, has the energy and know-how but somehow cannot get there.

===

Bad ideas in elm-explorations should be killed off. elm-explorations/markdown is very easy to break and make it generate runtime exceptions. It was useful to get markdown started, but now we have much better pure Elm solutions. I think it only hangs on because it is used for the package site.

On the other hand, has elm-explorations/webgl not worked out as an experiment? Should it not be going the other way and moving from explorations into the core?

9 Likes
55

I totally agree, a clear route to getting a project in elm-explorations would go a long way to dealing with people’s uneasiness around privileged access to kernel code. Maybe there could be "expert panels” for particular themes (video, storage etc) that review proposed APIs and implementation plans for modules, and lightly filter obviously bad ideas, letting the rest through, maybe with some feedback/guidance. The most important thing is that the process of both selecting the panel and reviewing module proposals should be as transparent possible.

5 Likes
56

That’s how Linus Torvalds operates, isn’t it? Appears to have worked well for the Linux kernel over the years.

A small hierarchy of trusted decision makers to filter, feedback, and help improve on ideas coming up from the community, with the core team making the final decisions of what makes it in or not.

4 Likes
57

Maybe a different way of framing it would also be useful: Elm itself is designed for user success, or making it easy to do the right thing, and hard to do the wrong thing.

Is this a way of thinking that could be applied to working on Elm too? Because right now it seems that the process of trying to contribute to Elm actually sets people up for failure. elm-explorations is one place where we’ve seen this now, but I think it’s the same problem that occurs when someone creates a PR or an issue that doesn’t get a timely or useful response, like in this thread.

I realize things are this way for a reason, and I’m not suggesting any change here is easy or simple. But maybe just asking “How can we set the user up for success both when using Elm as a tool, but also when interacting with the process around Elm?” is helpful.

3 Likes
58

I think the reason things are the way they are is likely to be lack of time on the part of our BDFL. It would take time and effort to set up a process for these Elm explorations to be coordinated, managed and evaluated.

Here is a suggestion of how I think we could set up our own Elm explorations that functions better and opens the door to contribution.

  1. Fork the compiler and rename it as requested above. I sort of like the name ‘Elm Pro’ - Elm but with a more professional attitude (not to imply that people here don’t have a profession attitude, I guess I mean an attitude that is better aligned to professional needs and adoption). Perhaps that is too close to the original name.

  2. Clone and modify the package repo to create a brand new empty one. This one will accept kernel code, so long as it is from the ‘elm-experiments’ or ‘elm-pro’ namespace (or some other names) on github.

  3. Modify compiler so it extends the moratorium on kernel code to ‘elm-experiments’ and ‘elm-pro’ namespaces. To use an ‘elm-experiment’ a flag must be passed to the compiler --experimental. The reason for this, is so that users must opt in and know that they are using experimental code - there is no come-back to complain if an experiment is abandoned or removed. Perhaps the version numbers on experiments could even be kept under 1.0.0.

  4. Modify compiler so it will also install packages from a new primary repo described in 2. If a package is not found in 2, it will fall-back to package.elm-lang.org. In fact, 2 will probably set itself up to mirror package.elm-lang.org, rather than redirect to it.

  5. The new package repo will not accept any non-kernel packages, unless they depend on kernel packages that already exist in it. This is so these pure Elm packages find their proper home in package.elm-lang.org.

  6. Create a written set of rules for a new project to be started in elm-experiements. Set expectations and a minimum barrier to entry but the aim would be to be quite open to new ideas.

  7. Decide and write down the rules for moving a project from elm-experiments to elm-pro once it has reached sufficient maturity and quality. The quality gate needs to be stronger here obviously than for creating an experiment. The process needs to be transparent and fair.

The forked compiler could find a new home somewhere like apache.org. I don’t really know if that is the right place, its just somewhere grown up that knows how to manage open source communities. You want somewhere that can bring some impartial supervision to a project - so there is somewhere to go if you need to complain, get advice on how to manage things, etc. Somewhere with some standards, like forcing a project to have a written set of rules and guidelines to contributing and so on. Anyway, maybe Apache is not possible because of its licence, does it clash with the existing Elm compiler licence?

The forked eco-system would have a stated aim that all new kernel based API functions must be pure functions. Exposing non-pure functions is grounds for a package to be removed, or rolled-back to its last pure version.

There is an opportunity for a forked compiler to get some patches applied to it to fix long-standing bugs. This is a slightly risky thing in a set up like this, as a package may depend on a compiler bug. I think some odd things around how arithmetic behaves in Elm should be left alone for example. But the recently referenced bug around the debugger overflowing the stack is an example of something that could be safely fixed (would need to fork and patch the elm/virtual-dom package to fix that one). You want to fix bugs but without changing the intended behaviour - its probably best to be fairly conservative about this, at least to begin with.

The closer the forked compiler remains to the original - the better. Could continue to track all new releases of it that way. The forked version would also build original version code, so no need to keep switching between them. You could even symlink it to elm on your system.

===

I don’t know about you, but since the coronavirus started, I’m down to about 2 days a weeks computer time due to home commitments. I also have done the due diligence on my current project and satisfied myself that the existing Elm eco-system can do everything I need it to. So I don’t have the time or energy to do something like this.

Also note, this is not meant as a 2 fingers to anybody or some kind of hostile attempt to undermine things. The idea popped into my head so I just want to put it out there and see what people think - in a grown up way please.

I also think the steps I outlined above are a minimum effort plan to getting a working elm-explorations and as a lazy engineer my brain always likes to throw up the MVP. It really is not going to do any good complaining about not having a working elm-explorations - its seems pretty clear that will not change anything.

7 Likes
59

@rupert Thank you for writing this out!

If this can be pulled off in a generally “approved” manner, I really think a community-driven fork for experiments would benefit upstream as well, as long as the goal really is and stays to be a research project.

  • This fork, being generally advertised as non-stable, can test and review patches to the compiler or core packages much more easily, since the (production-readiness) guarantees are a lot weaker.

  • People without any other immediate option will have a clear path to unblock themselves, as long as they are aware that they are entering unstable, unsupported territory - by downloading a seperate binary, reconfiguring all the tools, passing some additional --unsafe-beware-dragons-here flags.

  • Feedback on experiments would not rely on the core team reviewing and approving things anymore. The Elm core team does an amazing job, but only has limited resources themselves. There are probably more people sharing similar experiences to @Dan_Abrams. I’d rather collect all such projects and ideas in a central place for the community to try and discuss them, instead of having them end in a single “Proposal”-style Discourse post, where the OP will get demoralized by technical and process-related anwers of why “things don’t work like this here.” This will also keep those folks motivated to work on these projects, and with Elm in general.

  • The core team does not have to look into everything anymore, but can focus on the much smaller output of that project - the community can iterate and test different designs independently and much more quickly, meaning the core Elm team only has to look at experiments that worked. Proposals and ideas will be essentially filtered by the community itself.

  • Discussions like these (+ the original medium post) would have been resolved by a posting link.

I’d really like to focus on how to handle this as a community though. I’m not certain that clear communication from the fork will be enough to ensure not fully splitting off from the main project. How will we deal with people just using the “Pro” version? This will definitely happen, especially if answers to the common questions people encounter while learning Elm (localStorage, WebSockets, that-jQuery-Fancybox-Plugin-they-have-to-use-at-work integration) become “you could do this the official way, by manually building all this elaborate wiring where there’s a chance you might not understand half the code and concepts if you just started learning, or btw this community fork has a package for it that might work, just need to pass that extra flag, don’t worry about the details”

Can we prevent the official channels (Slack, Discourse) from having to deal with the problems of “pro” users unrelated to “core” Elm? Would this be a problem for the Discourse? On Slack, making and referring to a separate channel sounds reasonable. A more clearly separated branding would certainly help, but it also increases the possibility of separating the community as well.

How could something like this be organized?

PS: It may be a good idea to split this discussion into a new thread.

3 Likes
60

I like the idea of community based experiments that can feed back into core elm. I am concerned about it becoming a complete fork that’s just Elm + native access. Would it make sense to place restrictions on the forked compiler? For example, only allow the use of elm/* and elm-explorations/* packages. This should allow freedom to experiment while hindering the use of the fork for anything production.

If this is a fork for the purpose of experimenting/exploring, then I wouldn’t see this as something for “there’s this one bug and I want it fixed now” scenarios. Make it easy to test ideas and hard to build production apps.

2 Likes
61

Why do you think core team would want to merge forked compiler experiments if that does not happen now?

3 Likes